Interesting that you point that out, Aurbere. I wasn't saying that the Rakata were invincible, only that at the height of their power, one of them was probably one of the most powerful force users ever, based solely on their species' mastery of the Force.
The invasion, IIRC, was sometime between the golden days of the Rakata when they ruled without question, and the slave revolts that rocked their power base. Then the plague destroyed their force sensitivity and it was game over for the Infinite Empire.
Ill admit my entire source of info regarding the entirety of rakatian history may not be the largest, but I'm no fool when it comes to the subject either.
I have no idea if they made a solid case or reason or source regarding the plague, but giving its "selective nature" in killing only rakata and its subsequent stripping of the dark side of the force from those same being, made the case that it was a splinter group of slaves that created the plague. The rakata, not counting the plague, were hardly invincible, the problem with them in the eyes of the slaves, is that they were ALL force using, their use and mastery of the force can be called into question, however, given that it took a galaxy wide plague to finish them, they must have had many of the different sensory abilities to foresee and perceive danger and hostile intent. So compared to a normal being it would have been hard to catch a rakata off guard, as they would always be on guard for the slightest whiff of revolt.
Also another problem and downfall of the empire was the internal power struggle and civil war that ensued toward the fall of the infinite empire. Once it seemed that the rakata had the entire galaxy "that was known to them with their limited ability to travel the galaxy (since their version of the hyper drive was not as refined as the polished corellian version)" they turned on each other vying for power and influence. not only that, but their numbers were never large to begin with compared to the entire population of the galaxy.