It's mentioned a few times during the game and further enforced by Malgus' actions. The biggest problems that face the Empire is not it's economic stability. The Empire actually indulges in many profitable opportunites as you play through the planet quest chains, not to mention the bulk of it's work force is slave labour.
Sith infighting is obviously a major problem for the Empire but it's also it's xenophobic stand point. Soo many races/planets could have joined the Empire however were turned down due to the fact they were 'aliens'. With soo many planets breaking away from the Republic, this was prime opportunity to make them members of the Empire.
If the Sith could unify and the Dark Council threw it's Xenophobia to the side. The Republic would have no chance.