Remember the rules:
No outside help of any kind so Xizor's pheromones and charm are useless here, nobody can bring in the locals for help. Xizor's just got Black Sun resources. Kun's got his support, Traya has her support.
"People of the galaxy" arguments have been vaild before. See rule 1.) Arena: The Known Galaxy. Last time I checked, the known galaxy had trillions of people in it.
The no outside involvement rule is refering to major powers, Ex. The Galactic Empire. Any major organizations tht would rival or ally with those in the Kaggath no longer exist, however that is not to say that the rest of the beings in the galaxy have disappeared too.