I agree and I think this is the root of the argument. I hate to use another Nazi reference but I think it fits well here. The USA has done a number of unquestionably morally dubious things over the course of the 20th century. For instance, interning Japanese Americans during the Second World War. But that doesn't mean that during World War 2, the Allies were exactly the same as the Axis powers.
The Nazi argument is the best argument you can put forth. The Galactic Empire was a sci-fi copy of the Third Reich. The Empire even had Stormtroopers. If that doesn't say anything I don't know what will.