His swaying is referring to him collapsing to the floor, dead. Not him attacking. Though the film clearly shows that all Kolar does it raise his blade, and funnily enough it takes more time to cross a distance and raise one's blade to intercept an attack than to simply raise ones blade.
No, Kolar swayed, he did make a move, but Windu did nothing.
Also noting that Windu was likely recovering from surprise, and that again, Sidious' speed is force augmented. Even if this were not the case, Windu still would struggle to intercept that attack.