I own the CE and thus the authenticator. Not once have I used a purchased fleet pass. I use the emergency fleet pass (with the 18 hours cooldown) once every day after my dailies, to go buy/sell stuff on the GTN and I log off.
I don't see how fleet passes are really needed to enjoy the game. There is QT or taxis to get to the spaceport quickly, and on most planets there's even a quick shuttle to the fleet for Flashpoints.
If you don't have the patience for these quick ways to go to the fleet, maybe you should play a quicker game, like heads or tails maybe.