A lot of times I didn't like that the Imperial Agent story was so similar to the planet storylines that I couldn't tell I was doing a class quest unless I looked in my quest log.
That's actually one of the things that made the IA story GREAT to me. It blended so seamlessly into the planet's storyline that it made the entire planet feel like it was part of the IA story.
It never felt like "I'm here to do X, but I'm getting sidetracked on fifty million other random, unrelated things, yawn".
Each planet was one giant class quest.
Even Balmorra didn't feel as tedious with the agent as it did with the other classes I've run through there on.
I guess it depends on how you look at it though.