With addons in place, you start paying attention to the numbers and your performance instead of just playing and enjoying the game. They drain the fun out of a form of entertainment. That is why casual players don't enjoy them.
Pretty much this is the problem.
It's two different playstyles, and two different ideas of what's fun. For me, the numbers and performance are fun. The game itself can be enjoyable on it's own for sure, but in the end game, how may times are you going to run the same hard mode flashpoint just for the "experience" of it. In the end, the gearing up and numbers are all that the end game players have.
It's not right or wrong to enjoy the game one way or the other.
I don't care if you enjoy your immersive experience without the numbers, that's fine. Why do you care if I enjoy the numbers when the experience loses its luster?