With addons in place, you start paying attention to the numbers and your performance instead of just playing and enjoying the game. They drain the fun out of a form of entertainment. That is why casual players don't enjoy them.
Believe it or not, some people actually enjoy min/maxing their characters. Also, you use the term "casual" instead of "bad". Shame on you.