Try whispering them first. Most people are put off by an invite that has no message attached to it and may not realize it's coming from the person in the same area as them (I've gotten invites from planets away for no reason at all).
Have to agree with this. If a group invite suddenly obscures my view of what I'm doing without being PRECEDED by pink text in my chat window from that person, I will admit to feeling a brief stab of boiling fury followed by annoyance at myself for overreacting
Always ask before inviting.