That's not too harmful. It's the person who doesn't want to contribute to the rest of the group, but gets all huffy and annoying to you about it, and making you carry them.
Yeah, I don't mind if you AFK once in a while. It's the rest of it that gets irksome.
Then try that when you don't complete the encounter, or when it's the person being willfully uncooperative, not just a factor of the game mechanics.
You were willing to take somebody along, but that's because you could.