I have fun a lot in PvP, win or lose. I don't mind losing when I know it's because the other team is obviously better. Better tactics, et cetera. However, if they game is one of those where bashing my head against a brick wall will be more productive than staying, I'll leave. Too many afkers, people running around who obviously don't care about objectives, or half of the team a guild group sitting somewhere having a dance party (yes, I have seen it several times, same guild) is absurd. Granted, I don't leave often, but I have left. There is a point where you just cut your losses and step away.
Yes, it can be an issue at times, when there is a constant dropping. It's detrimental. The only thing I think would help it, though, would be after they've implemented rankings. An early leave counts as a loss and a reduction of ranking (on top of losing out on the free valor/exp/comms you get for a loss). That way, you don't get those exploiting it by leaving early to prevent a loss. AFKers are (sort of) taken care of. I haven't tested out the removal system enough to see how it works, but it ought to be if you are making no significant actions to help the team out for so long, you are automatically removed. Just slapping a debuff would just increase the AFK abuse, and make things even worse. There has to be a balance in the things implemented so that you aren't opening another avenue of abuse/exploitation and still maintain a level of enjoyment.