I use R as my keybind nearest enemy, it works fine. However, if no target is selected or my target is dead, a party member or my companion becomes my target, usually after they cast an aoe. I've tried turning on all options, such as auto targeting and sticky targeting, and turning them off, no change. I just have to live with it. One of the few things in this game that's really irritating.