I have found that the biggest issue with Arena is that if you go up against a better team and lose both matches you don't get anything for your troubles. Especially when someone leaves and you are 3x4. With me it makes it more difficult because it can take a while to load in and out of PVP.
I used to think that it was bad to leave PVP before the match is over but I can see the point now. As far as the issue of longer queue times waiting for a match, well they can give the option for one or both types of PVP. If someone only chooses one type and it is taking too long for a match they can change it to have both types. This gives the player a choice.
I also agree with people that suggest cross server PVP.