sometimes doing this will knock you out of the queue or prevent you from declining after three tries. i have had this happen but am not sure if it was a bug or what ever. also you must be careful who you select as the fourth person as there will be a butt load of dps queued ahead of your fourth.
That's why you have the tank or healer be the odd man out. There will be less of them and if you have a full group ready to go, you should get it pretty quick.