I'm on 3 different realms and I'm seeing the same good number of people. And for the low pop realms, I don't think you'll ever see a merge. Gaming companies don't seem to like doing that like blizz. Half of the wow realms are dead and blizz hasn't done anything about it in years.
I'm on 3 different realms, all of them have 200+ people in the fleet atm. If you have 5 people yours, you're either in another phase and you don't realize it or you have really bad luck with realm choice.