The downside as already pointed out is how punishing Fearful is someone screws up. The other things is the DPS loss which can be an issue if your DPS is low/undergeared. Not a big issue for you configuration as you described it since 3 of your 4 DPS can leap to the boss instantly and the other has force speed you have almost no DPS loss. Compare that to a group with a couple slingers and a gunnery commando (what we usually run with) and try doing swaps like that and watch your DPS plummet due to the constant running your turret classes now have to do.
Why do ranged classes have to run? Also, in our way, nobody has to leap anywhere: two DPS stayed on right, other two on left and its the bosses who swapped places.
We did it since then with ranged classes as well, tried the tactic of keeping melee on Toth and ranged on Zorn while swapping the tanks but it was still easier just to swap the DPS. Especially with ranged, they just had to press Tab. So even if they got Fearful via bad positioning - which happens, it's not always easy to calculate where exactly Toth will land - they can still do their stuff without problems.
To me, it seemed far easier if everybody could stay in their starting position and we only had to move the bosses themselves. When the tanks had to run through the entire field it almost always led to problems