What is the group limit for doing the boss?
At 1 stack, no more than 8 couldnt start without getting the debuff, or then it bugged out or we were too far away.
We pushed the dreadtooth to 4 yesterday, but managed to get only one go for it. We had 16 players, three geared healers and one in rakata/columni, and looked like we could use more healing there. Couldve tried with more, but as there was republic players ruining the run by starting the fight, and if one of our guys was too close to boss, we had to wait 5 min or something for the debuff to get removed.
You can pull Dreadtooth at 1 and 2 stacks with more than 8 people. Not sure why you got the debuff but my best guess is some people weren't quite engaged at the beginning of the fight.