I'm pretty sure he despawns after a certain number of pillars have been thrown to the ground. If you have an off tank, I'd recommend taunting him when your main tank gets mind trapped. We did this fight about 10+ times on nightmare last night before finally killing him, and we had our fair share of bugs (example: if someone is getting flung around the room when the floor drops out, they get teleported to the bottom instantly but are not in the same zone as the rest of the raid...)
This may be a glitch for you, but it also may be you are taking too long, I don't know how long you took.