I kept my ghosts >:3 BUSTIN' MAKES ME FEEL GOOD.
Anyway, if we are talking DS Nox I don't think he/she would just let those ghosts go. And I assume Nox could get more if he/she wanted.
Adding more would probably reintroduce the problem that was fixed by Chapter 3. You're supposed to bind/pact with only one spirit at a time. The reassembly and heal ritual fixed the Inquisitor's current problem, but there's no guarantee that trying to stretch it again would remain safe. In fact, I'd think he'd just have to do it all over again.
I told my spirits they had to stay, but one left anyway (Ashara's ancestor, if I recall correctly), as the contract was fulfilled, and he didn't feel like sticking around. Most did, though, thinking hanging out with me was better than sitting in some tomb somewhere.
Even with the backing of three spirits, though, I'd still go with the Wrath. As you said, it's in his very job description. Sometimes, the Wrath has to purge the entire council. Difficult as it may be, it's going to get done.