It's not gear that's the main problem, I've seen well geared players in SM not knowing their roles especially tanks not kitting bosses correctly (i.e. for fabricator in KP) and even seen some people doing stupid things like running into the lava pit full of dogs in EV just before gharg and when revived running on straight to gharg without waiting for people to ready up without knowing tactics.
I think for group finder there needs to be some sort of skill scoring before you progress on to each operation, maybe even some sort of training op before you can do the story modes then having to finish all the SM's before progressing on to HM and so on.
The problem is (and was probably before GF with pug groups) skill and experience.
If the problem is skill and experience, then they are only going to get that by doing that operation. Making it hard for them to get into that operation will only work against that. Something like a skill score is impossible. How would you increase that? By successfully beating bosses? First of all that doesn't tell anyone if you're really a good player or even if you understand the mechanics of that boss. Second of all, how can you increase that score, if you need it to be at a certain level to enter the operation.
It's like it was in WoW with gear and achievement requirements. If somebody wanted you to have gear, that you could only get in that raid, or the achievement for completing the raid, it was hard for new players to even get that, because the groups that you needed to get into to get them required that you would already have them.
WoWs raid finder sets a pretty good example of how they can teach you how the bosses work. It's very intuitive and it works as a training ground to teach you the basics. That only works in WoW because the difficulty is a bit lower and you can get the right gear for that raid in like a day.