You'll have 1/3rd less armor on one of your three biggest armor pieces.

Thats an hefty penalty imo for looks. To answer your question, it might not cause a wipe, however when I'm putting an ops together in a pug, I give a quick lookover to people's gear for anything harder than sm EV and KP. Someone with wrong armor pieces start on a bad note, especially since there is no gain for it other than "it looks good".

Edit : oh yah. And if you start being serious about being absolute BiS, I might just give you a very incredulous stare.