However, what if affection is low and I don't buy trinkets for them? Do higher level missions fail a lot with low affection?
Affection has nothing to do with fighting performance. It opens up some companion side quests, expands their story background, slightly improves their "gathering" missions performance and crit, etc. You may keep them at affection level 0 the entire time, and won't change your actual combat whatsoever.
Companion Affection: What it does and how to get it