Ok for anyone wondering, I figured it out. Somehow its the companions messing it up. I think the game is trying to let them do the action but they aren't doing it, even if I wait for them to be ready.
Dismissing the companion, activating the shortcut, and then resummoning the companion works.
Hope this helps someone else too