I just call my husband my husband, my boyfriend my boyfriend, and his wife my girlfriend, or my boyfriend's wife if I'm trying to keep context. I guess though that because I'm married, this makes it a little easier to do that.
However, I don't think of my boyfriend and girlfriend as 'secondaries.' I just call them partners or sweeties when describing my situation to other people. It's easy.
You're right, I don't think anyone needs to get worked up. I do think though that we should all just do what we think is best. Don't worry what others think of your wording choice. If you want to use the term 'secondary,' cool beans. If you don't, 'cool beans.' Anyone that would judge you for this is pretty silly I think and should worry about themselves. We're all so different that it doesn't make sense to compartmentalize (unless of course, that's your thing!)