Honestly, you should be able to talk about things bothering you in any of your relationships, what ever their nature may be. This is essential.
Secondly, I am quite concerned about the comment about the wife/husband swappers. Do you feel that this is about sex? Do you feel you are not considered a 'proper' relationship and just some fun on the side where after he is able able to return to his real and only relationship?
I ask, because it is possible to have more than one 'serious and meaningful' relationship in a polyamorous setting. Poly isn't just hierarchical and that's it and just the main partner is worth some consideration. You should really talk to your spouse and ask for some things you are missing or that don't sit well with you. You can't change the fact, if he doesn't want to change anything, but you should always be able to at least ask for it.
Facts: 30, female, bi, v-type relationship with Sward (husband, straight, mono) and Lin (boyfriend, straight, mono), poly-fi and co-primary.