One of the advantages of polyamory (or swinging) is that it takes away the burden of having to try to be everything for your partner. For example, I know couples who got into swinging because one was heavily into BDSM while the others was not interested. Everyone gets the needs met and no one is doing something they rather not do.
To a lesser extent, it is true of other things like going to a movie (I don't like chick flicks so my wife has a friend who goes with her to see those) or resturants (I don't like Indian food, but another friend of my wife does).
Your hurband should check out NRE and that may help him understand that your reaction is not a rejection or reflection on him.