I really don't know what to tell you. I know couples I wouldn't mind being married to along with my husband (and my husband wouldn't marry them, but wouldn't mind swinging with them) - and we simply know better than to ever bring it up because we are that sure of how conventional they are. Right now, I am pretty happy as is, and even if my husband ends up swinging a bit on his own, I don't feel a burning need for another relationship. Wouldn't toss one away if it developed though, either.
Probably you need a local outlet at which to meet other poly minded people, which can be not the easiest with a kidlet or more at home.