I'm runnign out the door right now and don't have time to put in a long response ... which I will later ... but I definitely understand your concerns better now.
I'll tell you that right now I'm in a relationship with someone who swings with his wife. I don't swing - I'm like you and need the emotional connection. It doesn't bother me that he swings - with or w/out his wife - and he understands that I prefer a stronger emotional bond. It's something that we've done a LOT of talking about A lot. And then even some more talking.
I have more thoughts about where my guy and I have come to some realizations and I'll post them later tonight when I have more time ... but yeah. It's workable (or has been so far for us). It just takes a lot of self-knowledge and communication.