Ah, okay. I think my confusion is one example of why it is suggested in the Forum Guidelines to have a separate login for each person who is part of a couple. Either that, or when each person in a shared username "speaks" in a post, make sure you identify who is the one speaking. Two people in one shared account gets confusing real fast - and besides, each of you may want to address certain things separately at some point.
Okay, so the title of your thread is "swingers or polys? what do we call it?" and then later in your post, you addressed not knowing what to call your relationship and the bit about jealousy. Then you (one of you or both of you) asked for "suggestions, feedback, etc." What is it that you want suggestions or feedback on, exactly? I'm still not clear.
Regarding what to call your relationship, I would say you are swingers who deepened into a friends with benefits type of thing. It is definitely NOT polyamory because your guy friend's wife did not give her consent to his having any relationships outside of their marriage, and cheating is not poly. But labels aren't always that important.
As for the jealousy, there are numerous threads here addressing that. You can do an advanced search or search the tags and will find a gazillion of them. We have a master thread that discusses various aspects of jealousy here: Jealousy, Envy, Insecurity, etc. General Discussion
. Do you have specific questions about it?