Just a comment or three here in case it may be helpful......
1> Discovery of poly (potential) from swinging is VERY common, and also the root of so many horror stories because people found themselves there before they had done their homework. That's about where you seem to be. I suggest taking a time out (NOT a disconnect) and doing some studying (all) and having some interesting but deep conversations. Try to envision what the arrangement COULD possibly look like once it matures beyond NRE phase.
2> You probably know, but the difference between swinging & poly for most people is founded on a difference between sexual activity and emotional connection. Many swingers intentionally ban (or try with mixed results) emotional entanglements because they are just afraid to open that can of worms. Some that end up opening it anyway (often unavoidable) discover it is really only gummy worms - quite palatable after all !
Others find venomous snakes and die from it ! You all need to decide if you are even up for the challenge of opening the can at all !
Oooops ! Interruption - gotta run. You get the idea........