I might be speaking out of place but I'm gonna say it anyway...
The 2 couples have been swinging for 2 years and "exclusive" for 2 months. Has it not occured to you to simply state "Whenever we bring up Polyamory, you guys seem very uncomfortable. Is that because we are putting too much emotion into this or what?"
Surely you all have talked about more intimate things than that... so why stop here? Do you really want to invest the emotions and intimacy into a relationship that isn't what you think it is?