"That's the thing, we have had a conversation about poly but their actions seem to be differing from what they say they want. The first 'date' we had we all talked about how empty the swinger scene seemed & how we all wanted relationships not meaningless sex."
That's a good starting point. The cues you're getting from them can mean different things: we just had a fight, one of us or both is uncertain, one of us got jealous because we were hit over the head with "hey, you don't do that with me" thoughts... or even something as mundane as, it was a really bad day Friday, and we should have just stayed home. You won't know unless you ask.
I hope that whatever shape your relationship takes, it is beneficial to all of you