Yes, it appears the next step is discussing the matter with the couple--and your bf. Wouldn't be cricket to start up with them without figuring out what's going on with him.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.