I wouldn't make too much out of them starting up when you leave the room. There are a lot of possibilities. It could be NRE. It could be that they feel awkward and feel sex would break the tension. It could be that it is easier to start up sex when it is two people than three. I think it is worth talking about it, but I would not let it turn into too big of an issue. Figure out why this bugs you and work from there.
I find a good way to see if polyamory is for you is to look up compersion and see if you think you can feel that. And then imagine if you feel you could love more than one person at a time. If both ideas seem plausible, you may be ok with polyamory.