I was wondering if he offered to introduce the two of you. Not anything sexual but just to meet and maybe get to know each other a little. This seems to be a typical thing in poly (unless the relationship is a DADT kind).
I am concerned though. Something about the way you tell your side of the story makes me wonder about his true motives and "poly" lifestyle. I hate falling back on these when online, but I have a bad feeling about this. He kees taking back or changing what he's saying or "coming forward" about something in these nice long conversations that you two have. He also is getting frustrated with your frustrations instead of trying to help you overcome your fears and concerns.
Sure, he may not be your official boyfriend, but if he cares about you and wants to be with you, he will try to help you come to terms with and understand his choices.
Life is about the journey and not the destination,
so what better way to know life
than to wander all the roads and paths set before you.