Would it be helpful for you to write everything out? That way you can get to the bottom of what you really want to discuss without having to cover every conversation they have with each other ect? Also it might be helpful to him if you can agree on a set amount of time per day or week to talk about poly stuff so that it doesn't seem like that's all every conversation the 2 of you have with each other is about.
Everything will be ok in the end. If it's not ok it's not the end.