I think that making an appointment to see a professional that can help you work through this is absolutely the best thing that you can do.
The logic of things like this are rarely obvious, and sometimes it takes a lot of digging and talking around things to hone in on the root of your issues. I don't think that anybody can really give you concrete advice on an internet forum, where they don't know your background, and can get into a conversation with you to explore this.
Poly can quite often bring insecurities to the fore, especially around the "why aren't I (good) enough for him/her". So this may well be a big opportunity for some personal growth for you.
"Listen, or your tongue will make you deaf." - Native American Proverb