Have you been able to identify what is making you feel uncomfortable about being poly? Based on what you wrote, you certainly sound like you have the poly mindset.
Is it to do with what your friends, colleagues or society would think (external factors) or is it more that you struggle with coming to terms of some aspect inside of you?
I hope this forum can help you find answers. Sometimes just knowing that you are not alone in your thinking can help.