Go with your heart
You need to decide in your heart do you want to be with him because you care about him and her or just him. You have to decide also if the doubts you are feeling right are worth the risk of a great relationship or if you can't get over those "nagging feeling" no disrespect intended of jealously because if you can't get over them then the relationship can and I knowfrom my own experiences does go very wrong very fast. I have a polyamorous relationship with two men and I can't budge from my rule about no other women than me in my relationship because of my own jealously I can share them with each other and they are just with me female wise but its not something you will get over easily. I have been in monogamous relationships too and I just didn't feel complete but I don't try to convert but I do want to try to help anyone that wants to be in a polyamorous relationship or in your case just thinking about it. I hope I helped at least a little.