Its difficult for me not to think about sex right now in particular because my wife is in Spokane and I am in Santa Barbara for the next six months. This was a way for us to begin our new lives out of college. I just started a career down in SB and she is finishing her career start their before moving here.
Your partner seems caught up on the sexual part of it as well if he is interested in you brining women, but not interested in men. I am thinking, I need to step away from sex completely and look at seeing another person intellectually. If that feels good I will consider other things.
The largest factor in my identifying as poly is my lack of jealousy. If things were flipped around, and my wife wanted a more open relationship, I wouldn't be afraid of losing her, and would be happy that the other person/people made her happy.