I think, based on what I've read, that you and your lover need to ask each other point blank "What do YOU want to gain out of a poly relationship?"
If honestly, all he desires is the sexual gratification of seeing you with another woman, then I don't think poly is really for you and him.
Though I'm very new to this, I have the understanding that poly-relationships ARE relationships. That's different from a one or two time sexual fling.
And it probably is going to be hurtful to any woman you and he approach offering a relationship if all that is really wanted is a few nights of fun.
So... make sure you know if you want a RELATIONSHIP or a FLING. Because otherwise people's emotions might get trampled on; you're and his as well as the 'unicorn's'.
All that being stated, I do wish you both luck in your journey of discovery as a couple.