There's a couple of confusing statements. He's excited by the thought of you being with other men but doesn't want to know. You're fairly sure that upon learning of this he will ....a. understand and b. will exercise discretion and keep this with in the three of you.
So with that being the case what's the risk for him ( bf) if you decide to do whats right for your marriage?
In effect if this continues and comes out down the road your response to your husband will be ......I wanted to come clean sooner but ( bf ) relationship ...his life is more important then ours. That's the possible backlash. You might want to get the "not wanting to know " more rigidly nailed down....and get that on the record. Because making sure worst case scenario's don't happen in your bf life might have really bad consequences in your own marriage.
I see this type of logical having a compounding effect.