This makes no sense to me. "Hey I know you said your husband is okay with us seeing each other but I'm not sure so instead of asking him or talking to him for verification I'd rather we just keep this secret from everyone okay?"
Also, there are poly/mono relationships that work. However, coming from a stand point of opening our marriage after an affair and going from mono to poly/mono, I can safely say I have NOT seen it work with it's mono/poly because someone is not ALLOWED to be poly.
If he had an affair then yes, there's a lot of trust to rebuild there, and I get that! (I was the cheater and had to do it, still working on it honestly) However, saying you aren't comfortable with it NOW is different than I can't trust you so you dont' get to ever!
Some of the things that helped us with rebuilding trust were:
Transparency - yep, that means talking about everything. If one of us was uncomfortable telling the other something it was probably something we REALLY need to tell so we do. It also meant that my emails, ims, phone calls, all open to him to view and go over if he wants.
Research - going over books together and going over the check lists in some of them, discussing pitfalls others list, and situations made it easier. Oh yes, after double digit years married to suddenly be discussing safe sex in such detail was weird at first, but worth it!
Counseling - can not stress this one enough. We did much of it between the two of us but still five years later we are going to see an actual therapist to keep things running smoothly and make sure we are good on past things that hurt.
Whatever rules you two come up with for now, are for now. It doesn't have to be forever. If you can't trust him now, totally understandable. However, that's something to fix, to work on, not just say you can't and won't ever.
Me: Late 30s pansexual poly. DH: My husband of 17 yrs and father of children. DC: LDR of +2 year