Yeah, if I were so scary/upsetting to someone that they couldn't stand to meet me, I'd feel like something was really wrong about the situation. OP, I think your request is entirely reasonable, as long as you really do plan to keep the meeting cordial and appropriate (if you originally wanted an open marriage because the idea of your wife with another man got you off, and you felt the need to inform the other man of that when he might really not wanna know, that would not be cool).
I agree with Mags that, ideally, your wife should care enough about your happiness to be willing to put her foot down on this matter. If she won't, it doesn't bode well for when bigger things may come up. She can find another guy to crush on and make out with if this guy isn't willing to be even slightly accommodating for your comfort.
The major players. Me, 30ish bi female. Gia, girlfriend of 4+ years. Clay, boyfriend/dom. Davis, ex/friend/"it's complicated." Eddie, roommate & fwb.
The supporting cast. Eric, Gia's husband. Bee, Gia and Eric's toddler. Dexter, Gia's lover. Helen, Eric's lover. Izzy and Nikki, Clay's partners. Liam, Eddie's husband.