It actually sounds like you both handled that bump in the road very well.
Negative, uncomfortable feelings happen-to everyone and OFTEN when we try new things.
I suggest going with "not at our place" FOR NOW. No need to rush.
Also suggest that you reach out to him. Invite him to lunch, golf, coffee, a beer. Somewhere you feel comfortable and confident doing something you enjoy.
Don't try to push a "where is this going talk". In fact-tell him, You dont want to get into that, you just want to get to know him. Let him know (briefly) that this is a new dynamic for you, you expect to have some "2 step forward, 1 step back" moments but you look forward to the challenge and hope he can be understanding of the complicated course you all are traversing.
Then-ask about his job or talk about sports, shoot a game of pool. Whatever.
Its so much easier to deal with our partner loving someone-when we like that person. Its much harder if we dont know them-we imagine the worst.
Get to know him.
As to your wife; let her know you love her and are committed to working through the hiccups for this. BUT, you need some extra tlc from her because this is more difficult than you imagined. Explain that your med changes are likely exacerbating the situation (I can SO understand that one personally). Let her know that for right now you need to limit the amount of physical affection she shares with him in front of you.
Also-something I find helpful is having my partner NOT reduce the amount of physical affection they normally give me just because a new partner is there. So for example, I get my kisses, hugs, cuddles on the couch, hand holding.
Both of my lovers live with me; they have a... Tentative friendship and how we got here was a mess. My husband is the one who is most insecure. I try to support his security, so, I am making dinner, I make a point of reaching out to him, being physically affectionate. My boyfriend understands, I love them both. There's no expectation that he can expect me to minimize my affectionate behavior with husband while he is present.
"Love As Thou Wilt"