That sounds very similar to things with me, Karma and his g/f. I usualy let him show the affection. When we go out as a group we do tend to all hold hands, and when he kisses one of us he then kisses the other. But I do tend to pull back a bit here at home.
I do a lot of the same thing, we usualy have dinner together, hang out for a bit, then I disapear with a book or whatever to leave them alone. I partialy feel that if I show him affection when she's with us, I am taking away from their time together. Or like I am trying to show my dominance over the situation. That's not what I want, so I just slip away.
I don't have any words of advice, but I hope things smooth out for you.