I do a similar thing, OP! My, um, sex friend and I are involved in a club that my husband has little interest in. I think I was too obvious in group meetings, looking dreamily at my friend's gorgeous face and flirting a little too much, and often trying to stay late if the meetings were at his house.
Some co-workers were involved in the group and I started some rumours. Some completely true rumours. Oops.
It's hard, isn't it, when you don't really know how open to be? Let us know what you decide.