I did some with my first husband, he had a high sex drive and loved sex. Also his first girlfriend was interested in me too, so although we didn't date it certainly was something to fantasize about.
My current husband has a low libido so I don't at all because it's unrealistic, I can't for instance fantasize about MFM threesomes with him because I know IRL it wouldn't be hot and sexy, it's just be awkward. I think I fantasized all sortsa stuff with him and/or some of the cute chicks he dated early on before I realized our sex drives were so different and his relationships with them weren't based on hot carnal sexytimes.
I don't fantasize about my bf and his partners, as I haven't spent much time with them and I am pretty sure at least one of them wouldn't appreciate it.
Really I keep my fantasies in my head - I think it's kinda invasive, the idea of one of my partners and their partner verbally fantasizing about me to each other. Conversely that's why I keep my fantasies to myself, I think unless a metamour invites a flirty relationship with me, they didn't invite me to talk dirty about them. I know lots of people don't have this feeling, but I can't shake it. Doesn't stop me from having whatever thoughts I want to in my head though!
Happiness will never come to those who fail to appreciate what they already have.