Yes, I've felt that way. I spent a lot of time worrying about it and trying to get it to make sense to me. After we became polyamorous, I actually spent months and months mulling over the idea that sharing sex with ONLY your primary partner is what made the relationship special--yes, I know that sex isn't really what made the relationship special but bear with me-- and if you have sex with other partners, what is the difference between your primary and anyone else?
Okay, the bad news is that I haven't been able to find a clear-cut, easy answer to that. Sorry, I know it should be easier but I just can't make it tip over the edge into understanding yet.
The good news is, I haven't lost feelings for my husband by "allowing" him to have sex with our girlfriend. (baby steps...) "Losing ownership" of his physical self was hard at first, but I can tell you that as I work through my issues, I think that I don't actually feel like I've lost ownership. I feel like everything I love about him is still there, and our physical intimacy is still loving and an expression of the bond we feel. I *am* insecure sometimes, but I feel connected to him, not at all like my platonic relationships. We still have the foundation of our relationship. (I hope that was more helpful than confusing.)