I'm not sure humans have the capacity to easily accept the notion of a "limited sex life", especially when surrounded by others who are being frequently satisfied, without severe psychological (and probably even hormonal) damage.
The first time I was with another woman after I married my wife (she knew about it, and gave me the "go-ahead"), I felt terribly cheap and dirty afterwards. I was harboring the notion that it was wrong and that I was hurting her by doing it. I may have hurt her by doing it, which didn't help things. I dealt with guilt and self-anger for quite awhile after that. Now, I don't feel cheap, I don't feel like I'm hurting her, and I don't feel like I'm doing something wrong. We're open, honest, caring, and supportive of each other and what we do, but it wasn't an instant transition. That being said, the change in my own mindset about it was what enabled me to really enjoy sex with another person, even have feelings for her.
I don't know if changing your mindset is going to work for you, but you have needs, and they're not being met. I'd suggest finding a way to meet those needs, rather than trying to find a way to not have the needs in the first place, because over the last 3,000 or so years of recorded history, we seem to have very little luck at that as a species.