It sounds like you have some good communication going on. And it is great that you are both willing to be so open.
You're not alone. There is rarely a perfect match in sexuality between two people.
As to whether you should go out - you'll get differing opinions here. Distracting yourself from those thoughts is reasonable as long as it is not complete escapism. Both sexual and non-sexual distractions helped me through some tough nights as I learned that I could, in fact, survive (and eventually be completely ok with) my partner being out with someone else. And I learned that having a lover did not reduce or diminish my feelings for my other lover. So I could better see my partner's point of view making the transition easier for me.
And she's living with you, not these other people. She has chosen you for a reason. Have you thought about what those reasons are? Understanding the answer to that question can help you feel more secure about your relationship.
Me, Dino: 58 single straight male
Gidget: 51, single, straight, sub
Liza:42, married, straight
Celia: 52, single straight