The only advice I can offer is that of an inexperienced polyamorous person. It sounds like you and your wife need to explore new options with other people? Maybe your wife can become a leg in a v relationship with someone else? If you enjoy to spend time with her and love her as a friend, then isn't it nice to have her as a roommate? As for your OSO, she sounds like another friend.
If you want to be with other woman, you should think about bringing another person into your family or to see other women outside (maybe multiple to keep the sex interesting). As long as you are open and communicating with all party's involved... all should be well.
From my understanding of a large polyamorous family, there is a lot of emotions and different opinions and wants... so I imagine its hard to work something out that makes everyone happy.
Like I said before... I am very inexperienced and have never had a poly family before. I am just writing how I feel with my limited knowledge on the subject. All I know.... is if you love hard and laugh often.. all will be ok.