Ugh, your situation sucks.
I dated a guy for four months. We had known each other for years prior to me asking his wife if I could borrow him for a night or two. They have an open marriage and I didn't want anything more than some mild companionship having been through hell in a relationship about a year earlier.
Well, it turns out that we were really good together in that way where he all of a sudden was saying things like, "If I had met you first...not that I'd want something bad to happen to my wife...."
It didn't take long for the wifey - who usually was away with anyone but him more than she was ever home - to pick up on this and start reclaiming him. I begged and pleaded for the three of us to sit down and negotiate some sort of peace. I wasn't looking to steal him away from her or his other girlfriend, but rather discuss issues and work through some of the more difficult spots.
Her idea was for her to watch us having sex. Ugh. No. Or to suggest that I read The Ethical Slut.
Ultimately after a lot of drama, she vetoed the whole relationship and even put an end to our friendship. So that was neat.
Move on. Move into a different kind of relationship or find a set of equals if poly is the hook you want to hang your hat on. Sure it sucks now and it will always suck a little bit, but when someone chooses to either let someone else do their thinking for them or to not consider a wider array of options, nothing can be done.
The guy and I still e-mail back and forth every now and then. He's living his life where he is and the nightmare that that relationship turned into actually informed me of how to ask for and negotiate a relationship with my men when it seemed like becoming a triad was the only way to save our lives together. So, in that sense, it wasn't a total waste.
Good luck and I'm sorry. If you want a good book to read, pick up Nick Hornsby's "A long way down." You can fix everything in time.