Honestly I think you and your husband need to take a break. Almost sounds like you want to punish him because you are insecure and miserable.
I am not going to rehash what others have said better than I can.
My husband and I are at a crossroads ourselves . He like you wants to work it out. Me I am tired of dealing with it. I have spoken my wants needs and boundaries until I am blue in the face. He has never respected that and has trampled over agreements over and over. He denies doing so. Lucky for him he hasn't made the request that I take a break from my bf. My bf is the one good thing I have to look forward to. He brings me peace and is my rock. If my husband made the demand you're thinking of making my husband would be gone. I am not going to throw away 17months of happiness for someone who is a source of stress.
40 yo straight female
Married in the eyes of the government to Butch since 2001...
Murf my monogamous second husband has been with me since May of 2012.
In a V relationship with an average 60/40 split of time. Only due to Murf's and Butch's crappy work schedules.