No one can make another person happy. So, you can't ever expect that you can make your husband happy "with just you" or otherwise.
Unfortunately, it sounds like you are in a difficult situation, because the man you want to build a monogamous life with (your husband) doesn't want to do that.
I can't say I know how you feel-because I'm the one who put my husband in the position you are in.
BUT, I can say-without question, that this is your choice.
You have to choose to accept him for who he is, as he is right now-or not.
There's no right or wrong answer, it's just a matter of reality.
You can't change him. Only he can.
He may never change, he may never want to, this isn't something you have a say-so in.
You can choose to stay-but doing so means accepting that he is going to have a "more than just sex" relationship with someone else.
You can choose not to stay and that means accepting the consequences of a broken marriage.
Both routes are going to require you to buck up and deal with some painful and frustrating consequences.
That's just part of life.
I would caution, for future reference;
no matter what ANYONE claims-a person can not ever guarantee that they will not become emotionally attached to another person.
They CAN guarantee how they will choose to act upon those emotions (or not act as the case may be), but emotions come and go at their own will. Sometimes in complete defiance of our personal preferences and often at the will of hormonal changes within us (yes men and women both have hormonal changes).
So, in the future, understand, that you can't expect that such agreements are viable or trustworthy-becuase they are not and never will be.