An agreement is when two people accept the terms. Now you want the terms to change. It is time for renegotiation, NOT asking for permission. Your husband is not the boss of you, he is your partner.
So, I would stop approaching it as waiting for him to change his mind and grant you permission -- you are giving him too much power over you that way. Approach it as an equal partner looking for a compromise, "This isn't working for me anymore and I am unsatisfied with the boundary we set up just to make you comfortable. I am asking that you push a little past your comfort zone and discuss this with me so we can reach a compromise in the middle where we both are happy, not just you. Let's start talking about your fears surrounding any involvement I might have with a man. Why does it scare you?"
It would behoove you both to get to the nitty-gritty of this with him. It isn't enough to simply accept his "I don't want that." Find out why, and what he feels threatened by, and address it lovingly - but firmly.