If you are willing to have him break agreements, then you are moving from poly to cheating. If he is willing to break agreements with her, he will do the same to you. There is no guarantee that the person he breaks the agreements with next time will be safe. However, there will be a next time because by allowing it you tell him it is ok to do. That being said, he is using her as a crutch instead of owning his agreements. If he doesn't like the agreements, it is up to him to renegotiate not you. The fact that he has not renegotiated but comes back with yet another excuse blaming her, so that you don't see that he is the real problem, tells me quite clearly that he doesn't perform that well even without condoms. Unless you sit and watch him with his wife, you will never know the truth of their sex life and you shouldn't.
Your relationship is with him, not her. You are dating someone with ED. You aren't going to have porn star sex. Get used to it. He needs to own it. Right now, it is less embarrassing for him to blame his wife than admit that his problem is not completely cured by the wonders of medicine.