I don't think it's ridiculous to not want to formula feed your infant. Quite the opposite. But is he being seen by a healthcare professional who monitors his weight? If he is active, alert, having wet and dirty diapers and waking for feeds, he is probably fine.
The only thing you can do is continue to protect your boundaries and health as well as your babies health. Your partner has to manage his responsibilities better and maintain his relationships. You can't force him to take steps to protect your relationship and sooner or later, if he doesn't change, you'll have to decide if you can live with the deal you're getting. If you can't, you'll have to break up & be active co parents