Sounds like you are not eye-to-eye on the relationship model. He wants to be with you with a side of "open" or "swing". Like a 1A in this article.
You want something else, and don't care for his casual encounters.
There's also the OTHER problem of his conflict resolution style. He attacks the person (you, you must not be cut out for poly, you are deficient somehow, uses things against you) rather than attacking the problem (not agreeing on relationship model, the need to get on the same page, etc)
That's not kind -- making it be a personal attack on you.
At 5 months in, it is not a HUGE relationship investment for you. Painful as it may be -- perhaps spend some time thinking if the return is worth your investment here? And if you could be better off ending it with him? Because you are not compatible and do not see eye to eye on HOW to be Open and in what framework? Then you are free to polyship with a more compatible dating partner?