A lot of people think that if you love someone, you can work through anything. Granted, I think love gives one the motivation to do the hard work, to try a lot of alternatives while looking for solutions. But the truth of the matter is, you can love somebody deeply and not be compatible on a day to day basis.
In your situation, you are mono, and love a poly man - a poly man who appears to be uncomfortable limiting his interactions, despite what he tells you. Does this make him a bad person? No. Does being mono make you a bad person? No. But it does make how you want to conduct your lives and relationships fundamentally different.
If you chose to leave the situation and pursue the configuration you desire, does that mean you don't love him? No, you still will. But it might enable you to stop begrudging who he is. I can assure you has no desire to hurt you, but the hurt you feel is probably one he doesn't fully understand. And just hypothesizing, you probably don't understand the hurts you inflict on him by not accepting who he is.