I got to this thread late and lots of good advice already.
Suffice to say, this isn't Poly. This is part Dirty Old Man Syndrome, part Asshole-itis, and you need to free yourself from this creep as fast as possible.
It is NOT ethical nonmonogamy when someone pushes another person around to accept a multiple-person arrangement. And then the fact that he would say no if you found another partner you like just says he's a plain Hypocrite, period.
This dude just sounds like bad news, plain and simple.