I don't think you should feel obligated to like the people your husband dates.
You need to have some seriously intense conversations with your husband to set up a 'poly agreement' so he understands the boundaries and you can feel safe. There are online articles and books that talk about this sort of thing in depth.
Now is the time to stand up and be heard. These agreements aren't concrete, they can change over time. But it's a hell of a lot easier to ease up on a strict rule than it is to try to impose a new one. Let him know that if he wants to be poly and have his marriage to you, then he has to go at YOUR pace and be willing to handle a lot of rules and boundaries at first while you get your bearings.
One of those understandings can be that he should not expect you to ever want to be romantically involved with any of his dates. Your issues with women, I understand. Its something to be worked through with a therapist, slowly, and it doesn't help at all to feel pressured.
If he does read this, then I suggest to him that he be very slow and deliberate here. Make the choice between your ability to be poly, or your marriage to her. Everything in between is a compromise, but you really should be certain which is of more value to you now.
This can end in heartbreak if not handled with maturity and careful consideration. Don't let him push you or make you feel guilty. It was brave of him to bring up poly to you, now be brave yourself in establishing your safe zone.