This is a difficult one, as I can see some potential problems in the future. Your husband's beliefs and desires for the future could very well effect the way he interacts with other partners, the conversations he has, the promises he makes, etc.
If your husband wants to be poly, WITH YOU, then he has to stick to agreements that you make together. If he doesn't want to stick to a particular agreement, it's HIS responsibility to communicate his needs and not accept an agreement until you both reach a satisfying compromise point.
Ultimately, as other people have said... everyone has their own limits.
If this helps, here's our general situation, limits, etc:
Myself, girlfriend and her husband in V, with three year old toddler.
Live together as family unit.
Believe in hierarchical poly / it suits us (primary/secondary).
Hubby - has welcomed me, but doesn't want any extra men moving in.
GF - wants our family unit, but wouldn't be opposed to extras moving in.
Me - want our family unit and not thrilled about anyone else moving in.
We are all agreed that secondaries do not meet our child, but meet all of us. We don't invite them to family events. My girlfriend also isn't thrilled about my secondaries meeting my friends, because she feels it is too 'relationshippy'.
So that's us.
You are not the bad guy for your opinions. You get to decide what you are comfortable with and what you want from life. Your husband isn't any better or more evolved than you for wanting to invite her. So, the first step, in my opinion, is to set some ground rules and guidelines together. What are yours like so far? Are they verbal agreements? Do you have a written list? Nothing but a short list of hard limits and non-negotiables? Or just take each situation as it comes?
We have a written list of about 9 guidelines that really help us not to walk all over each other when we are clouded with NRE. We read over our guidelines every 6 months to a year to see if everything is still relevant. Guidelines help us to avoid putting each other in a 'bad guy' situation; because everyone knows what the guidelines are. When we break one (we are human) we fess up to it and take responsibility... and our partner forgives us and understands.
We also use a numbers technique when we are dealing with a given situation. 0 means totally cool, no stress. 5 means a possible relationship deal breaker. 4 is very, very uncomfortable but not a breaking-up offence.
For example, I went out one night with a girl I'd just started dating. I was supposed to go home before 11pm, as she had to catch a last train home. While we were out, we were having a great time and wanted to go out dancing. This would mean her catching a cab home at about 3am, that she was happy to pay for. I text my girlfriend and asked if it would be ok - she said it was totally fine and to have a great time. The next morning, she told me that she was really angry and upset about being put in that situation and that I'd broken a guideline we have.
If she'd have told me that this was a level 4 or 5 stressor for her, I would never have stayed out. Even a level 3 would have made me reconsider. Her comfort is more important than my need to dance. Basically, to me, it's about giving your partner all the information and letting them make their decision based on that.
I don't believe that the primary couple makes the decision about everything; but that they make the decision on THEIR relationship. If I enter a relationship as someone's secondary, I follow the guidelines of their coupledom. If they have a guideline against me being in their family home, I'd never even bring it up, let alone suggest it or push it. Does that make me feel less important? No. I'm not less of a person - I made the decision to be a secondary to someone who has a different primary relationship. If I want to go to a partner's house, then I can make a decision to pick a partner who has that option available.
In short... how about looking at your guidelines, soft limits and hard limits? Re-negotiating existing agreements. If you can't compromise, can you carry on together? If you can compromise, will you both really stick to it? (i.e. if husband commits to a no-secondaries-in-the-house agreement... will he put this across to new partners and seek appropriate playmates?)
Good luck and please let us know how your discussion goes!
EDIT - seeing your last post as I was writing mine!
Not surprised you were upset about hubby placing the blame on you. If he's going to agree to something, he should treat it as HIS decision, or not agree. Perfectly reasonable for you to request he shows a solid front.
His communication issues with hers are essentially his thing to work out. If he can't meet your basic requirements, then it is worth considering whether your relationship is effective for you.
I think it's good to just let them be and not intervene. He has to learn to communicate by himself. I wouldn't get involve with my gf's secondaries in that sense, unless they specifically came to me wanting to talk, or wanting to understand something better. If my GF says something to a secondary that I disagree with (i.e. coming across like she's blaming me, like she's bad-mouthing me, or downplaying our relationship, or whatever else), I tell her how I feel and expect her to make the changes. If she doesn't want to, she's not the one for me; because I want a partner who acts on behalf of us as a couple.
You don't have to feel horrible about what you want from your relationship with your husband. Forget about 'normal' rules and relationships. You've now done the difficult thing and got your feelings out on the table... it's up to him to decide whether or not this fulfills him.
I'm sorry that guys disappear when they discover that you have CP - but I'm glad that you are exploring the complexities of sexuality!