No one has the right to tell someone else what they can or can't do. I don't think any of us here are arguing that.
StrayKitten, it's up to you to figure out how you feel about this. You know how your man feels about it. Now you need to figure out if it's something you truly want. If it's not, then go another route. Swinging, open relationship, emotionless sex basically... or monogamy... or a thousand of the other things that are out there. But if you decide you want full-on polyamory with the emotions and the whole package, you need to make sure that he is okay with that. Just as it's your right to make a decision one way or another, it's his right to choose whether or not to live with that choice. You may lose him. That's okay. You may not. That's okay too.
Just, don't bend to his will in spite of yours. That's what nycindie was getting at. As abrasive as it may have seemed, it's still true.
Sounds to me like you may not be polyamorous, so this is a non-issue.
"Regret for the things we did can be tempered by time; it is the regret for the things we did not do that is inconsolable." - Sydney Smith
Kyle: 27 year old male
Katie (rymmare): 25 year old female
Kids: girl: 5 years old, boy: 3 years old