It's never a good sign when one partner absolutely refuses to deal with something that is very important to the other(s). I suspect he shuts down because he's frightened by all the bad things that he imagines will happen.
I suggest writing a letter about it. Make it clear at the beginning of the letter how important the matter is to you. Then lay out your thoughts and feelings. He can read it bit by bit and work through it at his own speed.
If it's important to you, then it's something that needs to be addressed to see if the two of you can come to terms. Him refusing to address the issue is a refusal to allow you an equal role in the relationship. If he believes your needs and desires are as important as his, he'll listen and discuss the issue in some fashion, even if it's in faltering, stop-n-start fashion.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.