I'm sorry things are not working the way you like in this relationship. It's frustrating to be with someone whom you don't feel comfortable expressing yourself. Quite often this is due to the other party being defensive (like this guy) and not validating your feelings. I go through this all the time with my W (although it's getting better with therapy). It's frustrating to feel you have to constantly defend your feelings and have your partner try to convince you your feelings are somehow invalid. Feelings just are, and if your partner does not validate them, eventually you stop sharing. Then communication goes out the window, leading to misunderstanding and hurt. And then you don't have a snowball's chance in hell of having your wants understood and met.
There's a wonderful book out there called "The Seven Principles for Making a Marriage Work" by John Gottman. It contains wonderful advice on communicating in an effective way. The advice is applicable to partners in any relationship. I suggest reading it. It really is eye-opening reading about the mistakes people make when trying to discuss something important.
Eventually you're going to have to make your wants and needs known in a manner that suggests the status quo is not working for you. It's entirely appropriate for you to want some alone time with him that is not shared. If you're really his "primary", he'll understand this. Otherwise he's just being a bully who insists on getting his way with everything. Maybe suggest a compromise, like every other weekend you get him all to yourself.