cherilips, this is raising all kinds of red flags for me.
You've been his "dirty little secret?" Why on Earth would you want to move in with someone who won't otherwise acknowledge your existence to his friends and family? If they know about this other girl, how are they going to react when he moves in with some woman they don't even know about?
Moving in together will add stresses to your life and relationship. It will get harder, not easier. If he's already not doing his part at being supportive, it's only going to get worse when you add in the stress of shared living arrangements.
I especially don't like the part where he gets angry at you for expressing reasonable concerns. Moving to a new city and starting over is a big deal. Trust me, I've done it. It was enough of a challenge when we had already been living together for a year, and my partner was understanding and supportive.
When you move, you'll be giving up your whole support network. Take it from someone who's been there, a phone call is no replacement for a hug. If what you're walking into is not supportive, you'll be in for a rough ride.
“As I am sure any cat owner will be able to tell you,
someone else putting you in a box is entirely different
from getting into a box yourself.” —bisexualbaker