I am with the majority on this. It is absolutely none of his business and if you continue to justify and defend your decisions to this person, he will just continue to invade your privacy and life. This person will likely never except you; don't let them continue to hurt you.
I have another phrase you should learn and use as needed.
"I appreciate your concern but I am an adult and this is none of your business."