I was inspired to respond to you post.
"Those who matter won't mind; those who mind won't matter."
My wife and I are in a very similar situation where we are the "secret" we support, we are there all the time. When she needs a break to sort things out, to plan to focus for months we were understanding. It was not like we wanted to announce to the world our love and adoration however in a recent conversation, I said " I dont know one of your friends. You know my closest one, you have helped take down the Christmas light in front of our house. Our neighbors know your name. Our kids do not know you, but they know we talk to you every night, and are the topic of many conversations." That imbalance says to me. You do not validate our relationship in the way we want. We have a choice. Stay on the hamster wheel, or step off. At least this time. I am happy my children did not get to meet them to realize the disappointment of someone stepping in and stepping out for what ever reason.
There is much more to the story, I always shy away from telling the complete picture or offering only one side, for in this life, there are so many views, so many delicate intricacies of so many peoples lives in this it is only one view.. that one view however is my own.