I also think he has been clear, but that doesn't mean you know what will happen in the future. I believe you run the risk of divorce if he ever finds out, and if it goes on very long, that is likely. He says that you will remain together until the children are grown. He could change his mind along the way. I think his answers are ambiguous at best. Be careful, especially if the divorce laws in your locality could make it easy for him to gain custody of your children.
You are not in an envious position by any stretch of the imagination.