Based on what you've written only, I suspect you see the situation pretty clearly. There's a variety of reason he may not be that sexual with you and there's no need to get into them. It doesn't matter - it is what it is.
What seems important in this is that he's being respectful to your needs & desires and it behooves you to do the same for him. By this I mean that if you choose to take other lovers, you WILL need to be discreet as it's indicated from your writing that his concerns are for the social stigma - and likely the children.
So you have to be careful and smart. Respectful.
I think if you do this, all will likely be well.