Since the kids (and all the responsibilities that come with them) belong to both parents, one of them keeping a relationship, in part, as an escape from parenting isn't appropriate. The OP isn't dating others so it will always fall on her to pick up the slack. It can end up feeling like her spouse stays married to have a free babysitter on date night. He will have to make sure to afford the OP the amount of time and attention the OP needs to not fall into that feeling because if what you suggest IS what's going on, the OP has every reason to be troubled. It means she has agreed not only to share her spouse with someone else but to also be a single parent when parenting pinches her spouse's style.
I said nothing of parenting. One of her complaints was about being interrupted during sexual activities. Now if he is doing what you're saying, and I didn't get that from the OP but I may be wrong, then I agree with you completely.