I just got dumped by a boyfriend that considered me his secondary so my thoughts on it might be skewed
you can read my blog for details. I would wonder what his wife thinks of you having more.
I had a once a month date for three years and no sex. To me you have it good. Mind you I have other partners too so the time thing was an issue. Really though, it does work and that new relationship energy can last and last the way you are going right now. It can drive you crazy for more or keep you energized if you can ride the wave.
I would breath, smile and enjoy every moment for now. If more comes along then great, but really, to me it sounds like just enough if you have a family and other partner to think about.