I feel very similar to you but our kids have left home now. We were never sure if our fantasy should stay just that and whether the reality would cause problems. We now have a sexual relationship with a good friend of mine who is on her own. We 'get together' about once a month. All three of us are amazed that it has exceeded every expectation.
The only down side is for her, in that as she is on her own she misses us, but we all talk openly about this and stopped for a while as she was finding it emotionally difficult. I have found my husband and I have become even closer. However I feel this is because we have a 18 year loving, monogamous foundation.