Thank you all for the feedback.
To GirlInGray - Sweetie does know he isn't the cause just a symptom of the issues between me & Hubby. As I said, I sometimes ask Sweetie for a male POV and I usually need to give him some sort of background information. Thus he knows examples of problems causing our issues dating from well before he & I met.
Yes we both want more of a physical relationship, but we do have strong spiritual and emotional and mental relationships. I've asked Sweetie if he wants to continue or if he wants out - he's told me that he wants to continue our relationship.
I know that Hubby and I need to sit down and set a timetable and milestones. I'm not sure if I want to stay with him if he isn't willing.
To Aphrodite - I still love Hubby because he does have good characteristics too. I know that he will be there for me if I really, really need him to be. He is also making the most effort ever to give me some of the little physical things I need. Unfortunately, it took Sweetie coming into the picture for him to do so. And we have 23 years of being together, with at least 1/3 of those years being very happy.
To kdt2617 - I think a lot of it is emotions. I never really planned to get involved with someone else. I also think that part of the problem could be because our agreement was fairly vague. He interpreted to mean onetime only encounters, and I interpreted it to include FWB/secondary relationships. Another part of the problem may be because Hubby is 60 and Sweetie will be 52 in a couple weeks. Also the fact that both Sweetie and I are Pagans, while Hubby identifies as Catholic may make hubby insecure also.
You're right, I do need to probe deeper, it's just difficult because he keeps getting defensive and that basically ends the conversation.
To MrFarFromRight - you gave me a lot to think about and respond to. This is just to let you know I didn't forget you, but I am going to have to come back later to respond as I have some things I need to go do.
Thanks again to everyone. It is nice to have a place I can come to and get help.