I get wanting to please your partners only too well. But I wonder... what do you want? I feel that both of your partners feel like they are entitled to something from you (stricter boundaries or swinging). But they aren't pushing the issue either way, so I think that the bigger problem is that you believe in that entitlement. You think that they have a right to asking you these things (which is true) and that you have a duty to comply (which you don't).
I have struggled with guilt, and continue with that. It's a process. But you know, it's no different from any other emotion. You don't have to follow it. Just because you feel guilty doesn't mean you did something wrong. It is beneficial to try and look at the situation objectively to see whether there is actually something you should feel guilty about, or if it's coming from other places (like from something inside you, or from somebody deliberately trying to make you feel guilty).