Wow! I'm sorry I don't have any answers, but I do truly feel your pain. I'm one year into a similar relationship.
We have not had the "its okay to fall in love" conversation but the "love" word is used quite liberally and its pretty clear that all 4 of us are in this relationship very deeply even though that isn't what we originally intended (started as swingers too). "Love" is a scary word, but we would all at least easily admit we love each other as the best, closest friends we could possibly have.
My husband and I have been together since high school and married 25 years. I've never had a "love" relationship come to an end and sometimes find myself really scared of how much I fear it will hurt if something breaks up this quad relationship. What you are describing is my greatest fear...all it would take is one of the four of us to decide we want out and that will leave three of us deeply hurt. Then what?
I'm sorry I don't have any suggestions beyond those you already have in this thread. Please do keep us updated, though.