You've got a lot to offer!
I'll only touch on a couple of your points.
"But part of me feels sad for him that it can't happen.
Ah, compersion. Most mono's can't even comprehend that. You're ahead of the game there!
"I'm afraid he will find all these women interested in him, and nobody will be interested in me."
We've met, and no offense to him, even if I wasn't able to appreciate the feminine form, I would still be immensely more interested in spending time with you than him. In all honesty, I know do know some swingers who quit swinging because one partner got more attention, so it is always a risk. But sweetie, you have a lot to offer a potential partner: honesty, intelligence, curiosity, great figure, and I'm sure a heck of a lot more I don't even begin to know about.
As long as your relationship with your DH satisfies you both in enough ways for y'all to stay together, everything doesn't need to be "equal", especially not in number of partners. Sometimes quality of the relationship(s) one may have more than offsets any quantity issues.