We've run into the same thing. I think every swinging and polyamorous couple experiences this. If not, then my question is, dude what's your secret?
It is very easy for my wife to met men since there is so many out there looking for casual relationships and one night stands. As for me, I've had very little luck. This usually leaves me feeling a bit more insecure than jealous, but it is still a feeling that I have difficulty transcending. She has gone on hiatus for the last couple months until I can get something going. It's been tough for her having to say no and her patience is wearing thin.
The main thing is to be very open with your spouse. Make sure he knows when your planning to meet or get physically involved with someone. Make sure he's comfortable with it. It may be tempting to hide stuff to avoid confrontations, but that is not what this is about.