I hate to say this, but it does not sound like your marriage is ready for poly.
The prevailing wisdom is that a relationship must be strong and healthy in order for either partner to become involved with someone else, otherwise the existing problems will get worse and not better.
You and your husband need to work things out within your OWN relationship before EITHER of you can be in another.
It sounds like swinging was working alright for the both of you. Is there some reason you can't simply go back to "just sex" with other partners?