Is he having trouble with the move from swinging to relationship? Maybe the change scares him.
Maybe he thought casual sex was/is ok but developing feelings wasn't what he thought it would be and is now wishing it could be back to swinging again.
Can you put your bf on hold or leave things as they are or even step back a bit until your hubby can work through his issue? I know its not what you want to do but if its what you need to do are you willing to do it?