This situation happens often enough that it's considered stereotypical in some places: husband is enthusiastic about poly and wife is uncertain...then wife finds a boyfriend and husband freaks...husband then wants to return to monogamy while wife is now happily poly.
As for advice, I think your husband is going to need some time to grieve the loss of the relationship with his gf, so stepping gingerly for a bit seems appropriate. Beyond that, if you're happily doing poly and don't wish to stop, then that isn't much of an option at this point--he's going to have to adjust to you having other relationships whether he has any or not. He doesn't get to make that choice--that's your choice.
The idea that he would ask you to drop the bf *and not be bothered by that* is mindboggling. My reaction to that certainly isn't fit for a family publication (and I'll refrain from airing it verbatim here). That is wrong in so many ways.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.