I thought alot of the problem was his gf or wife? Not that it was him completely?
You have Plenty of people here to talk to regarding being poly. You don't NEED to keep him around as a poly friend, you WANT to keep him around to keep in contact with him. Be honest with yourself...