I agree with jkelly in that you could tell him that you don't have romantic feelings. I think it very important to tell anyone those sort of things. It's just plain respectful to be up front.... on that note, you didn't tell him you are dating a couple and are poly? Hmmmm.... also not very up front and can lead to worlds of trouble. Total honesty is always the best policy and practice for me personally. Anyone I meet who is dateworthy needs to know everything now... in the first sentence after "hi, do you want to go out with me." Otheriwise I have to keep tabs on what I have said and when and what I need to catch them up on when something comes up. Nope, for me it has definitely been better to just tell them and be on the same page.
There is no reason to say who you are dating, just that you are. That is respectful and shows integrity I think. I would hope that you told the couple that you went on a date? They need respecting also.
I know these things are hard to do, but you are right I think, "short term pain for long term gain" is the way to go.
If it were me I would tell him that he is a lovely man and you enjoy his company and would love to hang out again, but that you don't have feelings for him that are anything other than friendship. After that I would add that you already have some people you am dating and you are satisfied with dating just them for now.
short, to the point and hopefully he will be happy with a friend