"can't know"...what? That you know each other? That you're friends (good friends, close friends, whatever)? Or that you fuck each other?
Because the first two should be fine for others to know. I don't hide the existence of TGIB from my life. But only a few selected people know there's anything going on beyond "best friends", and frankly it's not really anyone else's business (besides my other partner(s), of course).
That said, my response probably would have been to look at him like he was crazy and say, "Well, DUH!!" or some other sarcastic response. "No shit, Sherlock" and "Gee, REALLY, cause I was gonna invite him over for a foursome" with my most withering look come to mind. Probably not helpful, sorry. I'd say the three of you probably need to sit down and get really clear on who can know what.
Pan Female, Hinge in a V between my mono (straight) husband, Monochrome and my poly (pan) partner, ThatGuyInBlack