Think about a situation, in which one has three resources: money, time, and efforts.
Some people have better money managing skills than others, and end up financially better off than peers with the same income. Similarly, there are people who "make time" by managing their time more efficiently then others. Lastly, the same can be said about managing one's efforts. Now, things get even more complicated when not everyone has the same starting pool of money, time, and effort/energy to spend.
Maybe J. is short in the time/effort department or thinks that he is so. In such a case, maybe he cannot "afford you" in terms of effort and time. I would also feel frustrated is someone is inviting me to her place for when everyone is asleep to "cuddle in the basement."
On the other hand, I could understand that because I spend very little time with my wife - we live apart during the week due to jobs. For this reason I would not meet people when I could spend that little time with my wife. I'd rather meet friends on "my time."
If J. can afford spending quality time with you, the question is how he better manages his time and efforts. If he is stretched thin and cannot put the time and effort in you, then you have a discuss.