Lots of good advice and viewpoints here.
One thing I'm wondering is J's motivation. I'm in a similar position - I'm in the UK and my girlfriend is in the US. I'm wondering if I can empathise with J in the sense that if I was lucky enough to be moving to the US for good... there'd be part of me that would want a casual, last European fling if I met someone really interesting. Do you think it's possible that because he feels there won't be any long term potential for the two of them, he's a bit more tempted to ignore certain boundaries to get what he wants?
Incidentally, my GF, her husband and I tell everyone that about our involvements before the first date even happens. If they are not OK with it, we don't go on that first date. We also expect our secondary partners to meet our primaries before sex happens (chat with them online when I'm in the UK). If a secondary partner starts to become a bit disrespectful of our primary relationships, or starts to cause trouble, that's usually a warning sign for us, too.
That being said, my GF's husband has terrible luck with women. And they all run for the hills when he tells them he's poly. It's gotten so bad for him, he's so saddened and frustrated because my GF has huge success with men. So she recently suggested that if he wanted to have a quick fling, he could leave out the poly part for the time being.
So, I think it really comes down to what's right for the two of you.
I think it's great that you don't want to restrict him - but you're not a robot. Sometimes it's not jealousy, but something else we feel... disrespect. That feeling can come out of nowhere and surprise us.
If you're poly, rather than single, it means that you owe it to each other to be considerate and still work as a partnership, instead of completely unattached people.
I would recommend that you do talk about certain guidelines for the future. Guidelines are just there to keep things steady; they can always be adjusted for certain situations.
I would definitely say that you should have a think about your own expectations and what is ok for you in this relationship. If J doesn't know something's bothering you, he's likely to just carry on as he is.
Oh, finally - no... I don't think you're being selfish in the slightest.
Good luck with everything and congratulations on him moving... I'm very jealous!!