I'm going to speak definitively, and assumptive here, I apologize
First off I want to say my hats off to you sir. Your respect, grace, examples of loving concepts, and understanding are commendable, and you deserve all the praise in the world for your high amounts of virtues. I've only met a few people who displayed the same amounts of wisdom in their relationships and they are far too few in my opinion.
2nd, this thread is super graceful, and I apologize if I ruin that grace...however, with the information that has been presented(all thx to your thorough sense of writing) I feel the danger level of the flags presented are !critical! level and are being underrated.
Red pepper sort of the beat me to the punch but i'll further add to her advice. Simply put, I believe fully, that for you and your SO to be happy and have what you want; the HOBF needs to move out there, transfer his job or attempt to get a new one; get a low-rent apartment that's somewhere close to you guys and make a commitment to his girlfriend(your SO), and to creating a co-existence between their relationship, and her relationship with you(all being relative to his perspective). That's the only way I see this playing out, from a futuristic perspective. Because, IMO, they're relationship can't last forever in "visits". Eventually they're going to need to be closer to each other. Your g/f already knows this and wants it.
I have a few assessment from the facts you've given.
1. Investigation needs to be put forth towards the true nature, and character of HOBF. I believe that he isn't simply "non-poly", but more like a poly disaster waiting to happen. Some basis for my conclusion comes from a blind personality profile. 1. He's happy with what he gets. This is telling a lot about the kind of person he is. He possibly hasn't really figured out what he really wants in life, or hasn't fully developed his true self and feels inclined to not think about what he wants. He's possibly depressed; either chemically, interpersonally, or both. A lot of ideas pop up. 2. She's been dating him and has known him longer than you, however he lives in a different state now. So either she moved, or he moved. I'm going to assume that she moved, given his "comfortability" to his location. This is also telling of the kind of person he is and adds to macro-reasons why he has problem communicating. He isn't matured, he's generally a fearful person, the list goes on...
2. Regardless of my assumptions, and including the information you've provided, it is my opinion that the majority of work is going to have to be with your SO and HOBF. He is the weakest link, simply put. As stated before, you are in a poly relationship, him included...Despite if he's poly or not. So basically, he needs to be...educated. It's really that simple. It would be different if your relationship with SO was different, but it seems you guys are in a committed partnership, and that's not going away(yay!).
For everything to work out in the future, HOBF needs to educated about communication, the important of compersion for his girlfriends' relationship, how his life is going to change, why you aren't going away, why it's important that he needs to figure out if he wants this relationship dynamic, and further concepts that are involved in your poly relationship. That's the skinny...
I wish all the luck in the world my friend. I believe if you keep your strength, and your SO really works with you on this, you can guys can have something beautiful. THANKS FOR SHARING YOUR STORY!
peace and love...
-"There hasn't been a person i've been with that I didn't love for 10 seconds to 10 years." David Duchovny