Thanks for sharing your story. In regards to your first question, your feelings are never petty. We are human and are entitled to feel whatever it is that we feel. You said that you and your SO are great at communicating and that's a huge step in the right direction. You said that you and your SO have talked about HOBFs visit in small doses. Any particular reason?
As far as your second question goes, based off of what I've read, I'd say that HOBF is the one with the issues and concerns that need to be addressed. Of course that's without actually being there. I do commend you for trying to figure out if you're the "red flag" where as most guys would immediately assume it's the other man. That in itself gives me some perspective about your character.
My only piece of advice has already been stated. In my experience there isn't anything that can't be worked out through good, HONEST, communication. Heavy emphasis on the honest part.
Live life to the fullest 'cause you never know if you're gonna wake up tomorrow!