It's late so I will apologize for the brevity of my thoughts in some areas LOL!
"but he's still afraid another guy will "sweep me away."
- This is a normal feeling. The importance is that it is not the dominant one, which should be happiness and love. Doesn't discount it but puts it into perspective for me.
"am I really living polyamorously by loving and being physical with another man while still loving my boyfriend"
YES!! There are different approaches to polyamory, which go beyond dictionary definitions. The key for me hinges on your emotions for the other men. In my definition of poly you should have an emotional connection with them as well..otherwise it is just sex..that approach is more like an open relationship.
"and is it still Polyamory when there is no sex with one of them?"
Boy you will get different answers to this one!
To me this question is more about why you aren't having sex. In my humble opinion physical intimacy is the defining act, which pushes a deep relationship into the realm of polyamory. If you are not having sex because external factors are holding you back then that person can still be considered a lover, so again this seems like a polyamorous connection. If you simply don't desire each other intimately than I would call it a deep friendly relationship.
Good luck, I hope you find your path in this one.