I can totally relate. My husband and I have been poly since early this year. I found a partner I really clicked with quite quickly and have been seeing him for six months. I had similar fears, that he'd suddenly say, 'Im not getting anywhere so I want you to stop seeing him.' It never happened like that. I had to very careful of my DHs feelings and my relationship with my BF didn't move like Id have liked.
Now DH has just started seeing someone, and he's much happier with my BF and how things are. It just seems to be harder for men for some reason.
Hang in there, believe him when he says he's fine. I didn't at first, and it was that disbelief that caused problems. Good luck.