Agreed -the lack of communication could be anything. My bf did that the first time we got really sexual, which was several months in. Turned out he was just freaking out a little and couldn't put it into words. (Now he knows that I at least need to hear "I can't put my thoughts into words right now but I'm thinking of you.")
Also, I don't think polyamory is to blame. This kind of thing happens over and over in the mono dating scene. People only wanting sex, and people not treating each other well. Hang in there. Try again. Go slowly. Love your husband.
Married to a monogamous man 15 yrs, mother of 2, dating C 3 yrs, and in a romantic friendship with L more than 20 yrs