While Curly and I were corresponding prior to meeting, she sent me a message that described men who weren't her type. Fit me to a 'T'. I chuckled and forgot about it, because at that point I had no expectation of ever meeting her or dating her. Shortly after we began dating, she apologized for that message (which is the only reason I remember it now).
Somehow, it didn't matter that I wasn't her type once we met. I do occasionally remind her that I'm not her type, too, especially on our wedding anniversaries.
We wouldn't be married if I'd ever gotten the idea that she wasn't wholly interested in me. Sorry, wouldn't happen. Any hint that she thought she was somehow settling for me and I would have been gone. I imagine anybody with any scraps of self-respect would be gone in a heartbeat.
From here, it doesn't appear as if you're inadequate at all. Looks to me like he's rather inadequate for you. If he's not able to provide what you need from a partner, he's inadequate.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.