You say you feel his wife is "superior"... It follows that you think you are inferior. I'm uncomfortable with the idea of you entering a relationship in which, from the get go, you feel inferior, and like you could be "discarded" at any time.
It doesn't seem very fair to me. And of course you'd feel bad about it, insecure, uncomfortable! I suggest you talk about all of that with them. If you can get the opportunity to talk to his wife, too, that might help you as well. Then you can see how that works for you.
The bottom line is, don't go too far if you don't think it's for you. It's important for everyone in a relationship to feel loved and respected and be fine with things. You shouldn't force it if you don't, or you'll always feel miserable about it.