Why are you considering yourself the third wheel to THEIR relationship? Are you involved in anyway with her as well? I would think a good first step is to start thinking of your relationship with him as YOUR relationship with him, not THEIR relationship with you. Then, figuring out what you want/need from a relationship is important, too. Are you sure you're mono or is there a chance you would be interested in a poly lifestyle (one where you date others) too? If you do decide you need to have one partner who is your "primary," then this relationship probably won't be the one for you since he already has a wife and kids.
Your idea that poly would be better for couples focused on sex is also not what poly is really about. That concept is more of a swinging mentality, and while the two overlap they aren't really the same. Poly relationships really should have the focus on the loving aspect vs. the physical.
It definitely is frustrating when someone can't offer you as much as you need/want, but too often it is a fact of life. Have you talked to him about how you're feeling?