The whole point of polyamory is to fall in love with more than one person. If you are to ever do poly, that's exactly what it will be!
Working polyamory, however, relies on honest, open communication--and that appears to be missing from your relationship. For example, you appear to have negotiated an open relationship--and not have actually worked out what that means between the two of you. You took it to mean you could find physical affection and company only and he appears to take it to mean something different.
Have the two of you ever sat down and hashed out exactly what it is you need from a relationship? Then figured out what you can provide for each other and what needs to be sought elsewhere? That's where it all begins, with talking about what you need and how to go about getting it and working in cooperation with your partner to achieve it.
There are also issues that need to be addressed that concern only the two of you and have nothing to do with other people. Why does he spy on you? Why would you marry a man who is so emotionally unavailable if you require lots of emotional interaction? You've hinted at some very fundamental problems and those need to be worked out before trying to add new people to the equation. Adding more people will not fix a relationship that doesn't work well.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.