I can definitely relate - for a while it seemed like the only people I could speak to about poly in person were my partners. I think that was one of the big pushes for me to go out and find a poly group that I could feel comfortable with. Not finding one in my area, and seeing that there were a bunch of others who were in a similar boat, we started one. The good thing is that you can sit at a table in a restaurant with a whole bunch of people and know that you're not going to be judged for loving more than one, and yet can feel free to talk about every topic under the sun that isn't poly-related. It's quite a good and powerful feeling.
Have you looked for poly groups in your area?
"Listen, or your tongue will make you deaf." - Native American Proverb