Short answer? Yes, it IS that hard for most people to understand.
Longer answer? For the most part, we're raised to believe that one single person in a monogamous relationship is the end all and be all that we should all strive for. Anything else is seen as "freaky" and "weird". I've always LOVED in a way that was poly, but was never open to a poly relationship. It resulted in alot of cheating, until my hubby helped set me straight on what I was feeling and what I was doing as a result. I thought "poly" meant sex and was disgusted by it when younger. Unfortunately, for people who don't understand and don't feel what you do, or simply can't undo their conditioning, the true meaning (or what you feel it is) of polyamory may need quite a bit of explaining. For myself, when I try to explain my poly I approach it as I did coming out as bisexual. Some will get it, some won't, most will make stupid comments. I grin and bear it and those who are true friends will eventually understand, or at least accept me for it. You'll find that if you can bear the childishness, most people really don't mean anything bad and will come around. At least, I hope you have that experience.
As far as the "penis that turned you", unless the picture you were showing was actually OF his penis, that's a bit crude.