Welcome to our forum.
A "unicorn configuration" is actually not all that rare. Where the problem comes in is where the "original couple" takes precedent over the added lady, objectifies the added lady, insists she be exclusive to them, etc.
But not all "unicorn hunters" (I hate to use the word because it's pejorative) are like that, many are quite flexible, reasonable, and kind. So "the bad ones" give "the good ones" a bad reputation.
It's an unfortunate circumstance but there's not a lot you can do about it. It's no good changing your life/desires because of how other people act. It would be like me choosing not to be poly because some poly people act bad (and because so many mainstream people think poly is a horrible thing).
I think you should do what's in your heart, and try not to worry about it too much (or not let it become a priority) if people give you a bad time about it. It's up to each person to not make assumptions about you, but rather to ask you if they have concerns or whatever.
That's my opinion anyway. Just be as patient as you can in explaining your situation.
Love means never having to say, "Put down that meat cleaver!"