Cool that the two of you have talked at such length - so many couples don't, and make assumptions which prove to be wrong.
If I may ask more of the questions so typical in sorting out what poly means... It looks like each of you will be having your own relationships. Is it more for sexual fun, or are you looking for more committed, romantic relationships with the word "love" in the mix? Do the new folks have to be single, or can they be in other relationships? Are you eventually looking for everyone to live together, or will each person have their own place?
"Listen, or your tongue will make you deaf." - Native American Proverb