Hi, I think if you read your post you'll see these flags that pop up:
1. Lack of Honesty
2. Swinger turned poly
3. GF drug user./ user of all sorts... do you want this drama in your life?
5. Emtional risk? (what about physical risk-drugs and the obvious sexual ones?)
You also have to think about what you want? Yous eem very focused on his needs while he is not trying to even understand yours. Very onesided ineed.
Many have gone down this path, I suggest you read all you can and see how it went here. I think a lot of people like the idea of poly but without actually working at it as you would any other relationship. Ask yourself the hard questions... what kind of future had you seen with this person before the gf was ever in the picture?