Generally the difference between swinging and poly is the romantic side of things. A swinging relationship is based on sex only, with a friendship usually being a secondary concern. FWBs are friendships with the sexual as the benefits (as opposed to swinging where the friendship is the added benefit). Poly is usually a bit more romantic than FWB, emphasizing the emotion and eventually including romantic love.
Figuring out what works for you is really the most important thing. Whatever label you choose to use will help others understand, but the terms overlap enough that they are flexible.
Whether or not you would be considered poly would depend on who you ask, I'm sure. The four of you care about each other, but is more of a friendship type caring or a romantic caring? When you move in together is it going to be a roommate type situation or more of a step towards the four of you partnering to share your lives completely?