One of my first poly "relationships" was with a woman I met via swinging. The thing that we lacked that we really needed? TONS of discussion. I mean, we all went in with expectations of swinging, so when she and I felt like we needed time for "us" her husband felt extremely left out. While he thought it would be exciting to know we were going out then having sex without him, in reality he found out that he just was NOT into the idea of her being with someone without him. At all. Made him feel unnecessary.
Luckily, my hubby was super awesome through it all. Once she and I went out separate ways, he and I decided to continue exploring polyamory. The other woman and her husband still swing on occasion, but have mostly closed up due to the insecurity that the whole thing revealed.
Moral of the story? Make sure you're all on the same page about expectations/boundaries. When you go into a situation planning for one thing and end up doing something completely different, something is bound to get a little out of whack and need some focus.