You might want to try online dating, especially OK Cupid. You can make a profile where you are seeking bi women, and write thoughtful things on your profile about what type(s) of relationship(s) you are seeking. You can mention that ideally you would want a bi woman who'd be open to exploring a threesome with you and your boyfriend.
I sympathize with your defensiveness about the "swinger" label. It took me a while to feel comfortable with various poly communities and to realize they were not judging me as much as I thought they were. Now I sometimes just take a deep breath and remind myself not to get defensive.
"Swinging" does have connotations of partner-swapping and sex at clubs in front of strangers, so it seems odd when it sounds like sometimes poly people define swinging as any type of sex outside a romantic relationship. (That's not quite what they mean, but it can sound that way at first).
However, you might want to do a more in-depth examination of the swinger community. You may not be a swinger, but maybe you and your boyfriend would click with a bi swinger woman?
I personally do not feel there is a huge black-and-white distinction between "casual sex" and "loving relationships." My best experiences have all started as casual, and my worst relationships have all been "loving."
Single, straight, female, solo, non-monogamous.