There is overlap in poly and swinging. However, if you are looking for a romantic relationship, you may want to try to go to more poly oriented activities (as well as swinging since you like that as well). I think that once you are open to the idea of poly, you may find new opportunities coming up since you are looking for it.
Once thing I have learned about being poly is that a relationship is not in a well-defined label. You can have very close, intimate friends or an occasional lover. Maybe you can enjoy the closeness you have with OH and OW as it is and let that relationship seek its own level. It doesn't stop you from experiencing other relationships and letting them develop as they may.