Welcome to the forum.
When thinking about your bf, there's one determination to make first: is he poly, or is he using the term poly as a more acceptable term than swinging? You might already know enough a out his past and bus views on relationships, sex, and love to answer that question. If he's just looking for escape, then you've got problems. If he's actually a swinger with no interest in relationships, then poly purists might take issue with his choice of terms.
But let's assume that he's honest poly. If so, he might have already de-programmed the mono thinking that say love is only 1 + 1. Thus, you don't need to be so worried about whether a new person could replace you. Poly folks are very good at seeing differences but appeciating them rather than comparing them.
As for the PCOS, my mono and I invited a couple over Sunday to celebrate their twins' first birthday, and the primary mother in that couple has PCOS. It's tough but possible.
Education and communication are your friends. Reading is important, but talking and istening are also essential. Communicate with each other. Use concrete examples. Ask what his long-term image is.
As a poly male, I can assure you that coming out is not an easy thing. It's a good sign he's talking to you rather than playing behind your back. In the meantime, when you feel anxious, ask yourself if you're worried about a now thing or a future thing. If it's something going on right then, think about it and figure out the root issue. If it's a future thing, try to let it go. The future can worry on its own just fine.