Hullo and welcome!
I think you'd do well to check out tags around 'NRE', because talking one to two hours per day is just that. It might help you to realize that NRE or New Relationship Energy or falling in love is a state of emotional turmoil and hormone overflow which doesn't last forever.
You describe Victor 'being brought into your lives' almost as if your wife was cheating on you. Was she? I sort of understand that while you felt swinging brought the two of you closer, now you feel poly is taking you apart. There is an inkling of truth in that, but like idealist pointed out, it's more a change in mindset than anything else. Many couples have experienced increased closeness with poly. Intimate relationship are a play of feelings of closeness and distance.
I don't think there is anything necessarily possessive about wanting you SO to stick to plans made with you and not derail them in order to spend impromptu time with new squeeze. To me, holding on to your time commitments is a sign of respect. People justify a lot of shit to themselves while in NRE. It doesn't make them bad people but doesn't make cheating or lying okay, either.
Do a tag search on 'jealousy', too. There is help available for dealing with these emotions. You are not jealous, you are feeling jealousy at certain points in time, and like all possessions, jealousy can be released, too.
Me: bi female in my twenties