You are not being unfair at all! Each of you brings individual wants to the relationships you form. No one has the right to dictate how you will love or who you should love. With that in mind, that doesn't mean that they have to accept your involvement with them if you chose to do your own thing.
It's about criteria to be in a relationship....do they meet yours? Do you meet theirs?
Be true to yourself and the future will be more natural regardless of the outcome.