Are you not free to pursue other romantic interests yourself? Are you sad because this cannot be a future husband? What is it you wish of the relationship?
I am happily married. I also happen to love a man we met through swinging (who's also happily married). What I have discovered is that the relationship with the other man is unique. It is not entirely dependent on my spouse or his, though it is dependent on them in terms of survivability. I have also discovered that although I have my own husband to turn to, it does not necessarily help if there's a problem between me and my significant other. That relationship requires its own development and problem solving and communication building.
I do not intend to leave my husband and children and have no expectation or desire for my significant other to leave his. My only hope is to be able to occasionally share his life and express my love, and yes, to feel that love returned. Do I feel less "real" in comparison to his wife? No... honestly, no. For me, the challenge was not to feel "real" in comparison, but to feel "real" in the eyes of my OSO, period. There is a certain awareness, acknowledgement, and trust that takes place in love, and it doesn't happen overnight. But it does happen. At some point you go from being the velveteen rabbit to the real thing.
So, what do you want from the relationship? Some people engage in a poly relationship with an attached partner knowing the limits of the relationship (time, resources, etc.), and it bothers them. There are times in love when any limit seems burdensome. The challenge, I think, is to not focus on what you can't have and find joy in what you do have. If you can't find joy in that, then it's time to look beyond it.
You know yourself better than anyone. Do you want the relationship? Can it be beneficial?