What Ciel said last.
Polyamory is based on love.
If your brother thinks love is immoral--particularly not having experienced a loving relationship--there's nothing you can say to him. You can only be who you are and hope he grows the hell up.
It reads to me as though he is regarding you with a judgmental eye, based on his own assumptions and suppositions. That is a form of idolatry--i.e., he is following the sibling-idol in his own inner mind, rather than allowing you, his actual sib, the dignity of your own evolving personhood.
Ask him whether he considers idolatry immoral.