And there's debates about what "hurts" somebody else. A lot of people think same-sex marriage would hurt them and their family (by eroding the society they live in, to put it succinctly). A less-charged example would be if I decided I wanted to paint my house orange and purple, and my neighbor argued that it would hurt him first of all because his eyes would be hurt, and secondly because it would affect property values.
So a line has to be drawn where you say, "Sorry buddy, but that's not enough so-called 'hurt' to give you special rights over another person's freedom."