I think it's great that you are as sex positive as you are. It's a rare thing.
I think the reason it might be a concern is that not everyone else is as comfortable as you. Sex can complicate things just because, as we hear here a lot, sometimes someone is not as into the other person as they wished. Especially sexually. Just not compatible sometimes in bed.
That then puts a strain on what would otherwise be a great friendship. There's always this "threat" of sex hanging over their heads. Not wanting to hurt someone's feelings etc.
So it's a risk. You hop in the sack together and it either works or it doesn't. If it doesn't - the majority of people are too uncomfortable to discuss that and up goes the wall. And a friendship often spoiled.
Not a lot of people are sex comfortable enough to just sit down and say " I love you to death as a friend but sexually we aren't a match" Fragile egos being what they are.
Make sense ?