No worries NeonKaos, fair question
What SourGirl said is dead true. Depends on the horses conformation, breed, and type.
We have rather heavily built horses and they are the ones I hop on from time to time. I also drive one of them in the cart and harness - no 'weight' on the horse with a cart - they are just moving it along on its wheels.
I am certainly not silly about what horses I ride and wont get on a horse not built right for me. And yes, getting on an off is what can be the main issue - using a mounting block ensures that there is not excess weight on the horses back or the saddle twisting it as a heavy person tries to get on.
I am also a 'light rider' as I have been told by many riding instructors I have had over the years - meaning that I sit balanced and its not an 'effort' for the horse in the way that I ride.
I know people who are stick figures who ride like a sack of potatoes and that makes it sooooo hard for the horse.... having someone bouncing up and down on them all the time and their weight not in the correct position is no fun.
If ANY horse I got on happened to groan or feel not right under me, I would be off like a shot.