First, the touching part -
Different people can have different boundaries as to what they consider acceptable behaviour for friends, close friends, lovers, etc. Often it's important to understand that just because you think that behaviour X goes beyond the limits of what you regard as acceptable, doesn't mean that anyone else should necessarily be thinking the same way. Part of what poly is about is breaking down some of the societal absolutes that most of us grew up with.
So it's important that both your OSO and your friend know about your boundaries, and have the opportunity to respect them. Chances are you have hinted at the fact that you are uncomfortable, but you need to be less subtle and actually express them as a boundary.
This goes doubly-so for what goes on in your own house.
If folks won't respect your boundaries, but instead try to work around them through cajoling, or guilting you into thinking you are wrong to have those feelings, then that should be a red flag to you.
As far as the time is concerned, I am guessing you don't need more time so much as more quality time with your OSO. As others have said, see if you can make this social thing less of a part of what you do together so that you can have more time alone to do that quality things that you want to do. If he has restrictions on his available time, then try to make sure that you both use the time you do have to the maximum.
It sounds like your OSO has a pretty level head on his shoulders, and knows what he wants and doesn't want. Make 100% certain that he knows where you are comfortable and where you are not.
"Listen, or your tongue will make you deaf." - Native American Proverb