I think it is impossible for us here to comment on the other people in your life. They all have needs, gaps which drive their behaviour.
Focus on you. The opinions of other will only hurt if you have a bad relationship with yourself.
I'm not sure if you're in UK or US. But the Farm contact above looks cool. Hard work but great fun. I've done that. It's a way of stepping away from all the noise of life. Stepping away from all the people who know your history, all the baggage and just starting with a clean slate, getting to know who you are and deciding free from constraints who you want to be.
I've worked with ex-offenders and their biggest weakness is the people they surround themselves with because those people do not want to see them to turn their lives around because then they'd feel bad about their own choices. It's a vicious circle. Step out of it as soon as you can. Be the best you can be.
I do understand that need for human closeness and physical loneliness is tough. It doesn't sound like you were nurtured as a child and that need will feel very real and painful. But you are with a woman who is there but making you feel even more lonely.
Good luck. Be strong.