I think you've the right of it when you say he wants to swing. Friends-with-benefits is cool as long as it works for all involved. Each of the FWB relationships I've had, though, involve a bit more care and attachment than a platonic friendship.
If you're wanting an emotional attachment and he's just wanting sex, then it's not worth pursuing. I've never seen that sort of thing work out well. If you're tying up your emotional attention with him, that's emotional resources you don't have available to explore other opportunities.