I think you need to tell them that you need to suspend sexual activity with them because you are developing romantic feelings for M. If they are open to a deeper relationship with you then they will be the ones to initiate it. If they are not open to a deeper relationship, then you will know and be able to get the distance you need to extricate yourself from the feelings you have.
All waiting is going to do is get you more emotionally invested and make it harder to break from them in the future.
Me: 48 - Married, straight, male
Shiela: My wife.
Suzanne: My FWB
Adam: Shiela's LDR