There's nothing saying that you have to be friends with a metamour but if you aren't I wouldn't suggest moving in together...ever. It's just going to make for a miserable living situation for all of you. Why are you feeling like you have to all live together? Is there a possibility that as your relationship develops that you will be able to ask for more time with him? Also if he can't meet all your needs is there anything preventing you from finding another relationship to fill those unmet needs?
Everything will be ok in the end. If it's not ok it's not the end.