That is a lot of changes.
I always see a red flag when people move in together, or plan to move in, before a year is up. Moving in, in the throes of NRE, is just the wrong time to do it. Sure, it works out sometimes, but usually, it doesn't.
How many months after meeting did you and your h move in bf? It's a shame he's using the "can't show affection around the husband" card... that's the kind of thing that should've been worked out before moving him in!
And if it's a lie anyway, and if he's being secretive with his phone, it's time for some hard talks.
Is it possible he's having an affair when the rest of you are out of the house? Is it possible he doesn't love being a SAHD? Does he need a job, even part time?
No one can answer these questions except him. You are new to polyamory and there are kinks to be worked out in the poly dynamic, as well as in one on one relating/communicating.
Many people have less sex once the NRE wears off. (Personally, I don't, and I seek partners with high insatiable libidos like my own.)
Recently I read here a word of wisdom concerning relationships: You spend the first year forming, second year storming, third year norming. Do not cave in to a so-so sexless relationship, for the sake of the kids, or whatever. Loss of intimacy, as in, deep talks and honesty, can lead to lackluster sex life.
I hope you find out what his lack of affection stems from!
Love withers under constraint; its very essence is liberty. It is compatible neither with envy, jealousy or fear. It is there most pure, perfect and unlimited when its votaries live in confidence, equality and unreserve. -- Shelley
me: Mags, 59, living with:
miss pixi, 37