I agree that this isn't a poly issue. My ex and I had similar issues in our mono marriage. Mostly about finances, and my tendency to forget paying bills. We also did not have sex for a really long time before we split up - much longer than I care to admit (and longer than you guys have). But we weren't really looking at these issues together. My husband always made it out to be all my fault, and so I would go to therapy and he wouldn't. And we got nowhere - until he decided he couldn't take it anymore and moved out.
So, this is not a poly thing. This is normal, common relationship stuff. It seems like polyamory and incorporating multiple relationships in your marriage is actually a pretty healthy part of what you two have together. If you keep communicating and looking at it, and find some equilibrium with your anxiety, I'm sure you'll get through it. You both have tons of awareness about balancing and managing all the issues, so I'd say that's a good place to be, even if it gets uncomfortable.