You're thinking of it more like in terms of calories expended, like work in joules or something. But with me, my "extramarital" relationship really does not require that kind of work-effort, unless you consider having wacky sex for 14 hours "work". I would quit my day job for that.
And my live-in spouse-partner? Besides the "work" of needing to remind them to do stuff (ADD, etc.), there isn't that much "work" either, at least not on an every-day basis.
I still think that a lot of this "work" people speak of is the result of feeling like they've invested "this much" in a relationship so far and they don't want to "lose" that.