That suggests raging feelings of inadequacy on part of your primary. The secondary is in a secure position while the primary is not, leading to feeling inadequate to the "challenge" posed to your affections--yeah, it appears he views you as an object, rather than as a person.
The whole "secondary is beneath you" line seems to be a projection. The primary is worried that he, himself, is beneath you and the secondary provides you with a choice that is superior--having stable employment and able to offer security. So he presents the situation as being reversed in the hopes that you won't notice his inferior position in economic terms.
That ties back into his notion that you're an object to be won--that you can be had just by the economic investment made/offered. It's a backwards, old-fashioned view that, sadly, has yet to die out.
The whole "clean slate" deal may simply be an effort to buy time until he's working again and feels able to compete to buy your love.
When speaking of various forms of non-monogamy...it ain't poly if you're just fucking around.
While polyamory, open relationships, and swinging are all distinctly different approaches to non-monogamy, they are not mutually exlusive. Folks can, and some do, engage in more than one of them at a time--and it's all good.