You could have it, yes. It's a lot of hard work though.
Things like communication, outside interests, respective family and friends, misunderstandings, LACK of communication (I'm sure there are other *things*) all lead to a rich, full life AND could also get in the way of having that 'fantasy' life you're dreaming of.
You MUST remember that the mundane parts of life (paying the bills, rent/mortgage, etc.) must all be dealt with as well.
I might write more in a bit, gotta think a little more.
There are as many ways to do polyamory as there are people practicing it!