Of course you have the right to go at your own pace!
I think we have to be very careful not to get too hung up on the details of conversations when one of the parties isn't even present. Firstly you don't know that she actually said "you should" she may have said "you can". One word can make a huge difference, it can be a bit like a game of Chinese whispers. And even if she did haven't you ever said to someone "hey you should come round for dinner one night"? Secondly I think whatever happens with this relationship it is going to be really healthy for you in learning to be assertive. Just because she's his wife doesn't mean she suddenly holds any sort of power over you to do things you're uncomfortable with. And I'm sure she wouldn't even think that she does.
As far as the labels go. Yep...been there had that. Some people are happy to wear a label because it helps us understand ourselves and what's going on around us. Others feel like they are being put in a box. OSO can't get her head around the difference between polyamory and polygamy for goodness sake and she hates labels as well. Even Z is still playing with the idea, I think he likes to think of himself as totally unique in the whole world. It really helps to keep a bit of s sense of humour about the whole thing. I'm the mono and the only one who really sees the polyamory in our situation