Relationships are about compromise. If you're not happy but he is, that's not a compromise as is vis versa. 3 1/2 months is not really long. I have been in my first poly relationship for about 2 months and it is still very new to all of us.
It's going to take time to figure everyone out and if this relationship is viable. I know that you're feeling selfish but you also have to realize that you are never going to be mono. You are a poly person. You can act mono, but that's not who you really are.
I cannot promise you that this is all going to work out for you. I can't even promise that to myself and my own new poly relationship. What I do know is that im discovering something about myself I didn't know before and it takes awhile just to figure yourself out sometimes. You should not feel guilty that you are poly. You should not feel like taking care of yourself and your needs are selfish. Selfish would be not willing to communicate and compromise. Selfish would mean you wouldnt be here with these feeling and thoughts.
If your relationship does not work, it didn't fail because you were selfish. People change, circumstances in life change. It is no ones fault. Just do the best you can and try to stay positive.