Interesting, I am in the opposite position - that of secondary, struggling to figure out where I stand with his primary. There are times when I can tell him being with me makes her feel awful and that, in turn, makes me feel awful/guilty and both of us feeling awful/guilty makes him feel awful, which adds to the awful. It's an awful cycle of awful! There is no magic way to feel better about the situation - as others have mentioned, it takes a lot of work, but if there is love and respect on all sides it is doable.
I think it is important to acknowledge that everyoneneeds to feel loved, desired and "safe" in their relationship. You are obviously very, very important to him and how you feel matters! If I have learned anything (being a new poly) it is that communication is key. Don't pretend it is okay if it isn't - express how you feel! I really hope you can work through your emotions and find joy in your relationship. Send me a message if you ever want to chat!