@Magdylyn, thank you for your kind words and insight. I really appreciate them. I have since spoken to him and elaborated my feelings to him. Why I believe I am uncomfortable with the situation and then have gained some understanding on his feelings. I know it will take some time to build up my self esteem but I believe I am heading in that direction.
@MsChristy, thank you for your reply and insight as well. I am 25 years old. A little bit of a late bloomer. We haven't been dating too long but he has done nothing to prove himself distrustful, and yet I believe you hit the nail on the head that I did not fully trust him yet. I couldn't say whether I am poly or not yet though, I have been around poly people and seen the relationships for about seven years. Just had never been in one myself and did not realize how it would effect me.