Even though i'm sure he enjoyed his free time, sometimes it can take an adjustment period to smooth those kinds of issues out. Sometimes a partner "thinks" they can try such a lifestyle, but once it's really happening, the envy starts coming out. Either he will become good with it, and not bother you during your times out with your friend, or it will go the opposite way. You stated you are new to this...lots of feelings are bound to start coming out, that you never anticipated. I am also pretty new to Poly, but feel it is who I am. I am good with all that is going on with our quad. I would suggest that you set a time for your return but also talk about letting you just be you, during those times you are away. If you both enjoy what you are getting from other's..you will work hard at making things work. I would say, in a new Poly relationship, it is very normal for at least one of the spouses to experience some emotions, once your actually out on your date.
Personally, I like to work it out, so all four have similar times to hang out, with a time to meet back. That way it keeps it as "fair" as it can be. I have come to the conclusion that not everything is completely fair...there is a lot of give and take. I have found compersion...I always feel really good when Scooby is with Bobbi...she spoils the hell out of him, as Jack does me...so it causes me nothing but joy. But I think it takes a while to be able to feel that for your spouse. Good luck...reading here on the forum, can help you understand what other's have experienced and may give you some great insight. My feeling is, that your primary had second thoughts after you left the door, even though he enjoyed his "out" time. I find this to be typical.
Your future in this will depend on how you both work this out, so that it can be enjoyable for all involved.