Polyamory is all about open communication. Your concerns are real and valid.
You should talk to your friend, sooner rather than later. If you are uncomfortable talking in person, email is an acceptable alternative.
Sex, even casual sex, is serious business and there is nothing wrong with acknowledging that something at least somewhat important happened between you.
If I read correctly, the salient points are:
1) You don't want any secrets as that would certainly damage his relationship with his partner, and living dishonestly is just a bad thing all around.
2) You are happy that it happened, and would enjoy exploring further. (I advise that you don't go further until he's spoken to his partner and then to you about the results.)
3) You also like his partner and would be cool with including him as well, though that is not a requirement.
4) You value the friendship and want to be certain that sex will not interfere with it.
If I've read correctly, then these are the points you need to make to your friend. You are not being clingy or pushy. Just express these points in a calm and rational manner. They are things that need to be said.
Good luck! If all works out well, looks like you could be headed for a wonderful adventure. Don't ignore your feelings or dismiss them. Remember, open, honest, and timely communication is the key to making any relationship work.