It depends on the individuals. People who "do poly" because they need company all the time, or don't like being alone are poorly suited to it IMO. People who are very independent and enjoy alone time (by themselves) are better suited to it. this is taking for granted that everybody is not going to spend all their time together even if they all live together. And you have to consider that the hinge person also needs alone time by themselves. Are you a person that values your own company any personal space to a great degree, or not? What about his wife? What about himself?
You could run into problems if there is some sort of "couple privilege" going on. You said that you and the wife are friends. How? Did you know her before? Does she date other people? Does she have any say in the times, places, etc. that you spend with him? Does he need to check with her before he can make plans with you? Do you all hang out together? Etc.
I can certainly imagine someone in your position feeling like they are not getting "their share" of time. You need to be honest and realistic with yourself and so do they. If there's any sort of competition going on, you need to identify that and figure out a way to stop it and get everyone on the same page.