this courage-thing can be hard. I understand why you are scared of the consequences, but all I can say: don't take as long as it took me to finally sort your thoughts out. It has taken me more than 3 years
Just talk about it with your husband, he seems to be quite likely to actually understand what you are talking about really quickly. My husband was able to wrap his mind around the whole business within weeks, I was really surprised by his speed.
This doesn't has to be the case but your husband has at least thought about a possible connection between the two of you if he calls this other man your 'boyfriend', even if it is just a joke. Was the exact same situation with my husband and me back then.
Therefore: wishing you some sort of conclusion to come your way and good luck ^.^
Facts: 30, female, bi, v-type relationship with Sward (husband, straight, mono) and Lin (boyfriend, straight, mono), poly-fi and co-primary.