I don't know what feelings that are worried about? It sounds to me like you have given them as much latitude to pursue their relationship as they need. Is it that you don't want to have a sexual relationship with them that bums your husband out? Your family sounds like it is built on a solid foundation and that each of you communicate well with each other.
If a sexual relationship (i must clarify that i dont assume that either of you desire a purely sexual relationship with another that does not involve deeper feelings and connection) with you and another is what you and you husband desire then why do you not pursue the person that meets your needs as well as his?