Here is a suggestion and not a reprimand. Something I learned from a favorite advice columnist, Mr. Savage.
Problems with condoms and dicks... as it doesn't seem to be an allergic reaction, this might help with the problem of taking so long. Practice. Not with intercourse, but with manual overides. Put on a condom and get yourself off or have your lady/ies help you manually. Keep working on it, try different amounts of lube in and out of the condom. Practice does make perfect and this might just help so that you can limit the risks and consequences all around.
Life is about the journey and not the destination,
so what better way to know life
than to wander all the roads and paths set before you.