Could this be a contributing factor to my sex life slowing way down or is it just the way that it affects my confidence?

Is this my husband??? You shouldn't worry about it. It's all part of getting older--but not bad. You may not believe me, but you don't even need to dye your hair or worry about a receding hairline because women really don't care. In fact, if you take care of the rest of your body, i.e. tanning, skin care, nails, and keeping your eyebrows in shape, women will look right past your hair. I am mid-twenties and I personally am attracted to men that look older. (Not like grampa, though.)