He's getting progressively worse over the years...?

I think maybe you should let him know how his behaviors make you feel, and that it's making him seem less appealing. Or give him an ultimatum that he needs to get his act together or it's over. You can't wait around forever for things to improve..