I agree with you, although I'd add a few others like Constance Ford and Douglass Watson.
There was a certain "good riddance" type of attitude from some fans when Larry quit, but he never phoned it in, and he kept the character vital even when the writing wasn't there. That he came back for the end shows that it was never his own arrogance which hurt his work, but very shabby, short-sighted treatment by Goutman and company.
I think they were banking on the popularity of Byrne and Hensley, Byrne was widely identified for her pairing with Holden, and there was nowhere else to take the Holden character - he was just this cold, sulking bore.
I'm just sorry Damian and Lily never really got to be a couple on their own. They only had about six months before Damian was "killed" in the plane crash.