As I think other cast and crew and certainly Michele Lee pointed out, Murray was supposed to be kind of the big get from movies, etc. when the show began. (He's amazing in Advise and Consent and capped his career not long ago with a wonderful performance in Twin Peaks Season 3) He later noted he regretted leaving after the show ran on and on and on. I really liked him and Sid, but I agree it was for the best. Those first two eps of S3 are absolutely brilliant and it opened up everything that came after.
Episodes like the biker gang from hell (Born Free), Cricket, the ridiculous S2 hostage crisis (Moments of Truth), Three Sisters, the Lilimae Vegas romp (Rose and the Briar) and Silver Shadows all show the upper limits of this approach, so yes it was time to change. And I happen to like Three Sisters and Silver Shadows, but they are off-brand lol.