This is an interesting question. I can't speak for all soaps but thinking about ATWT, it seems to have begun, in earnest, when writers started writing stories as if the characters were players in some type of reality show.
By the mid-90s you had shows like The Real World that not only were they ratings juggernauts but they had snared an entire generation of viewer-- a younger generation and we know that soaps were usually on a quest to grab those younger viewers (sometimes to extremes that resulted in disaster).
It just seemed as though by the late 90s, some soaps (e.g. ATWT) cast their younger characters who looked as if they were plucked from a reality show.
We know that many reality shows had continuing stories but their story arcs usually went from week to week (every week a new explosive conflict or fight, revelation or scandal in the house). Some soaps may have picked up on this and started shortening their story arcs trying to appeal to the reality show crowd (who skewed younger).
I think the unfortunate thing that many of the soaps didn't pick up on (until it was too late) was that they needn't mess with the technical aspects of how they wrote the stories, what they needed to tinker with was the content of those stories. Some of the stories just didn't move with the times, in terms of contemporary issues and by the '00s, even those stories that did have contemporary themes seemed poorly executed, sort of sloppily written.
Today's soaps seem to be experiencing something of an identity crisis, of sorts, IMO. They can't compete with the cable and the Netflixes and Amazons or Hulus of the world because they're on daytime network basic broadcast TV. At this point, CW shows and many primetime shows like Scandal are too racy for the daytime shows, so the daytime soaps take a kind of 'middle ground' which can sometimes leave viewers feeling unsatisfied by what they see.
Not to mention that all the limitations of writing for daytime will not exactly attract a whole lot of writing talent.
Let's put it this way, if you're a talented writer and you have a chance to put your energy toward landing a gig in daytime soaps or writing your own webseries that could get picked up by Netflix or get you noticed for your own HBO show, which path would you choose?