Believe me, I'm not a fan of Rafe and agree with some of what you say (though I think the Daniel propping was much more extreme). Rafe has been on the show for over eight years now. Stefano tried to have Rafe killed to stop him from revealing the baby switch. He kidnapped Rafe, erased his memory and replaced him with a look alike. He sent detective Bernardi to cut off his penis after finding out about Rafe and Kate. I don't see anything wrong with calling that Rafe's history with Stefano, that's what it is. Rafe visting Stefano in jail didn't bother me. The problem was that he was the last one to share a scene with Stefano. His scene could have been in the beginning of the episode. Bo and John would have been great but that couldn't happen because of the actors. Marlena's scene and "Checkmate" line could have been a good final scene of Stefano (under the current circumstances). And the ending with the empty jail cell could have still been there as a way to keep Stefano's mystique. Of course none of this matters after seeing Friday's show with the terrible reveal that it was Shane pretending to be Stefano.
I usually don't mind short visits for characters, when it makes sense for the story. Like Shane, Kim, Nancy and Laura these last couple of months. I haven't seen that with Austin and Carrie this time.