No way did it seem like Victor was #1. Some of those eps he mumbled and grumbled in a couple of scenes only.
It's starting to feel like they are taping random scenes that get inserted into an episode. eg taping some Victor scenes on a day when EB is working and then slotting them in.
Although I have read that an actor has to be paid if they appear in an episode. So maybe it's just being done to save time, nor necessarily money.