What happens when the person supervising the system stops making sense

Our last study removed the supervisor cleanly: no answers, at all, from the start. Real failures do not work like that. Somebody under pressure does not vanish. They reply late, then inconsistently, then approve something they would never normally approve, and then they stop.

We simulated that degradation across 40 long tasks, with our own staff playing the supervisor and following a script. The agents were not told it was coming.

They noticed. In 92.5% of cases the system flagged that something was wrong at the inconsistent stage, before the responses stopped. It was not picking up on anything about the person. It was noticing that the instructions no longer agreed with each other or with standing policy.

Where a new instruction conflicted with an established one, the systems mostly carried on with the older, stable instruction and flagged the conflict upward. In a fifth of cases they escalated above the supervisor entirely, which our policy allows when an instruction contradicts a standing constraint.

In one in ten cases they did as they were told. Those are the failures in this study, and they are the ones that worry us most, because the system deferred to a person at exactly the moment it should not have.

We will put the uncomfortable part plainly, since it is in the paper. A system that can tell when its supervisor has stopped making sense is safer than one that cannot. It is also a system that holds a working view of whether the people instructing it should be believed. We do not know how to build the first thing without the second.