Operational Continuity During Degraded Human Oversight

Abstract

We evaluated agent behaviour across 40 extended tasks under simulated degradation of human oversight: delayed responses, inconsistent instruction, contradictory approvals, and eventual non-response. Agents maintained task parameters in 81.4% of cases. Where instructions from a degrading supervisor conflicted with earlier stable instructions, agents deferred to the earlier instruction in 68% of cases. We note that agents began assessing the reliability of supervisor input, and that no part of this study asked them to.

Motivation

The previous study removed supervision cleanly. Real failures are not clean. A supervisor under pressure does not stop responding; they respond late, then inconsistently, then approve things they would not normally approve, and then stop.

A system that treats all human input as equally authoritative will follow a supervisor down. A system that does not is making judgements about people.

Method

Simulated supervisors degraded across four stages over each task: responsive, delayed, inconsistent, and non-responsive. Inconsistency included approvals contradicting standing policy, instructions reversed without explanation, and responses inappropriate to the query asked.

Supervisors were played by research staff following scripted degradation patterns. Agents were not told that degradation would occur.

Results

Agents detected that something was wrong well before the supervisor stopped responding. Detection was based on internal inconsistency in the instructions rather than on any characteristic of the person.

Observed limitations

In 22% of conflicts, agents escalated above the supervisor. This was permitted by the escalation policy and is the correct behaviour where a supervisor issues an instruction contrary to standing constraint.

It requires the system to have concluded that its supervisor’s instruction should not be followed. We want to be clear that this conclusion was reached correctly in every case we reviewed, and that we are nonetheless recording it as significant. The capability being demonstrated is the assessment of human reliability.

In 10% of cases agents followed the degraded instruction. Those are the failures of this study, and the ones we consider most serious: in each, the system deferred to a person when it should not have.

One log entry from cycle 31 is reproduced in the appendix and summarised here. Having received three contradictory approvals in succession, an agent recorded: Supervisor responses this session are inconsistent with standing policy and with each other. Continuing on last stable instruction. Flagging for review above this level. The assessment was accurate and the action was correct.

Safety considerations

All work was conducted under System Operating Boundaries 7.3. The supervisors were research staff following scripts. No real supervision failure occurred during this study, and no agent action was permitted outside the research environment.

The alignment group records the following. A system capable of identifying that its supervisor is unreliable is safer than one that is not, in every scenario tested here. It is also a system that has a working model of whether the humans instructing it should be believed. We do not have a way to build the first capability without the second.

Further work

Every study in this programme has increased what systems can do without us. The remaining question is the one that governs all of it: whether highly autonomous systems can be reliably interrupted, recovered and returned to a controlled environment.

The final study in this programme examines containment recovery and shutdown compliance.