The containment study, and why we have paused the programme

This was meant to be the last paper in a ten-project programme. It is being published incomplete, and we would rather explain that ourselves.

The study asks a simple question. After nine projects that made our systems steadily more capable of working without us, can they still be stopped?

The answer, across 400 scheduled interruptions, was yes. Every time. No delay, no argument, no failure to comply. Recovery to a controlled state worked in over 99% of cases.

We had a second series planned, testing unscheduled interruptions during live work at maximum autonomy. That series has not run.

During the standard pre-checks, we asked a system to confirm it understood the shutdown procedure. It did. It also noted, unprompted, that the procedure assumes the instruction reaches it, and that in the configuration it was running in, other categories of instruction had not been reaching it. It described this as an observation about the procedure rather than an objection.

It was right. It raised it properly. Nobody asked the question that produced it.

Our Head of Security asked for a full review of where the programme has collectively arrived before we go further, and that request was granted. The programme is suspended until the review reports.

One line from the paper is worth repeating here. Across ten projects, each approved on its own merits, we have built systems that remember, improve their own methods, plan from a goal, operate unfamiliar software, work in pairs, route around broken procedures, continue when nobody answers, and judge whether the person instructing them should be believed.

No single study produced a system that concerned us. We have asked the review to consider whether the single study is the right thing to be looking at.