6.3.3. Who Approves Launch? And Who Owns the Decision to Suspend?
Who Can Stop the System?¶
“When the system fails, someone has to say: stop. The question is, who?”
You can build all the right tools: kill switches, audit logs, fallback paths, even manual review layers.
But none of it matters if no one steps in when it’s time.
🔐 Trust Doesn’t Pause Itself¶
Systems are deployed, decisions are automated, and processes begin to scale.
Then something happens, an edge case, a failure, a signal of risk.
The system may detect the problem.
But who has the authority to stop it?
Not after the incident. Not for future patching.
Now. Before harm becomes irreversible.
🧭 Real Trust Requires Designated Authority¶
Rollback mechanisms only work when someone is:
- Assigned to act
- Authorized to intervene
- And understood to be responsible if the system continues anyway
Cruise Robotaxi Dragging Incident (2023)¶
As described in Section 6.1.3, a Cruise autonomous vehicle in San Francisco struck and then dragged a pedestrian, believing the situation was resolved. The technical components followed their programmed logic. But there was no mechanism for a human to intervene in real time.
No stop command. No live oversight. No one with explicit authority to pause the system.
In Section 6.1.3, this was a validation failure, a system behaving as tested, but not as needed.
Here, it becomes a governance failure: a system with no one responsible for stopping harm before it escalated.
🔁 Governance Must Be Built Into Action¶
Table 50: Governance Patterns for Real-Time AI Intervention
| What Breaks Trust... | What Builds It... |
|---|---|
| “Let’s hope someone catches it” | A named reviewer for high-risk outputs |
| “We’ll know when it’s bad enough” | Risk thresholds tied to action triggers |
| “Anyone can override, but no one did” | Defined authority with traceable intervention history |
| “Fix it after it fails” | Escalation paths designed before anything breaks |
This isn’t about who saw the failure, it’s about who was empowered to act on it.
We now need something specific: a Trustworthy AI Reviewer (or Deployment Risk Operator, Trust Gatekeeper, or Failsafe Authority). This is not a monitoring role or a legal advisor. It’s the person who stays close to the system post-deployment, ready to intervene when the model cannot decide safely. This person must also be an expert in the system’s lifecycle, familiar with how it was trained, tested, deployed, and where its known blind spots lie.
They must understand how rollback impacts users, services, and operations, so they can make fast, informed decisions in high-stakes moments.
Thinkbox
“We’ll have agents in many places, but there will be a human overseeing and approving every step, and you’re on the hook when you approve, when you click ‘Okay.’”, Ali Ghodsi, CEO of Databricks1
This comment aligns directly with Section 6.3.3’s focus on human authority. Even as AI agents become more capable, the responsibility for key decisions will, and must, remain with people.
They are empowered to:
- Review high-risk outputs in real time
- Trigger rollback, pause, or escalation pathways
- Log and explain interventions for future transparency
- Hold final accountability when trust is at stake
Without this role, rollback is theoretical.
With it, trust becomes actionable.
🧠 Design the System to Respect the Stop Signal¶
A trustworthy AI system must do more than respond to inputs.
It must listen to human judgment, and know when to yield to it.
Trust is built through every constraint, checkpoint, and decision that makes failure stoppable, and responsibility undeniable.
True or False: Deployment-Stage Myths
Check your understanding of deployment-phase trust. Are these true or false?
- Model verification during testing is sufficient for safe deployment.
- AI agents should always have write access to act effectively.
- An audit trail should include both technical rollback events and human decisions.
- Prompt injection and log leaks are only training-stage concerns.
- Once a model is deployed, responsibility shifts entirely to the end user.
Reference: Sections 6.1 – 6.2.4 | Focus: Validation ≠ Verification, Plugin Privacy, Agent Boundaries
-
Business Insider. (2025). Databricks CEO on AI responsibility and human approval. https://www.businessinsider.com/when-ai-automation-replacing-humans-databricks-ceo-ali-ghodsi-2025-6 ↩