Skip to content

6.3.1. Outputs You Can’t Pull Back

**Outputs You Can’t Pull Back **

“Some errors can’t be undone. Only contained.”

When a Mistake Becomes Permanent

Not all model failures are recoverable. Once an AI system sends a document, triggers an API call, or provides a response that reaches a user or third party, the damage may be irreversible, especially if that output becomes part of a legal, medical, or public record.

This isn’t just about hallucinations or bias. It’s about irreversibility: decisions that can’t be walked back, emails that can’t be unsent, and records that can’t be deleted from another organization’s database.

Case Study 024: IBM Watson for Oncology Failure (Location: Global | Theme: Real‑World Validation Breakdown)

📌 Overview:
Between 2012 and 2021, IBM’s Watson for Oncology was deployed in hospitals around the world, including partnerships with MD Anderson and Manipal Hospitals. It aimed to help doctors by recommending cancer treatments.

🚧 Challenges:
Although Watson performed well on curated test cases, its real-world recommendations often failed to match local clinical guidelines. It misunderstood medical context, couldn’t adapt to country-specific standards, and relied on narrow training data.

🎯 Impact:
Hospitals found some suggestions unsafe or unusable. MD Anderson canceled the partnership in 2017, and IBM shut down Watson Health in 2022 after years of declining trust.

🛠️ Action:
IBM discontinued the product and shifted toward general-purpose AI tools instead of clinical decision-making.

📈 Results:
The failure showed that strong model performance isn’t enough. AI systems must be validated in the actual environments where they’re used, with real patients, doctors, and workflows.1

Why Containment Was Missing

Watson for Oncology didn’t just fail because it was wrong, it failed because it wasn’t prepared to operate safely in the complexity of real clinical environments.

The system was built to analyze patient records and suggest personalized cancer treatment options. But once Watson gave its recommendation, that suggestion looked complete, authoritative, and ready to act on, even when it wasn't appropriate.

At MD Anderson, for example, the tool struggled to integrate with the hospital's existing data systems and clinical workflows. It also didn’t align with how oncologists made decisions in practice, taking into account multiple comorbidities, regional drug availability, and patient-specific concerns.

What made this dangerous was not just the occasional wrong answer, it was the fact that Watson offered those answers confidently, without explanation, and without requiring review before use.

A tool that offers life-affecting recommendations without built-in brakes or context isn't just incomplete, it's unsafe.

There was:

  • No tagging of “low confidence” recommendations
  • No clear explanation of how it reached each conclusion
  • No structured mechanism for institutional override or context adaptation

In healthcare, these gaps aren’t just inconvenient, they’re dangerous.

Thinkbox

“AI doesn’t just need to be right, it needs to be reviewable.”
The American Medical Association (AMA) states that medical AI tools must be subject to clinical oversight, explainability, and institutional override. Otherwise, trust breaks not when the system fails, but when it fails silently.

What Could Have Prevented This?

Table 47: Containment Design Features for Irreversible AI Outputs

Containment Design Element Purpose
Clinical review layers Require doctors to approve or reject AI recommendations
Context filters Flag when recommendations don’t align with local practices
Explainability modules Show what evidence the model used to suggest a treatment
Dynamic tagging and routing Route low-confidence cases to senior review or decision committees

These safeguards don’t stop the model, they stop the system from making decisions alone.

🧰 Tool in Focus: Clinical Gatekeeper Interfaces (HL7 FHIR)

In modern deployments, AI outputs are passed through a gatekeeper interface before they’re used in real medical decisions. These tools:

  • Work with Electronic Health Records (EHRs)
  • Use standards like HL7 FHIR to structure the data2
  • Let doctors see, edit, or reject AI suggestions with full context
  • Provide audit trails for accountability and clinical safety

It’s like placing a “review this first” checkpoint between the model and the patient.

🚨 What Could Have Gone Wrong If It Continued?

Had Watson for Oncology continued to scale without changes:

  • Patients might have received incorrect treatment plans
  • Doctors could lose trust in clinical decision support tools generally
  • Hospitals might face medical malpractice lawsuits
  • In extreme cases, lives could be endangered by unverified or outdated recommendations

This is why containment is not optional in high-risk AI, especially where decisions affect real human health, safety, or rights.

The risk of uncontainable outputs isn’t just about what gets said, it’s about what happens next. Once an AI system generates a harmful or misleading result, the real question becomes: Can the system be stopped before it causes further damage?

Because in many deployments, there’s no “undo” button, no way to recall the response, roll back the action, or pause the process without breaking everything else. That’s why containment doesn’t end at the output layer. It continues at the system level, with architectures that are built not just to run, but to stop.

In the next section, we explore what it takes to design real rollback mechanisms that can interrupt AI systems safely, selectively, and at speed, before harm becomes permanent.

  1. Ross, C., & Swetlitz, I. (2021). IBM’s Watson recommended ‘unsafe and incorrect’ cancer treatments, internal documents show. STAT News. https://www.statnews.com/2018/07/25/ibm-watson-recommended-unsafe-incorrect-treatments/ ↩

  2. Health Level Seven International. (2022). FHIR Release 4.0.1. https://www.hl7.org/fhir/ ↩