Skip to content

7.2.3. Oversight as a Shared Responsibility- Lessons from Education AI

Oversight as a Shared Responsibility- Lessons from Education AI

The design of oversight tools is often framed as a top-down exercise: engineers build dashboards, policymakers define thresholds, and reviewers monitor outputs. But in many real-world deployments, especially in public-facing systems like education, this model fails.

Why?: Because oversight becomes disconnected from those it most affects.

Students, teachers, and parents are often left out of monitoring loops, even though they are the first to experience misclassifications, unfair treatment, or missing context. When they try to intervene, they find that system logic is opaque, model decisions are sealed, and no one is listening downstream.

This section explores what changes when oversight is co-designed with the people on the ground.

Trust isn’t just built from above, it’s validated from below.

Case Revisited: Ofqual Grading Algorithm (UK, 2020)

As introduced in Chapter 3, the UK’s Office of Qualifications and Examinations Regulation (Ofqual) implemented an algorithm to determine A-Level grades after COVID-19 exam cancellations. The system relied on statistical standardization rather than individual performance, leading to the downgrading of nearly 40% of students, disproportionately from under-resourced schools.

What Chapter 3 examined as a risk design failure also stands as a clear case of oversight exclusion.

  • Teachers, the most qualified evaluators of student potential, had no visibility or override capacity.
  • Students were given no explanation or appeal route for algorithm-generated results.
  • The public discovered system-wide harm only after it had already been deployed at scale.

The system did not just misjudge students. It silenced the people who could correct it.

The public outcry was swift. Within a week, the algorithm was withdrawn, and students were awarded teacher-assessed grades. Parliamentary review and national inquiries followed. But the lesson was lasting: oversight cannot succeed when those closest to the system are locked out of it.

Real-World Follow-Up: School Risk Scoring Tools (UK, US, India)

In 2021, several school districts in the UK, US, and India began using AI-based tools to predict student performance, recommend support interventions, or flag dropout risk. The models often used attendance data, prior grades, behavioral records, and demographic features.

While the goal was early support, in many cases:

  • Students were never told how their risk level was calculated
  • Teachers could not challenge or override low-confidence predictions
  • Parents received automated letters without understanding the reasoning

In one district, students discovered that a single missed assignment disproportionately affected their predicted outcome, due to a weighting formula only visible to the vendor1.

Trust collapsed, not because the model was wrong, but because there was no interface for contestation, correction, or conversation.

Table 55: Comparing Oversight Failures in Education AI

Case Example Oversight Failure Type Who Was Excluded Outcome
Ofqual Grading Algorithm No appeal, no override, no explanation Teachers and students Public backlash, policy reversal
School Risk Scoring Tools No review, no transparency, algorithmic overreach Students, teachers, parents Erosion of trust, disengagement, error amplification

These cases show that even well-intended AI can erode legitimacy if those most affected are excluded from oversight.

Co-Designing Oversight in Education AI

A growing movement among educators and technologists now promotes participatory oversight models, where affected users can:

  • View and challenge the reasoning behind predictions
  • Submit contextual explanations (e.g., why an assignment was missed)
  • Collaborate in defining what “risk” means in their school’s context

Table 56: Participatory Oversight Benefits in Education AI

Co-Design Feature Benefit
Student-facing dashboards Enables learners to see how they’re assessed and offer corrections
Teacher override tools Allows human discretion to balance algorithmic suggestion
Explanation previews Helps teachers explain model results to families
Feedback integration Encourages schools to tune the model based on lived experience

These design choices don’t reduce performance. They improve legitimacy.

People don’t need perfect models, they need models that respect their ability to respond.

Institutional Frameworks Supporting Participatory Oversight

The UNESCO Recommendation on the Ethics of AI explicitly calls for “public participation in the oversight of algorithmic systems in education, healthcare, and justice”2. It emphasizes that oversight without participation often reinforces existing inequality.

In the EU, the AI Act (Recital 70) supports citizen-facing transparency rights, especially in high-impact contexts like education3.

Together, these frameworks mark a shift: oversight is no longer just a technical or legal function. It is a civic relationship between systems and the people they affect.

Thinkbox

“If you’re building AI for people, they deserve a seat at the table.”
A 2022 OECD report on trustworthy AI in education found that AI tools perform better and are trusted more when co-designed with educators and learners. Participatory design not only improves usability, it improves oversight legitimacy4.

Bibliography

  1. Perrotta, C., & Selwyn, N. (2022). Algorithmic Scoring and Student Risk in Education Systems. British Journal of Sociology of Education, 43(1), 35–50. https://doi.org/10.1080/01425692.2021.1990017 ↩

  2. UNESCO. (2021). Recommendation on the Ethics of Artificial Intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000380455 ↩

  3. European Union. (2024). EU Artificial Intelligence Act – Final Text (Recital 70). https://artificialintelligenceact.eu/the-act ↩

  4. OECD. (2022). AI and the Future of Skills: AI and Education – Trustworthy Use. https://www.oecd.org/education/trustworthy-ai-education.htm ↩