LONDON - Algorithms increasingly make decisions in the healthcare space, playing a big role in determining health outcomes. They can determine the precise dose of medicine for an individual patient or spot an anomaly on a scan that requires further investigation, sometimes better than an expert human eye.

In the discourse around the use of artificial intelligence (AI), we often hear that humans must remain in the loop. This human oversight element is, e.g., at the heart of the requirements for high-risk AI and conformity assessments in the recent European Union AI Act. The draft legislation would mandate that high-risk AI must allow for its oversight by humans to offset risks. A good example of high-risk AI is when algorithms determine the allocation of public services - who gets what.

But what does human oversight really mean, and how does it work in practice? This is a fundamental question, not only because it cuts to the heart of the human-machine relationship. Most importantly, it serves as a prime example of how a concept that looks wholesome could really be the cheese in a giant mousetrap to legitimize the lack of accountability around AI.

The Context

Algorithmic systems contribute ever more to decisions on health, public administration, allocation of benefits, or even whether an immigrant gains refugee status, citizenship, or neither. Concern for human rights, privacy and non-discrimination has led to a global movement toward ethics. This has spurred the production of numerous ethical guidelines and policy recommendations which are supposed to analyze, prevent, and mitigate the risks or collateral damage of an algorithmic society.

Insisting on the appropriate degree of human oversight is widely seen as a key policy toward this end and lies at the heart of society’s calls for procedural safeguar