What organizations need to prepare for now.

The AI Act, the first major legal framework for the use of AI in the European context, came into force on August 1, 2024. One principle that the regulation emphasizes is that of “human oversight”: actors who provide and use AI should be empowered to make informed decisions about its use. However, the mere involvement of natural persons in the use of AI is not enough, as the past has shown. In our article, we look at the famous case of the COMPAS Recidivism Algorithm and show its relevance for other AI application contexts. We highlight three challenges that will be crucial for the implementation of functioning human oversight in the sense of the AI Act: automation bias, the monitoring and testing of AI applications and the traceability of machine decisions.

To provide AI systems with a legal framework, the European Council finally approved the “Artificial Intelligence Act”i (AI Act for short) on May 21, 2024. Launched by the EU Commission in April 2021 and further developed throughout the last years, the AI Act defines a whole series of principles and measures designed to ensure the safe and responsible use of AI systems throughout their life cycle. The AI Act is a product-centric regulation and follows a risk-based approach (see Info Box 1: The AI-Act). These risk categories can be used to regulate which principles and measures to implement, and by which relevant actors. The list of principles and measures is long and includes for certain AI systems, amongst others, criteria for data quality, for documentation and labeling obligations, for accuracy, robustness, and cyber security, as well as for transparency.i

One principle that is also applied by the AI Act is the principle of “human oversight”. According to this principle, certain AI systems must be designed and developed in such a way that “that they can be effectively overseen by natural persons during the period in which they are in use,” including using “appropriate human-machine interface tools” (HMI) tools (p.196 of the AI Act).

The principle of human oversight was already part of the General Data Protection Regulation (GDPR) published in 2016.ii  The GDPR granted data subjects the right not to be subject exclusively to automated decisions regarding the processing of their personal data if these have legal or similarly significant effects. In accordance with this principle, various stakeholders—amongst others, providers and deployers—must now consider under the AI Act which measures, regulations, and processes to harness for such human supervision. The AI Act itself does not initially define their exact implementation in further detail.

The good news is that the idea of adding a human to an algorithm as a supervisor is not new. Often summarized under the term ‘human-in-the-loop,’ a variety of approaches have emerged over the past few years that can help to identify and counteract problematic decision-making tendencies in algorithm-based decision-making systems. The bad news is that the past has also shown that involving a human in machine-based decision-making processes does not automatically produce unbiased, unprejudiced decisions.

The COMPAS Recidivism Algorithm as an example of machine bias.

The famous case of the COMPAS Recidivism Algorithm in Broward County, Florida, provides a pivotal example.iii The COMPAS algorithm followed an individual risk scoring approach to calculate the probability that an offender standing trial would commit another offense before the court hearing or would reoffend after his or her conviction (i.e., become a recidivist). Judges consulted the algorithm when it came to deciding whether an offender should be granted the possibility of bail or whether it was necessary to impose preventive detention.

According to an analysis by the non-profit organization ProPublicaiv, however, this algorithm tended to falsely classify black defendants as future recidivists (“false positives”) almost twice as often as white defendants. At the same time, white defendants were more frequently misclassified as low risk (“false negatives”) than black defendants. However, these significant differences in the treatment of white and black individuals could not be explained by the defendants' prior convictions or the nature of the offenses for which they were arrested. This called into question the validity of the calculated score and led to worldwide discussions about the ethical, legal, and social implications of using such software outside the USA.

Finding clear causes for this discriminatory tendency of the COMPAS algorithm has proven difficult to date, partly because the company Equivant/Northpointe, which developed the algorithm, only publishes limited information about it. Although the institutions using the algorithm have full access to its structure, independent third parties or the wider public do not. Critics also emphasize that the traceability of the algorithm's decisions is made even more difficult by the risk model used, which is unnecessarily complex with 137 variables.v

Another distinctive feature about the data the algorithm used was that it excluded the sensitive data category “race,” according to which the algorithm apparently discriminated.vi Instead, it was based on variables intended to reflect the defendants' family and social environment and a psychological self-assessment, conducted using a questionnaire. However, skin color was not explicitly included as a variable in either data source.

This “bias” of the COMPAS algorithm seems to have intuitively favored white individuals. This is why the COMPAS algorithm entered the textbooks as a prototype for “machine bias”vii: It was no longer only natural persons who were able to make biased decisions, but also machines.

The human-in-the-loop approach as a panacea?

The COMPAS case is relevant in two respects for current considerations on how the human oversight required by the AI Act can be implemented. Firstly, it is an early example of a human-in-the-loop system: The scores the algorithm provided are made available to the responsible judges to support their decisions. They are not automated without human involvement. On the other hand, the case is one of many examples that show that a human supervisor alone cannot eliminate all the risks associated with the use of AI. In the case of COMPAS, it was humans who made the final decision, even trained judges who are obliged to be impartial. Nevertheless, the judges seemed to make biased and problematic decisions, and COMPAS continues to be used legally in many US states.viii, ix

What we learn from this: Data-based decision-making systems are here to stay. At the same time, however, they do not automatically make more accurate, objective decisions—even if humans are involved in the decision-making process. This shifts the focus away from merely supervising an AI's decisions by a human towards finding the right structures in which the interaction between human actors and AI is finely tuned to the respective application purpose, use case and context. This results in several fundamental challenges for applying organizations, explained below.

AI literacy: It takes more than just a human-in-the-loop.

In general, the users of AI systems are the most important influencing factor for the responsible use of AI. This is an aspect that AI has in common with topics such as data protection or cyber security. Also here, it is crucial that users are sensitized and reflect on decisions in their day-to-day work before addressing these issues. The AI Act provides a framework that prescribes guidelines for the responsible use of AI. However, we have seen from the example of human oversight that the implementation of its principles will still raise many questions and that relevant best practices are only just emerging at best.

Until this changes, all relevant stakeholders must be enabled to make informed decisions about the use of AI. The AI Act refers to this as “AI literacy”i. It comprises the interplay of technical knowledge, experience, education and training, the context in which the AI systems are to be used, and the persons or groups of persons for whom the AI systems are to be used. By considering these aspects, AI literacy should give all relevant stakeholders the necessary concepts to make informed decisions about the use of AI systems.

Organizations that make use of AI are now required to equip their staff with the right skills, resources, and competencies to handle AI responsibly in their day-to-day work. This applies to all AI systems, even if they are not assigned to one of the explicit risk levels in the AI Act and therefore must meet predefined requirements that go beyond transparency and information obligations. Only then can the human oversight required by the AI Act succeed. 

Info Box 1: The AI-Act

The Artificial Intelligence Act (in brief: AI Act) is the first product regulation on artificial intelligence that creates a legal framework for the development and use of artificial intelligence (AI) in the EU.i The AI Act also affects companies and organizations outside the EU if they distribute AI systems in the EU or if output from their AI systems is used in the EU.xvii The regulation defines key terms such as “artificial intelligence” and takes a risk-based approach when it comes to regulating AI systems through appropriate measures. The AI Act can be divided into four risk levels for AI systems:

Unacceptable Risk

With this risk level, the AI Act prohibits several AI systems whose risks are unacceptable in its view (see Chapter II, p.171 of the AI Act). These few systems that compromise fundamental rights include, for example, systems that exploit “vulnerabilities of a natural person”, influence people’s behavior using “subliminal techniques” or “purposefully manipulative or deceptive techniques”,” or that evaluate natural persons’ "social behavior” (i.e., social scoring) (see also pp. 24, 29, 42, 152 of the AI Act).

Systemic Risk

This risk level concerns systems of so-called “General-purpose AI models” (GPAI) and their “high-impact capabilities” (see Chapter V, p.259 of the AI Act). These general AI models, which also include Large Language Models (LLMs) such as ChatGPT, can perform a variety of tasks and serve as the basis for many AI systems.

High Risk

This risk level is the main focus of the AI Act (see Chapter III, pp.178-256 of the AI Act). It results in extensive obligations for AI systems regarding documentation, transparency, and human oversight. A whole range of systems fall under this risk level (see Annex III, pp.1-6), including those related to the “biometric verification” and “categorization” of persons, “critical infrastructure”, “employment, workers management and access to self-employment”, and the “access to and enjoyment of essential private services and essential public services and benefits”.

Other Considerations

Certain AI systems pose minimal or no risk to the rights or safety of natural persons. The AI Act provides for minimal obligations for such AI systems. For some AI systems, there are additional transparency obligations under the AI Act (see Chapter IV, p. 256 of the AI Act). These relate to all AI systems that interact with people, generate, or manipulate content and are used to recognize emotions or associate social categories based on biometric data. Examples include spam filters, video games, search algorithms, the infamous deep fakes and chatbots. It must be made clear to users that they are interacting with an AI system.

Would you like more information?

If you want to get more information about this insight please get in touch with our experts who would be pleased to hear from you.