Background

The rapid developments resulting from artificial intelligence (AI) will have enormous implications for public administrations. AI systems are already being developed or introduced to help public administrations improve the quality and efficiency of services they provide. In addition to tasks such as sorting and classification, translation and editing, and research, AI will also play a role in more complex decision making by analysing data or even suggesting solutions or policy options. AI also poses major challenges, when it comes to the accuracy of results, potential bias, explaining outcomes (‘explainability’) and human oversight, not least when it is or will be used in decision-making processes that affect the public directly.

Members of the European Network of Ombudsmen have already inquired into issues resulting from the use of AI by public administrations, such as the absence of clear rules governing the use of AI in the public sector[1], significant algorithmic errors and bias[2], explainability and effective human oversight on the input of AI to decisions taken towards individuals.[3] The European Ombudsman has drawn on this important work in the course of this initiative.

The strategic initiative

The Ombudsman opened this strategic initiative in March 2024 to obtain more information about the Commission’s practice and plans to use AI in its decision making.[4]

Shortly before that, the Commission published a communication ‘Artificial Intelligence in the European Commission (AI@EC)’ [5], which set out its approach to using AI. The Commission currently identifies areas for AI use with potentially the biggest positive impact on its staff’s daily work, such as reducing repetitive and time-consuming tasks, but also equipping them with tools to improve policy design and facilitate policy implementation. [6] The Commission also confirmed its commitment to use only lawful, safe and trustworthy AI systems.[7] Moreover, it shared information about AI systems that it already uses, develops, or plans to develop or acquire in the future.

The Ombudsman sought more detailed information about three systems listed in the communication which, in her view, may have a direct impact on the right to good administration, by influencing how the Commission interacts with, and makes decisions towards, individuals, companies, and organisations. These systems concerned:

• the analysis of feedback from the public,
• the management of competition cases, and
• the handling of infringement complaints.

The Ombudsman asked about specific functionalities of these tools and general risks to good administration that can result from their use. She also asked about the Commission’s procedure for deciding on the use of these tools, the transparency of this procedure, the involvement of stakeholders from outside EU administration, as well as the explainability and human oversight of these tools.[8]

While the Ombudsman’s initiative was ongoing, the European Parliament and the Council adopted the AI Act[9], which governs how AI systems in the EU are developed and used.[10] The Ombudsman also sought information from the Commission regarding how this will affect the Commission’s own development and use of AI.

The Commission provided a written reply to the Ombudsman’s questions on 24 July 2024.[11] On 24 September 2024, representatives of the Ombudsman met with representatives of the Commission to discuss the written reply and ask follow up questions.[12]

Information provided by the Commission

The Commission restated its commitment to ensuring a “safe, transparent and human-centred use of AI in its work”. The Commission is preparing internally for the implementation of the AI Act.

As regards transparency of the AI tools developed and used by the Commission, the Commission mentioned several ways in which it can make information about such tools available to the public. However, the Commission decides on publishing this information based on the “maturity status” of the systems.

Although the Commission acknowledged that a public consultation could provide valuable insights on its plans to develop or use specific AI tools, it emphasised that it is not required to carry out any external consultation in taking decisions about using technology to enhance its operations.

As regards the rules applicable to the AI tools developed and used by the Commission, the Commission representatives said during the meeting with the representatives of the Ombudsman that the AI@EC Communication is the main framework. This Communication refers to “operational guidelines on the development and use of lawful, safe, and trustworthy AI”, which are to be set up, and to the ‘Ethics guidelines for trustworthy AI’ developed by the High-Level Expert Group on AI.[13] The Commission representatives said that AI systems which are not deemed ‘high-risk’ will not necessarily have to comply with the AI Act’s requirements for high-risk systems. At the same time, the Commission is developing internal guidelines to facilitate compliance with the AI Act.

The three AI systems about which the Ombudsman requested more information are still in the development or procurement phase. The Commission will decide on their use only after the implementation of all technical features and a period of testing by selected staff are completed.

• The system for analysis of feedback from the public is envisaged to assist the Commission in analysing hundreds of thousands of contributions from the public made on the ‘Better Regulation - Have you say’ portal. The tool will summarise the contributions, search through them by topics or key phrases, and enable ‘sentiment analysis’ (how contributors feel about the Commission’s proposals).
• The Commission intends to procure a ‘competition case management’ system from an external provider to support its case handlers in analysing voluminous case files in competition cases. The Commission said such tools were already widely used by other competition authorities around the world, which all face the challenge of thoroughly analysing, in a very limited time, hundreds of documents obtained from companies.
• ‘Complaints Handling with AI’ (CHAI) is envisaged to support the Commission in handling thousands of complaints per year, brought by citizens and companies, regarding infringements of EU law by national authorities. The tool is supposed to assist in the search of relevant sources and information and in drafting replies with generative AI. The Commission stressed that the case handlers would remain in full control of the draft replies to the complainants.

The Commission representatives emphasised that the Commission’s AI systems are supposed to support but not replace human decision makers.

Moreover, while the Commission is still reflecting on the risks posed by AI systems, including risks to the fundamental right to good administration, the Commission is committed to prevent these risks by:

• following relevant laws and rules, including the AI Act and data protection laws, and by creating new internal guidelines;
• undertaking technical solutions to ensure equal treatment and non-discrimination in the use of AI;
• defining standard procedures for overseeing AI;
• designing AI tools to support objective and impartial decision-making;
• putting in place mechanisms ensuring the transparency and explainability of AI, including a mechanism for handling inquiries and feedback related to AI tools;
• training staff on how to use these tools and make them aware of their capacities and limitations.

Good administrative practice in the use of AI

Transparency and public participation in the Commission’s decisions about developing and deploying AI systems

Transparency is key in fostering the public’s trust that the EU administration will use AI responsibly and will remain accountable for its decisions made with input from AI. In this context, the Ombudsman welcomes the fact that the Commission has proactively published the Communication ‘AI@EC’ in which it has set out its approach to developing and using AI. It is particularly commendable that the Commission has listed the AI systems it already uses, the systems it is developing and intends to develop, and that it has made public more detailed information about these systems. The Ombudsman encourages the Commission to provide periodic updates of this communication in order to maintain the high level of proactive transparency in this area. The Commission could also consider making available a register of its AI systems.[14]

The Commission stated that it is not required to carry out “external consultations” regarding its decision making on how it uses AI. The Ombudsman understands this position in the current context in which the AI tools envisaged by the Commission do not lead to ‘full automation’ or perhaps even ‘significant automation’ of its decision-making processes. The Commission emphasised that the tools it envisages will only support its staff.

However, depending on their functionalities, the AI tools discussed in the context of this strategic initiative, such as competition case management or CHAI, may in fact ‘semi-automate’ the Commission’s decision making in the area of competition law or infringement complaint handling, which means they can influence the choices made by human case handlers or decision-makers. The AI Act acknowledges that some AI systems may “materially influence” the outcome of decision making.[15] Moreover, as AI is a rapidly developing technology, it is possible that, in the future, the Commission will seek to introduce other powerful AI tools that will influence other decision-making processes. It is possible that the influence of AI cannot be fully addressed with human oversight, as it is uncertain whether human oversight can ever be comprehensive in these situations. Humans responsible for monitoring the systems cannot be expected fully to repeat the reasoning process already carried out by AI and to ensure its correctness, as they would lose any efficiency gains.

In this context, the Ombudsman encourages the Commission further to reflect on how to ensure the transparency of powerful AI it may envisage to use in the future, including what information it makes available to the public. The Ombudsman believes that the Commission should also not rule out the possibility that, in the future, it may be necessary to carry out some form of public consultation on its decision making before the Commission’s projects reach full maturity. The public should have a say on whether the EU administration can and should introduce powerful AI tools that will influence specific decision-making processes. Public debate on these matters may also be instrumental in identifying and solving potential issues.

Also, the Commission should consider the limits of its internal autonomy to use AI in its decision making, complaint handling or other forms of interactions with the public. Deploying more powerful AI systems, which may “materially influence” the outcomes of decision-making, may imply a delegation of power to autonomous AI systems. Under EU case law, the EU institutions can delegate to other entities only technical assessments, or in the words of the Court of Justice, “clearly defined executive powers the exercise of which can... be subject to strict review in the light of objective criteria determined by the delegating authority”. A “discretionary power implying a wide margin of discretion”, which means making policy choices, cannot be delegated.[16] Technical powers delegated by EU institutions to other entities must be precisely delineated in legislative provisions and be subject to an effective oversight.[17] The Commission, as well as other EU institutions and bodies, should consider how these limits and acceptable forms of the delegation of power may affect their use of more powerful AI systems that support their decision-making or policy-making, given the autonomous, that is, not entirely predictable and explainable operation of such systems.[18] In particular, they should consider whether the tasks delegated to AI are indeed technical, “clearly defined”, and “subject to strict review in light of objective criteria”.

Moreover, under the AI Act, public authorities and some other bodies exercising public tasks are obliged to carry out a ‘fundamental rights impact assessment’ prior to putting to use high-risk AI systems. The AI Office, which is part of the Commission, is expected to develop a template or a questionnaire for this impact assessment.[19] This assessment must also encompass the fundamental right to good administration.[20] While this requirement applies only to high-risk systems, the Commission should consider applying it more broadly to any AI systems that it will introduce in the future to support its complaint handling or decision making affecting the public.[21] The impact assessment process should be transparent and the Commission should also consider stakeholder or public consultation, which could help ensure public trust that it will use AI transparently and in full respect of fundamental rights.[22]

The AI Act and the principles of good administration

The Ombudsman welcomes measures undertaken by the Commission to ensure that its use of AI complies with the AI Act, notably by introducing internal guidelines concerning prohibited AI practices, including a questionnaire distributed among the Commission’s departments, and internal guidelines for identifying high-risk systems that the Commission is still working on. The Ombudsman believes that the latter guidelines in particular will be crucial for the implementation of the AI Act within the Commission, as most of the substantive requirements laid down in the AI Act apply to ‘high-risk systems’.[23]

At the same time, the Ombudsman notes that the AI Act’s requirements regarding the development and deployment of high-risk AI systems largely correspond and give effect to general principles of good administration that the EU administration must uphold when using any AI systems, not only those that are deemed ‘high-risk’ under the AI Act.

Requirements under the AI Act (for high-risk systems)[24]	Principles of good administration
- maintain risk management solutions for high-risk AI systems - draw up and keep up to date technical documentation concerning these systems - enable relevant events to be recorded while the systems are operating (through ‘logs’) - ensure cybersecurity	- exercise diligence, care and caution[25]
- train the high-risk AI systems on relevant, representative, error-free, and bias-free data - ensure that the outputs of the AI systems are accurate and robust	- take decisions based on complete, accurate, reliable, and consistent evidence[26]
- ensure that the outputs of the AI systems can be interpreted by humans	- give reasons for any decisions[27]
- ensure that the outputs of the AI systems are subject to human oversight	- assume full responsibility for decisions (no unilateral and unlimited delegation of decision making to third parties[28])

The requirements and general principles mentioned above also correspond to general principles of ethical AI, as developed by the High-Level Expert Group on AI, to which the Commission referred in its Communication ‘AI@EC’ as the basis for its future operational guidelines. These ethical principles include accountability, technical robustness, transparency, and human oversight.[29] Moreover, they further correspond to the requirements laid down in the Council of Europe’s Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law,[30] which the Commission has already signed.[31]

The Commission, as well as other EU institutions and bodies, should always keep these general principles of good administration in mind, given that the notion of ‘high-risk’ AI systems under the AI Act covers only selected AI systems developed and deployed by the EU administration, and not all its AI systems by default. According to the AI Act, AI systems used by public administrations may be deemed ‘high-risk’ if their use corresponds to an exhaustive list in Annex III of the AI Act.[32] Annex III lists systems used in the areas of education, employment, essential private and public services and benefits, law enforcement, migration and asylum, administration of justice (judicial authorities) and democratic processes (electoral authorities), among others.[33] Only some AI systems used by the EU administration may fall under these categories. In any case, a system that “does not pose a significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making” is not considered high-risk. This is especially the case if the system “is intended to perform a narrow procedural task” or a “preparatory task”.[34] A self-assessment by the system’s developer and the entity deploying the system is therefore crucial in practice.

As far as the use of AI by the EU administration is concerned, the application of many[35] of the AI Act’s concrete requirements may also turn out to be limited in other ways:

· many requirements concern the development of the systems only, rather than their use;

· the requirements will in principle apply only from 2030;[36] and

· specific systems used in decision-making processes may not fall under the definition of AI contained in the AI Act.[37]

The Commission has not yet assessed which of the AI systems it uses (including those discussed in the context of this strategic initiative) fall under the high-risk category and, consequently, whether they will have to comply with most of the requirements of the AI Act mentioned above, apart from the requirement of transparency (making people aware when they interact with AI), which appears to apply broadly.[38]

The Ombudsman takes the view that, in its use of AI systems to support - in particular -decision-making, complaint-handling processes, and other interactions with the public, the Commission is bound to uphold the principles of good administration, regardless of whether the requirements of the AI Act apply to those systems. The Ombudsman encourages the Commission further to reflect on how to give practical effect to the principles of good administration in its development and use of AI systems. The Commission, as well as other EU institutions and bodies, should take into account that the concrete requirements of the AI Act may give effect to the general principles of good administration and, consequently, may need to apply regardless of whether an AI system is considered ‘high-risk’ under the AI Act.

How to ensure good administrative practice in the use of AI - Observations of the Ombudsman

1. ‘Compliance check’ of AI systems regarding the principles of good administration

The Commission said that its processes for developing or purchasing any IT systems, including AI systems, include a ‘compliance check’ with all legal requirements, including the AI Act. The Ombudsman believes that this check must also encompass the principles of good administration, especially if the AI Act’s concrete requirements are inapplicable. Indeed, the EU administration must ensure that all AI systems that support its work and decision making produce accurate and fair results, that their use is traceable and can be explained or at least understood by the public, and that they enable the administration to provide reasons for decisions, exercise effective human oversight and assume full responsibility for any decisions.

2. Ensuring effective oversight of AI

The EU administration should ensure that decision making is not delegated to AI systems but, rather, that these systems are used as tools to support decision-making processes. AI systems can play an important role in researching and drafting, but human decision makers must remain fully responsible for the final decisions. This means that decision makers cannot merely rubberstamp proposals made by AI systems, but must verify them meaningfully.

In the context of this strategic initiative, the Commission emphasised that none of the AI systems that the Commission uses or intends to use will replace human decision makers. Commission staff will ultimately be responsible for the accuracy of any outputs, for instance, in the case of the AI tool used for handling infringement complaints.

However, merely providing the possibility for human decision makers, such as individual case handlers, to depart from suggestions of AI systems, may not be sufficient to ensure effective human oversight. According to the AI Act, entities deploying high-risk systems should “assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support”.[39] Consequently, EU civil servants responsible for the human oversight of AI must be proactively enabled and encouraged to exercise this oversight. The relevant staff members must receive adequate training about risks and errors produced by AI systems and how to critically examine and verify the outputs of such systems. Moreover, internal working procedures should make it clear that staff responsible for the oversight of AI systems are encouraged to use their initiative to depart from the AI system’s suggestions in specific cases. Such effective human oversight is crucial not only to detect errors but also to ensure that the administrative practice can be changed and improved.

3. Full transparency towards the public

Full transparency is key to fostering public trust that the EU administration will use AI responsibly. For systems used to produce decisions and replies to the public, such as the CHAI, the Commission should consider whether it may need to proactively inform the addressees that the replies have been prepared with AI assistance, and inform them about the nature of this assistance (such as whether it was research and/or drafting related). This practice would correspond to requirements contained in the AI Act for high-risk systems, which provide for “the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure”[40] and the obligation on entities deploying AI systems to “inform the natural persons that they are subject to the use of the high-risk AI system”.[41]

4. Voluntary code of conduct on the use of AI systems

The Commission should consider adopting a ‘code of conduct for voluntary application of specific requirements’, as mentioned in the AI Act, for its AI systems that are not deemed high‑risk. Such a code might also form part of the “operational guidelines on the development and use of lawful, safe and trustworthy AI systems”, which the Commission intends to adopt.[42]

According to the AI Act, the AI Office, which is part of the Commission, is obliged to encourage and facilitate the drawing up of such codes of conduct intended to foster the voluntary application to AI systems not deemed to be high-risk of some or all of the requirements laid down in the AI Act towards high-risk systems.[43] Such codes of conduct may be drawn up by entities deploying AI systems, such as the Commission itself, with the involvement of interested stakeholders, civil society, and academia. They may cover a category of AI systems, taking into account the similarity of their intended purpose.

In this context, the Commission could consider whether AI systems that it uses to assist in interactions with the public, decision-making or complaint-handling procedures should follow some or all of the AI Act’s requirements, such as: having risk management systems, technical documentation, and logs; training the systems on relevant data and making sure they consider all relevant information in individual cases; ensuring the outputs of the systems can be explained to and/or interpreted by users, and are subject to human oversight. As already mentioned, when AI systems are used by public administrations, these requirements appear to correspond to and give effect to the principles of good administration.

Conclusion

This initiative has enabled the Ombudsman to gain insight into the use of AI by the Commission, and how this complies with the principles of good administration. She trusts that the observations contained in this note will help to ensure that the Commission complies with the principles of good administration while developing and using AI.[44]

On this basis, the Ombudsman closes this strategic initiative. She thanks the Commission for the information provided and invites it to take into account the observations made above. She looks forward to exchanging with the Commission on these matters in the future.

Emily O'Reilly
European Ombudsman

Strasbourg, 06/12/2024

 

[1] See an example from the Finnish Chancellor of Justice, https://oikeuskansleri.fi/en/-/automated-decision-making-in-kela.

[2] See an example from the Dutch National Ombudsman, https://www.nationaleombudsman.nl/affaire-kinderopvangtoeslagen-overzicht-nationale-ombudsman.

[3] See comments from the Estonian Chancellor of Justice shared at the 2023 conference of the European Network of Ombudsmen, https://www.ombudsman.europa.eu/en/multimedia/videos/en/4413.

[4] https://www.ombudsman.europa.eu/en/opening-summary/en/183503.

[5] Artificial Intelligence in the European Commission (AI@EC) Communication, https://commission.europa.eu/publications/artificial-intelligence-european-commission-aiec-communication_en.

[6] Artificial Intelligence in the European Commission (AI@EC) Communication, p. 2.

[7] Ibid, p. 4.

[8] See the Annex to the opening letter, https://www.ombudsman.europa.eu/en/opening-summary/en/183503.

[9] Regulation 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence, OJ L 2024/1689, https://eur-lex.europa.eu/eli/reg/2024/1689/oj.

[10] Regulation 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence, OJ L 2024/1689, https://eur-lex.europa.eu/eli/reg/2024/1689/oj.

[11] https://www.ombudsman.europa.eu/en/doc/correspondence/en/191562.

[12] https://www.ombudsman.europa.eu/en/doc/inspection-report/en/195011.

[13] https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai.

[14] The register is mentioned in the Communication ‘AI@EC’ at p. 5. The EU database of AI systems required by the AI Act will include mostly ‘high-risk’ systems. See, Article 49(3) of AI Act.

[15] Article 6(3) of AI Act.

[16] Judgment of the Court of Justice, Case C-270/12, UK v Parliament and the Council, https://curia.europa.eu/juris/document/document.jsf?text=&docid=146621&pageIndex=0&doclang=en&mode=lst&dir=&occ=first&part=1&cid=7732888, paragraphs 41-42.

[17] Ibid, paragraph 53.

[18] The experience of other members of the European Network of Ombudsmen, in particular the Finnish Chancellor of Justice, shows that it may be necessary for the legislature to specify the conditions under which decision-making processes within the administration can be automated using AI, https://oikeuskansleri.fi/en/-/automated-decision-making-in-kela. See, Article 298 TFEU.

[19] An impact assessment methodology is also developed by the Council of Europe. See, https://rm.coe.int/cai-bu-2022-03-outline-of-huderia-risk-and-impact-assessment-methodolo/1680a81e14.

[20] Recital 48 of the AI Act.

[21] Such impact assessment is also recommended by academics. See, https://www.europeanlawinstitute.eu/fileadmin/user_upload/p_eli/Publications/ELI_Model_Rules_on_Impact_Assessment_of_ADMSs_Used_by_Public_Administration.pdf.

[22] See the requirement of public consultation in the Council of Europe’s Convention on Artificial Intelligence in Article 19, https://rm.coe.int/1680afae3c.

[23] Chapter III of the AI Act.

[24] Articles 9-15 of the AI Act.

[25] Judgment of the Court of Justice of 16 December 2008, Case C-47/07 P, Masdar v Commission, https://curia.europa.eu/juris/document/document.jsf?text=&docid=76076&pageIndex=0&doclang=en&mode=lst&dir=&occ=first&part=1&cid=7730313, paragraphs 92 and 93.

[26] Judgment of 15 February 2005, Case C-12/03 P, Commission v Tetra Laval, https://curia.europa.eu/juris/document/document.jsf?text=%2522factually%2Baccurate%252C%2Breliable%2522&docid=49926&pageIndex=0&doclang=en&mode=req&dir=&occ=first&part=1&cid=7731828#ctx1, paragraph 39.

[27] Article 41(2), third indent, of the Charter of Fundamental Rights.

[28] Case C-270/12, UK v Parliament and Council, paragraphs 41 and 42.

[29] https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai, p. 14.

[30] See, especially, Article 8 - Transparency and oversight, Article 9 - Accountability and responsibility, and Article 12 - Reliability, https://www.coe.int/en/web/artificial-intelligence/the-framework-convention-on-artificial-intelligence.

[31] https://www.eeas.europa.eu/delegations/council-europe/european-commission-signs-historic-council-europe-framework-convention-artificial-intelligence-and_en?s=51.

[32] See also Article 6(1) of the AI Act.

[33] Article 6(2) of the AI Act.

[34] Article 6(3) of the AI Act.

[35] According to Article 50 of the AI Act, the requirement of transparency, understood as informing natural persons when they interact with AI, applies to all AI systems.

[36] Article 111 of the AI Act.

[37] The definition of AI does not cover systems “that are based on the rules defined solely by natural persons to automatically execute operations”. Recital 12 of the AI Act.

[38] Article 51 of the AI Act.

[39] Article 26(2) of the AI Act.

[40] Article 86(1) of the AI Act.

[41] Article 26(11) of the AI Act.

[42] As mentioned in its Communication ‘AI@EC’ at p. 4.

[43] Article 95 of the AI Act.

[44] In view of the supervisory and compliance tasks of the EDPS under the AI Act, the Ombudsman’s observations presented in this note focus on how the EU institutions and bodies can ensure respect for the principles of good administration in the use of AI.