Background

1. In recent years, artificial intelligence (AI) has permeated every aspect of our lives, from the trivial to the highly consequential, such as decision making related to medical diagnoses or social security benefits.

2. The European Ombudsman has been following this topic closely and, in March 2021, hosted a webinar on AI and e-government in public administration for the European Network of Ombudsmen (ENO). Many of the questions that AI raises pertain to core areas of an ombudsman’s work, such as transparency, accountability of decision-making, ethics and fundamental rights. Some ombudsman offices at national level have already handled complaints on and issued guidance^[1] related to the use of AI.

3. In 2021, the European Commission presented a proposal for a new EU regulation governing AI: the draft ‘AI Act’.^[2] The proposal is of immediate interest to the European Ombudsman and ENO members. It envisages that authorities will be designated or established at national level to ensure the implementation of the regulation. When EU institutions, offices, bodies and agencies fall within the scope of the regulation, it is envisaged that the European Data Protection Supervisor (EDPS) will act as the relevant supervisory authority. The proposal aims to ensure a high level of protection of fundamental rights, including the right to good administration, as well as compliance with principles of good administration.^[3] 

4. In this context, in June 2021, the Ombudsman wrote to the Commission and to the EDPS requesting meetings to learn more about the proposal, and to discuss how the future rules would operate with regard to the EU administration and public administrations in general. The purpose of this strategic initiative was to keep ENO members informed about these important developments at EU level and to help prepare the European Ombudsman’s Office for potential future work in this area, notably in terms of dealing with possible complaints alleging maladministration by EU institutions.^[4]

The proposed AI Act and standards related to transparency, fundamental rights and principles of good administration

5. The proposal for an AI Act aims to lay down uniform rules for AI systems in the EU market. It puts forward rules applicable to the whole AI lifecycle including placing on the market, putting into service and using AI systems.

6. The proposal for the AI Act uses a risk-based approach, meaning that regulatory intervention is needed only when necessary depending on the level of risk to safety and fundamental rights that an AI system is likely to pose. The proposal classifies AI systems in four categories:

• unacceptable risk (there are four prohibited AI practices, including exploitation of vulnerabilities of certain people),
• high-risk AI systems (permitted but subject to compliance with requirements),
• AI posing certain transparency-related risks (permitted subject to informationobligations), and
• AI posing minimal or no risk (permitted without additional restrictions but possible compliance with voluntary codes of conduct).

7. Under the proposed AI Act provider obligations include registering stand-alone AI systems in a public EU database and reporting to market surveillance authorities serious incidents and malfunctions that can pose risks to fundamental rights.

8. User obligations will include ensuring human oversight when using AI systems (essential for public authorities) and informing the provider or distributor about any serious incident or any malfunctioning.

9. The draft AI Act includes various transparency provisions, notably the obligation to inform users proactively that they are interacting with an AI system. These transparency obligations are envisaged as complementary to those in existing legislation, such as in consumer protection or data protection legislation at EU and national level.

10. In order to address, in particular, opacity and ensure transparency towards the wider public, the proposed AI Act includes provisions on:

1. transparency and traceability of high-risk AI systems

2. access rights for competent supervisory authorities

3. transparency towards affected people (people must be informed when they are using certain AI systems, including when not classified as high-risk)

4. public oversight: EU-wide publicly accessible database of stand-alone high-risk AI.

11. When individuals feel that their rights have been infringed by an AI system, it is envisaged that they will be able to seek redress through the market surveillance authorities and/or other existing authorities, such as ombudsman institutions, data protection authorities, consumer protection authorities, anti-discrimination authorities, and so on. These authorities would then have the power to request access to all relevant information concerning the AI system in question. Being able to contact existing institutions means that the public will not need to address new and unknown authorities, and that there is a consistent approach to dealing with issues. Where necessary, the relevant authorities will have the powers to analyse the AI system in question, even including the source code.

12. As to the governance structure of the Act, it is based on a system covering national and EU level. The national level would play a key role in enforcement through national market surveillance authorities. The EU level would coordinate implementation and exchange information through an EU AI Board. There would also be a Commission expert group for technical and scientific advice where all relevant stakeholders are represented (civil society, academia, businesses, etc.).

13. Under the draft AI Act, ombudsman offices could potentially be identified by Member States as the relevant national body for guaranteeing fundamental rights protection. These bodies will have the right to access essential documents and information, and be able to avail of new mechanisms for cooperation with market surveillance authorities.

14. In terms of enforcement, it will be important to consider how national authorities will respond to their new roles under the draft AI Act, which may result in different types of national authorities having overlapping or interrelated powers and mandates. For instance, the draft Act includes a proposal (in Article 63) that, for AI systems used for law enforcement and migration, asylum and border control management, the market surveillance authority will be the relevant authority for overseeing data protection. If there are cases where different national authorities have overlapping or interrelated powers, these authorities should closely cooperate to ensure consistent enforcement.

15. The proposal is currently being negotiated by the Council and the European Parliament. Once adopted, there would be a two-year transition period, after which the AI Act should become directly applicable in its entirety and obligatory for operators.

AI and the EU administration

16. EU institutions, bodies, offices and agencies will be covered by the proposed AI Act when they are acting as providers or users of AI systems.

17. When EU institutions, bodies, offices and agencies are developing the systems in-house (not ‘buying them off the shelf as a finalised product’ from the market), they will be considered ‘provider’ when putting those systems into service for their own use. They would accordingly have to comply with the relevant requirements, including data quality, transparency and human oversight.

18. If a public authority is not developing the system in-house, but buying it ‘off the shelf’, it would be deemed a user and have to comply with all ensuing obligations, including transparency obligations towards affected persons.

19. AI is already being used by the EU administration in different areas, for example translation: the platform of the Conference on the Future of Europe includes a translation tool^[5]. Other examples include the early warning and preparedness system of the European Union Agency for Asylum and the tools used by the European Border and Coast Guard Agency (Frontex) for risk profiling, for example of vessels.

20. Some examples of AI applications being planned by the EU administration include:

• A project by the Commission’s Directorate‑General for Communications (DG COMM) to enhance interaction with citizens on social media.
• A project by DG COMM and Europe Direct to set up a knowledge repository of the information that staff members need in order to reply to citizens’ questions about their rights, in particular linking and retrieving information that could be relevant. AI might also be used for the classification of questions.
• Eurostat is considering using virtual assistants (chat bots) in relation to its communication about statistical information.
• The EU Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) is planning to boost the performance of its databases, in particular the Schengen Information System, through AI, including improving the accuracy of biometric data.
• The European Personnel Selection Office is looking into the possibility of using AI for recruitment activities.

21. In line with its upcoming obligations under the proposed AI Act, the Commission has already started work to develop a code of conduct and guidelines for the development and use of AI.

The EDPS and the proposed AI Act

22. Based on the current text of the proposal for an AI act, the EDPS would have three main roles:

1.  Competent supervisory authority: the EDPS will ensure that EU institutions comply with the AI Act, which includes monitoring and enforcing the Act.

2. Notifying body: at least for some cases, the EDPS will conduct conformity assessments for EU institutions that will develop high-risk AI systems.

3. Market surveillance authority: it is not yet clear what this will entail. The EDPS has expressed its doubts in its EDPB-EDPS Joint Opinion on the AI Act and asked the Commission for clarifications as regards what the AI market for EU institutions, bodies, offices and agencies will comprise, and if its surveillance will concern solely EU institutions or also third parties and private companies providing AI systems and services.

23. The EDPS will also be part of the new European AI Board, which will issue opinions and recommendations on AI matters. In broad terms, this will entail preparatory work to keep up with the latest updates on AI technology and to follow standardisation processes.

24. The EDPS considers that further clarifications regarding the AI legislation are needed in order to specify the extent of its new roles and to see how the AI Act will affect and interplay with the existing data protection legislation.^[6]

25. According to the EDPS, there is room for improvement in the proposed AI Act in terms of the obligations of transparency towards the user, that is informing the public as to when and how AI systems are used.

26. The EDPS is already conducting supervision activities as some EU institutions have started developing machine-learning systems. The EDPS is performing audits and asking for information on how EU institutions are developing machine-learning models using personal data. The EDPS is also consulted by institutions that are developing or procuring AI systems, in order to help them properly consider data protection already at the design stage, identify possible risks for the rights and freedoms of data subjects and mitigate them in an adequate way.

Conclusion

27. The principles of good administration include service-minded interaction with individuals, reasonableness and proportionality. They imply explaining decisions in a clear way and being empathetic with those affected by such decisions. In other words, good administration implies being humane and human. When humans are removed from the equation of service delivery, it is clear that there may be challenges. The ever-increasing shift to e-government and the use of artificial intelligence by public administrations will have an enormous impact on the work of ombudsman institutions, which will be on the frontline of dealing with the implications of this shift.

28. Research conducted by the EU’s Fundamental Rights Agency (FRA) suggests that, in the majority of cases, AI is used for efficiency purposes.^[7] This should, of course, not be the sole focus. AI affects not only the fundamental rights to privacy and data protection but also the fundamental right to equality and non-discrimination (with the rights of special or vulnerable groups often absent from the discussion) and the fundamental right of access to effective remedies (including the awareness that people have the right to complain). Decisions need to be fair and transparent, and pathways to challenge decisions need to be available and accessible.

29. The FRA has produced a checklist of key considerations to help businesses and administrations respect fundamental rights when using AI.^[8] This is a useful resource for ombudsman offices. The conclusions of the 2022 ENO conference^[9] included the proposal to draw up a list of good practices for public administrations using AI. Such a list could be valuable, especially given that it will still take several years before the standards and safeguards proposed in the draft AI Act come into force, while in the meantime public administrations already use AI and have imminent plans to roll out more AI to support their functions.^[10]  The European Ombudsman will reflect on how to develop such a list, in cooperation with ENO members.

30. The emergence of AI also represents a human resources challenge. Both public and private sectors have started to search for experts in the AI field, and a shortage in experienced professionals could become a challenge for the public sector in particular. This challenge will also apply to ombudsman offices, and the European Ombudsman will reflect on how to address this, including within the ENO.

31. Assuming the AI Act is adopted, there will be ample room for further cooperation in the ENO. The governance structure of enforcement, implementation and exchange of information, proposed in the draft AI Act, is based on a system of interaction between national and EU bodies. Ombudsman offices could also be identified by Member States as national bodies supervising fundamental rights protection. The European Ombudsman will continue to assess what practical steps could be taken within the ENO in this area.

32. This strategic initiative has demonstrated that, in the future, there are likely to be open questions regarding transparency towards the public (for example, concerning the proposed public EU-wide register of stand-alone AI systems). It is also to be expected that there will be difficulties as regards accountability: meeting the requirements for human oversight; transparency of the metrics used in AI testing and validation processes; how traceability will be ensured; and compliance with the obligation to keep logs and documentation.

33. This strategic initiative enabled the European Ombudsman and the ENO to gain a good insight into the future EU regulatory landscape for AI, into its deployment by the EU administration and across the public sector more generally. In addition to continuing to share relevant information with the ENO, for which the digitalisation of public administrations is a priority, the European Ombudsman will use this strategic initiative to inform possible future work in this area and to prepare for potential future complaints.

 

 

 

[1] An example of such guidance is the report of the Dutch Ombudsman, ‘The citizen is not a dataset’: https://www.nationaleombudsman.nl/nieuws/onderzoeken/the-citizen-is-not-a-dataset

[2] Proposal for a regulation of the European Parliament and Council laying down harmonised rules on artificial intelligence (AI Act) https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206

[3] Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206

See also https://digital-strategy.ec.europa.eu/en/policies/european-approach-artificial-intelligence

[4] The full text of the meeting reports can be consulted here:

https://www.ombudsman.europa.eu/en/doc/inspection-report/en/149338

https://www.ombudsman.europa.eu/en/doc/inspection-report/en/154290

[5] More information on the platform: https://ec.europa.eu/commission/presscorner/api/files/attachment/868811/Future%20of%20Europe%20Conference%20Introducing%20platform.pdf.pdf  

[6] See EDPS-EDBP Joint Opinion on the proposal for an Artificial Intelligence Act: https://edps.europa.eu/node/7140_en

[7] See FRA´s 2020 report ‘Getting the Future Right: Artificial Intelligence and Fundamental Rights.’, available at: https://fra.europa.eu/sites/default/files/fra_uploads/fra-2020-artificial-intelligence_en.pdf

[8] https://fra.europa.eu/en/publications-and-resources/infographics/fundamental-rights-ai-what-consider

[9] https://www.ombudsman.europa.eu/en/event/en/1438

[10] During the 2022 ENO conference, the Danish Parliamentary Ombudsman suggested that ombudsmen could already try engaging with public authorities during the development phase of new digital systems using AI, to help designers take into account legal obligations and the principles of good administration.