Background to the complaint

1. On 22 September 2022, the European Personnel Selection Office (EPSO) launched an ‘open competition’ to recruit EU civil servants in a series of different fields (EPSO/AST/154/22).[1] The ‘Notice of Competition’ specified that the selection procedure would be organised in different phases, and that the ‘delivery mode’ of the tests - that is, whether the tests would be in-person or organised remotely - would be specified in the letters of invitation to the tests, which candidates would receive.

2. The invitation letters sent to candidates (on 4 November 2022) subsequently specified that the initial phase of ‘pre‑selection tests’ would take place remotely only. The letters included instructions concerning the format and operation of the tests, including requirements concerning technical equipment (for example, concerning devices, access right levels for computers and operating systems) and the nature of the physical environment (in particular, concerning the isolated space or room in which candidates would sit the test).

3. The tests took place between 22 November and 2 December 2022.

4. The Ombudsman received a considerable number of complaints concerning the selection procedure. The complaints concerned a range of issues, notably:

a) the practical implications of the technical and physical requirements, which made it difficult for some candidates to take the tests, and how EPSO dealt with complaints related to this;

b) technical issues experienced during the tests, and how EPSO dealt with complaints related to this;

c) how EPSO arranged for the rescheduling of tests; and

d) general issues related to EPSO’s communication around the selection procedure and how it dealt with complaints, including the requirements for submitting complaints.

5. The Ombudsman subsequently received similar complaints related to selection procedures organised by EPSO in which remote testing was used.

6. On 31 March 2023, after the Ombudsman had already opened this own-initiative inquiry, EPSO announced[2] that selection procedure EPSO/AST/154/22 was cancelled.

The inquiry

7. To address the various issues raised, the Ombudsman decided to open an own-initiative inquiry. The Ombudsman set out a series of questions to EPSO with a view to understand how it had prepared and determined the conditions of the remote tests, how it dealt with the various practical and technical issues experienced by candidates and about its decision to cancel EPSO/AST/154/22.[3]

8. In the course of the inquiry, the Ombudsman received EPSO’s replies[4]. The Ombudsman inquiry team also held meetings with EPSO and inspected a number of documents[5].

Tests taking place only remotely

Reply from EPSO

9. EPSO said that the decision to move towards online tests, rather than in-person, is a strategic choice endorsed by EPSO’s management board[6]. Remote testing is an efficient method of conducting selection procedures with large numbers of candidates, enabling candidates to participate equally regardless of geographic location, while improving flexibility and avoiding travel costs. As such, the move aims to increase the diversity of applicants in terms of background and geographic origin. It also aims to shorten the time that selection procedures take.

10. Before deciding to organise the tests remotely only for the competition in question, EPSO assessed operational risks, such as potential connectivity issues and technical problems. This assessment was based on previous experience with voluntary online testing, which EPSO had made available since 2021, and the experience with smaller scale recruitment ‘competitions’ in the second half of 2022, for which tests had been organised exclusively remotely.

11. Based on this assessment, EPSO concluded that, on balance, online testing would provide better access for many candidates and that it would not inconvenience more candidates than were inconvenienced by the requirement to travel to a test centre.

12. EPSO uses an external contractor for organising and proctoring tests in selection procedures. The current contractor is Prometric. According to the current contract, Prometric is not required to provide physical test centres. EPSO is currently preparing a new call for tenders, which will require the contractor to have test centres, so that it can have some capacity to organise physical tests in specific circumstances.

The Ombudsman's assessment

13. The Ombudsman understands EPSO’s objectives to shorten the time that selection procedures take and achieve greater diversity of candidates by making tests more accessible. However, in putting in place the remote testing model, EPSO does not appear to have sufficiently taken into account the potential negative implications of the technical and physical requirements for sitting the test (as set out below), in particular for candidates from less-well-off economic backgrounds.

14. In presenting the move to exclusively remote testing, EPSO essentially implied that there were no potential disadvantages. This approach was also evident in how it dealt with complaints. In terms of ensuring public acceptance of the move and managing expectations, it would have been better had EPSO provided a more comprehensive explanation of the rationale behind and potential implications of remote testing for candidates. This is something it should take into account in the future.

15. Before pushing ahead with the move to remote testing, EPSO should carry out a more thorough assessment. EPSO indicated that it is already considering this as part of the forthcoming call for tenders for the provision of test services, which it is in the process of preparing. The overarching goal should be to guarantee equality of opportunity, and any technical or physical requirements for remote tests should be assessed against that background. Where it is not possible to address the negative implications of such requirements on equality of opportunity, EPSO should ensure that selection procedures continue to provide candidates with the possibility of taking tests in physical test centres. EPSO should also monitor the impacts of changes to test formats on equality of opportunity. The Ombudsman will make a suggestion to this end below.

Requirements for candidates in remote testing

Reply from EPSO

16. EPSO’s instructions to candidates list a number of requirements, which candidates should meet in order to be able to sit remote tests.[7] These include technical requirements concerning computer devices (only laptops or desktop computers, and not tablets) and screen size, as well as operating systems and browsers supported, access rights level and internet connection.

17. For the test application provided by its contractor, only Windows and macOS above certain versions are guaranteed as supported; earlier versions or other systems are listed as not supported. Because of limited resources, EPSO is not able itself to systematically check which operating systems are compatible with the test application.

18. The instructions state that candidates need to have access rights to make software changes on the computers they use (known as ‘administrator rights’).

19. The test application has been designed for a single 19-inch or larger screen, with the graphics and text size reflecting this. This means that it is difficult to read all test documents on smaller screens, notably where candidates are using tools such as the notepad and calculator. EPSO is aware of the problem, which applies particularly to laptops, which often have smaller screens. However, there are no plans to optimise either the test application or the exam structure (text, documents and graphic sizes) for smaller screens in the near future.

20. The instructions also include requirements regarding the physical space in which candidates should take tests.

21. According to the instructions, candidates sit the test indoors in rooms with walls but not glass walls. EPSO stated that this requirement aims to prevent third parties (including other potential candidates) from seeing the exam content from outside the exam room.

22. According to the instructions, candidates should also cover the windows. EPSO stated that this requirement is intended to help with the quality of the lighting inside. Amongst other problems, varying lighting quality and direct sunlight glare can affect the quality of the video recordings of the exam.

23. EPSO confirmed that, where applicants raised concerns about their inability to comply with some of these requirements, it did not offer alternatives because this was not foreseen.

The Ombudsman's assessment

24. The Ombudsman understands that, in practice, it is not possible for candidates to use corporate computers (that is computers belonging to their employers) because candidates usually do not have ‘administrator rights’, and it is highly unlikely that any employer would waive any security settings for what is a private usage (sitting EPSO tests). The same is true for candidates that have computers with operating systems that are not supported.

25. The test application and the test content is designed for larger screens, but some candidates may not have access to such screens and have laptop screens only. As EPSO acknowledged, this means that candidates relying solely on smaller laptop screens would have trouble reading the test material and, essentially, be at a disadvantage in taking the test.

26. It is clearly problematic if some candidates are unable to properly use the test application due to the equipment to which they have access. As mentioned above, this undermines equality of opportunity and, in particular, could place people from a less-well-off economic background at a disadvantage. EPSO should address this issue, for example by requiring its contractor to optimise the test application or structure, so it is compatible with different screen sizes (including standard laptop screens), or taking other steps to ensure that candidates are not disadvantaged due to the equipment to which they have access. The Ombudsman will make a suggestion to this end below.

27. Where candidates raised with EPSO concerns about their inability to meet requirements, EPSO essentially dismissed these concerns and implied that it was the responsibility of candidates to overcome any barriers, for example by sourcing alternative computers. This was clearly not service-minded and regrettable.

28. It may be difficult for candidates to understand why some of the requirements are in place, notably for the requirements concerning the physical space. As such, the Ombudsman encourages EPSO to explain why certain requirements are necessary. Providing clear information to candidates can enhance public trust in and acceptance of requirements that may, at first glance, be difficult to understand. The Ombudsman will make a suggestion to this end below.

Information provided by EPSO about the selection procedure

Reply from EPSO

29. In addition to the Notice of Competition, EPSO also made detailed instructions available to candidates in their invitations. EPSO provided the Ombudsman with two sets of instructions for candidates on testing in the context of selection procedures involving remote testing, which were from November 2022 and September 2023.

30. EPSO said that it is revising the instructions to candidates that will be provided for future competitions with a view to making them shorter and more user friendly.[8]

31. The September 2023 instructions cover a wide range of topics, including: technical requirements; requirements for the physical area for sitting tests; how to book an appointment; concerning the prior ‘system check’ for the test application; the process of the exam; the rules governing remote proctoring; confidentiality and data protection rules, among other topics.

32. The instructions contain links to other websites (including a help website for technical support and the website of EPSO’s contractor) and contact details (such as for technical support prior to and during testing, and the EPSO online contact form).

33. During the test, candidates do not have access to the instructions. They are not allowed to have papers, so cannot print the instructions, and the instructions are also not available in the test application. EPSO stated that it would be difficult to ask its contractor to revise the test platform to incorporate the instructions because the application is also used by other clients of the contractor.

The Ombudsman's assessment

34. Given the new approach to testing, which includes requirements for candidates and new procedures, the revised instructions have to cover a wide range of issues. By EPSO’s own acknowledgment, some of the instructions are not entirely clear and there is a need to make them shorter and more user friendly.

35. Providing clear information and being as transparent as possible is a core requirement of public administration, even more so where public bodies introduce new procedures and rules, as EPSO has done. As mentioned above, the instructions should also provide clear explanations for technical or other requirements. Providing such transparency is crucial to ensuring public acceptance. As such, the Ombudsman welcomes EPSO’s commitment to revise the instructions to make them shorter and more user-friendly. She trusts that this will also address any inconsistent language such as identified in the context of this inquiry.

36. However, it is unreasonable to expect that candidates should memorise the instructions and be able to recall them in the inherently stressful situation of a test. To address this, EPSO should ensure that, in future, candidates have access to the instructions during the test, either by means of allowing them to bring a printed copy or by ensuring the instructions can be accessed through the test platform. This is particularly important when it comes to information on how to troubleshoot during an exam and how to make a complaint (see further below). The Ombudsman will make a suggestion to this end below.

Problems during the tests and complaint handling

Reply from EPSO

37. The instructions describe the procedures for addressing problems encountered during tests, in particular the requirement to first raise problems with the Prometric technical support, which provides ‘tickets’ to those reporting issues. The instructions and other referenced documents refer interchangeably to different terms: ‘incident ticket’, ‘case ticket’ and ‘Prometric ticket’.

38. The instructions also specify that, if candidates experience a technical problem when performing the ‘enhanced system check’ of their computer prior to the day of the test and if Prometric cannot find a timely solution, candidates should make arrangements to take the test from another computer. EPSO clarified that this is not a mandatory instruction and that it does not require candidates to have two computers at their disposal.

39. How EPSO follows up on complaints is set out in the instructions and the information on EPSO’s website. EPSO requires candidates to show evidence that they have tried to resolve their technical issues with Prometric’s technical support during the test, when the problem occurred, by submitting their ticket.

40. EPSO said that a complaint would be registered as such even if the complainant used the wrong form (the contact form instead of the complaint form): both forms are processed by EPSO’s Candidate Contact Service. That service determines whether messages qualify as complaints and if they require investigation. Candidates then receive a reply or their complaint is assigned to a different team within EPSO.

41. All complaints submitted through the online contact form are recorded in a database and receive a number. Where candidates send a complaint that is not in line with the instructions, EPSO sends a reply, reminding candidates of the rules.

42. The complaints of candidates who did not troubleshoot during the exam or who complained outside of the deadline were rejected. In cases where candidates failed to complete the test because they did not follow the instructions (and their exam was terminated by the proctors for this reason), their complaints were also rejected.

The Ombudsman's assessment

43. EPSO should provide clear and comprehensive information on how to make a complaint and how it deals with complaints. In particular, it should set out on its website basic information about the requirements for submitting complaints, how it deals with complaints and the timelines. As set out above, the requirements for submitting a complaint should also be available to candidates during the test. Among other things, it should ensure that the terminology it uses to describe different elements of the procedure is consistent throughout the different relevant documents. This is especially important where it introduces a new approach to testing, where there may be an increase in complaints related to technical issues. The Ombudsman will make a suggestion to this end below.

44. While EPSO described its practice and procedure for handling complaints, the Ombudsman’s inquiry and complaints submitted to the Ombudsman indicate that EPSO did not always deal with complaints consistently. This concerned both how it initially processed complaints, for example concerning admissibility and the requirement to have a ticket, as well as how it dealt with complaints concerning similar issues.

45. In the context of the Ombudsman’s inquiry, EPSO stated that it has in place guidelines concerning complaint handling. EPSO should review these guidelines, taking into account the issues identified in this inquiry, and ensure they are clear and comprehensive, and that its staff are sufficiently aware of the guidelines and have proper training to ensure they apply them consistently. The Ombudsman will make a suggestion to this end below.

46. If EPSO requires candidates to first have contacted its external contractor’s technical support before submitting a complaint about issues during a remote test, it should clearly set out this requirement in the notice of competition, rather than the instructions. However, it is important that EPSO demonstrate flexibility in investigating complaints, as there may be scenarios where it is not possible for candidates to contact the technical support and/or get tickets.

Rescheduling of tests

Reply from EPSO

47. With reference to selection procedure EPSO/AST/154/22, EPSO said that, in the context of its complaint handling, it offered the possibility to resit tests in the following situations:

• where technical issues were outside of candidates’ control and occurred despite their due diligence;
• where candidates had gone through the mandatory ‘enhanced system readiness check’ and mock exam, if the issue concerned the launching of the test; and/or
• where candidates tried to troubleshoot with Prometric during the exam itself and provided proof of these attempts (for example, a ticket number of their contact with the technical support).

48. EPSO said that, where the investigation into a complaint was not conclusive, the benefit of the doubt was always given to candidates, who were consistently allowed to resit the test. If candidates experienced technical issues concerning launching the test but had not undergone the mandatory checks beforehand, they were not offered the possibility of resitting the test.

49. According to the instructions, ”if, under exceptional circumstances, a test needs to be rescheduled or a candidate retested, the communication between EPSO and the candidate is handled by the test provider via email”. EPSO acknowledged that the wording of the instructions is not entirely clear. The test provider does not handle complaints on behalf of EPSO. EPSO itself replies to candidates.

50. Currently, EPSO does not provide candidates the possibility to choose the date and time when they could reschedule their tests, but provides candidates with a fixed date and time. EPSO is working with its contractor to revise the procedure for rescheduling tests, with a view to enabling candidates to book a date and time to resit tests.

51. EPSO said that rescheduling is allowed only once in cases of technical problems. The reason for this is that EPSO develops only two different tests (with different content) for each selection procedure: one for the tests as initially scheduled and one, in case of contingency, where tests need to be rescheduled. The development of test content takes months. The possibilities for rescheduling more than once are therefore limited and, typically, would be available only for candidates that have not seen the content of one of the tests. Under normal conditions, only a handful of complainants experience a technical issue in their rescheduled test. Where this occurs, candidates can submit a complaint, and EPSO's legal team or the selection board can decide that the candidate have another opportunity to take the test. This could take place either at EPSO’s premises or remotely.

The Ombudsman's assessment

52. The Ombudsman welcomes the fact that EPSO is looking into improving its policy concerning the rescheduling of tests, with a view to enabling candidates to choose a date and time that suits them as opposed to being assigned a date and time, which has been the practice to date.

53. The Ombudsman is of the view that, where candidates experience issues in rescheduled tests that are not their fault, EPSO should allow such candidates to reschedule again. The Ombudsman will make a suggestion to this end below.

54. Regarding other aspects of the information given to candidates, EPSO should ensure that the information concerning the rescheduling of tests is clear and accurate.

The decision to cancel selection procedure EPSO/AST/154/22

Reply from EPSO

55. EPSO said that the decision to cancel the competition was taken by EPSO at the request of its Management Board, after careful analysis taking into consideration the different problems that occurred, in particular as regards the implementation of remotely proctored tests, including technical problems and data protection concerns.

56. EPSO shared with the Ombudsman the documents outlining the operational and legal analysis underpinning the cancellation decision, including an assessment of other possible courses of action.[9]

The Ombudsman's assessment

57. Following the cancellation of the competition, the Ombudsman received a significant number of complaints on the matter. It is understandable that candidates who passed the tests are deeply disappointed with the cancellation, and would have preferred another solution to address the difficulties surrounding the selection procedure.

58. In the context of this inquiry, EPSO provided further details on the cancellation decision and the different scenarios that were considered. However, it is regrettable that EPSO did not do so proactively at the time of announcing the cancellation. Providing clear information to the public in a timely manner can enhance the legitimacy of and improve the acceptance of decisions by those negatively affected by them.

Conclusion

The Ombudsman closes the inquiry with the conclusion that:

No further inquiries are justified.

EPSO will be informed of this decision. The Ombudsman will also seek to bring her decision to the attention of the complainants who submitted complaints concerning EPSO’s use of remote testing.

Suggestions for improvement

1.  EPSO should make a comprehensive assessment of the potential impact on equality of opportunity of any requirements for remote tests. Where it is not possible to address the negative implications of such requirements on equality of opportunity, EPSO should ensure that selection procedures continue to provide candidates with the possibility of taking tests in physical test centres. EPSO should also monitor the impacts on equality of opportunity of changes to test formats.

2. EPSO should in particular ensure that its technical requirements do not prevent candidates from taking remote tests or place at a disadvantage candidates who do not have access to devices that meet these requirements. EPSO should work with its contractor with a view to developing the test application to address this. Where it is not possible to eliminate potentially exclusionary technical requirements, EPSO should explore potential alternative arrangements for candidates.

3. As the introduction of remote testing has led to considerable requirements concerning equipment and access to appropriate physical areas, the Ombudsman encourages EPSO to explain more clearly on its website why such requirements are necessary. Providing clear information to candidates can enhance public trust in and acceptance of requirements that may, at first glance, be difficult to understand, and which are likely to impact on the ability of some candidates to participate. Where applicants report legitimate concerns about the requirements, EPSO should deal with such concerns in a service-minded manner.

4. EPSO should ensure that, in future, candidates have access to the instructions during the test, in particular on how to troubleshoot during an exam and how to make a complaint. This can be ensured either by means of allowing candidates to bring a printed copy of the instructions or by ensuring the instructions can be accessed through the test platform.

5. EPSO should provide clear and comprehensive information on how to make a complaint and how it deals with complaints. In particular, it should set out on its website basic information about the requirements for submitting complaints, how it deals with complaints and the timelines. Among other things, it should ensure that the terminology it uses to describe different elements of the procedure is consistent throughout the different relevant documents.

6. EPSO should review its guidelines on complaint handling, taking into account the issues identified in this inquiry, and ensure they are clear and comprehensive, and that its staff are sufficiently aware of the guidelines and have proper training to ensure they apply them consistently. As noted, in some circumstances, for example with regard to the requirement to have first contacted its contractor’s technical support, EPSO should ensure a certain degree of flexibility in how it processes complaints.

7. Where candidates experience issues in rescheduled tests that are not their fault, EPSO should allow such candidates to reschedule again.

 

[1] https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:C2022/363A/01&from=EN

[2] EPSO did so by publishing an announcement on its website: https://eu-careers.europa.eu/en/cancellation-competition-epso-ast-154-22-assistants-ast3.  

[3] https://www.ombudsman.europa.eu/en/opening-summary/en/167288

https://www.ombudsman.europa.eu/en/doc/correspondence/en/168090

https://www.ombudsman.europa.eu/en/doc/correspondence/en/169795

[4] https://www.ombudsman.europa.eu/en/doc/correspondence/en/175767

https://www.ombudsman.europa.eu/en/doc/correspondence/en/175768

[5] https://www.ombudsman.europa.eu/en/doc/inspection-report/en/180852

[6] EPSO has a management board, which comprises one member per EU institution and three staff representatives with observer status appointed by the institutions’ staff committees. The management board’s main tasks include approving rules on how EPSO operates and is organised, including with regard to selection procedures.

[7] https://eu-careers.europa.eu/en/it-requirements-passing-epsos-remotely-proctored-tests-0#:~:text=You%20must%20use%20only%20a,mouse%20and%20keyboard%20are%20allowed.&text=The%20operating%20system%20must%20be,macOS%20Sonoma%20is%20not%20supported)

[8] A current online version of the instructions can be accessed here: https://eu-careers.europa.eu/en/remotely-proctored-testing

[9] https://www.ombudsman.europa.eu/en/doc/correspondence/en/175768