First meeting

Date: Thursday, 28 September 2023

Remote meeting via Webex

Present

European Personnel Selection Office:

· Head of Sector, Legal Affairs;

· Head of Unit for Testing Operations;

· Head of Unit for Outreach and External Relations

European Ombudsman:

· Tina Nilsson, Head of the Case-handling Unit;

· Valentina Stoeva, Inquiries Officer;

· Christophe Lesauvage, Legal expert;

· Zoé Lejeune, Inquiries Trainee

Purpose of the meeting

The purpose of the meeting was for the Ombudsman inquiry team to understand how the European Personnel Selection Office (EPSO) prepared and determined the conditions of remote tests, as well as how it dealt with the various practical and technical issues experienced by candidates. The meeting will be followed by a second meeting at EPSO’s premises, consisting of a video demonstration of remote testing and a continuation of the discussion on the above issues.

Introduction and procedural information

The Ombudsman inquiry team introduced themselves, thanked EPSO’s representatives for meeting with them and set out the purpose of the meeting. They outlined the legal framework that applies to meetings held by the Ombudsman, in particular, that the Ombudsman would not disclose any information identified by EPSO as confidential without EPSO’s prior consent.^[1]

The inquiry team explained that they would draw up a draft report on the meeting to be sent to EPSO to ensure that the contents were factually accurate and complete. No confidential information would be included in the report or otherwise provided to any third party.

The inquiry team clarified that the meeting concerns a European Ombudsman own-initiative inquiry. The Ombudsman received a considerable number of complaints concerning the use of remote testing only for the 'pre-selection tests' in a selection procedure for recruiting EU civil servants (Assistants AST3 - EPSO/AST/154/22), organised by EPSO. The complaints concerned the practical implications of the decision that the tests should take place only remotely, as well as how EPSO dealt with practical and technical issues that arose. While EPSO subsequently cancelled the selection procedure, the Ombudsman decided to continue her inquiry, also with a view to there being a considerable number of complaints related to subsequent selection procedures that also had remote only testing. As such, the inquiry is not related to one selection procedure only but aims to verify the issues involved, to help EPSO to define good administrative practices in its testing activities, in its communication with candidates and in how it handles complaints.

Information exchanged

1) The diversity of candidates

EPSO’s representatives explained that online testing enables EPSO to reach a larger number of candidates. This is because, in the past, many candidates faced financial or practical barriers to taking tests at a physical test centre. Candidates had to travel to the cities where test centres were located (for most Member States, in the capital city only) or even, at a later stage of the selection procedure, to Brussels. Remote testing eliminates travel and accommodation costs, for which candidates were only partially reimbursed. There is not enough ‘history’ to conclude that, statistically, the new model led to more diversity among candidates. Remote testing was only introduced in 2022. Nevertheless, EPSO’s representatives believe that remote testing creates more equality of opportunity and facilitates the participation of more people.

EPSO does not require candidates to provide information on, and hence does not monitor, whether there is diversity based on the backgrounds of candidates, for example related to gender, ethnicity, sexual orientation, and socio-economic status. EPSO adopted its Action Plan on Equality and Diversity in May 2022, which sets out objectives for promoting diversity. However, other priorities than gathering information on equality and diversity were addressed first in the development of the new model, in particular operational aspects. Furthermore, EPSO depends on the Directorate-General for Information Technologies (DIGIT) to monitor diversity aspects. EPSO’s representatives also mentioned that EPSO launched a diversity survey in 2021 based on voluntary participation and published the result on its website in 2022.

EPSO collects data on the geographical balance of candidates because candidates have to indicate their nationality as part of the application process. EPSO provides this information to Member States but not publicly.

EPSO’s representatives added that, as part of the application process, candidates may now choose to indicate their gender, through the following categories: female, male, non-binary, other or prefer not to say.

2) How EPSO deals with complaints

EPSO’s representatives explained that EPSO receives several types of complaints: complaints of a legal nature, complaints of a technical nature and emails from candidates merely expressing that they do not like the new model. The latter are not considered to be administrative complaints. EPSO does not track the number of complaints in the larger sense.

There is no tracking system and no statistics on the percentage of complaints concerning remote testing. More generally, there is no specific system for tracking complaints by topic. Complaints are very diverse and it is not operationally feasible to track them by topic.

EPSO’s representatives explained that EPSO’s website distinguishes between questions and complaints, and provides two distinct forms accordingly. Complaints and queries first pass through a front desk service, in charge of the “triage” of complaints. The front desk deals with certain complaints immediately when an investigation is not needed. If an investigation is needed, complaints are sent to the operational unit or the legal unit. In this sense, EPSO dispatches complaints by topics. EPSO can also track complaints and queries as they are registered with tickets. EPSO’s representatives noted that the same person can send different complaints or the same complaint several times. The number of complaints is therefore higher than the number of complainants.

3) EPSO’s use of tickets, contact and complaint forms

EPSO’s representatives explained that, when a candidate sends a query or a complaint, they receive a ticket number. Candidates may then turn to EPSO again, adding new elements. EPSO will register the new elements with a new ticket number. For instance, a person could have difficulties to connect and complain about that issue. The same person could add other elements requiring another investigation. From an operational point of view, the second complaint could be assigned to a different EPSO staff member. EPSO’s representatives further noted that EPSO receives more or less 100 complaints or queries per day.

EPSO’s representatives clarified that there is an internal tracking system related to complaints. If the same candidate complains several times, colleagues will see all complaints.

EPSO’s representatives further noted that candidates can reply by email to EPSO's emails, but their replies will be registered with a new ticket. EPSO needs to attribute tickets for operational purposes. Without a ticket number, complaints and queries would get lost. Complainants can also have an email conversation with EPSO without having to use the online complaint form. For instance, if EPSO asked a candidate for evidence by email, the candidate can provide the evidence simply by replying to the email. The issue of the ticket number is only relevant for internal purposes (the ‘back office’).

Finally, EPSO clarified that a complaint would be registered as such even if the complainant used the wrong form (the contact form instead of the complaint form). All forms go through EPSO’s Candidate Contact Service irrespective of the form chosen. That Service determines whether messages qualify as complaints.

4) The differences between the initial and updated instructions given to candidates by EPSO (technical instructions versions of November 2022 and of September 2023)

EPSO’s representatives noted that the main difference between the two sets of instructions is the ‘delivery mode’ of the exam, i.e. how the content of the exam is presented to candidates and how candidates are invigilated. Initially, in late November-early December last year, candidates received instructions related to the live proctored ‘delivery mode’. Candidates were proctored by staff members of EPSO’s contractor during the exam. The proctors’ role was to guide candidates through pre-examination checks and to invigilate them during the exam.

The updated instructions describe another ‘delivery mode’: automated proctored testing. EPSO started using a new model in September 2023: automated proctored testing. The first round of automated proctored exams has recently been completed. In this mode, candidates are guided through the checks by an automated tool, with a reduced number of checks (basically only identity checks). Candidates are recorded during the exam. Recordings are stored on a review portal to which EPSO has access. After the exam, EPSO will watch the recordings and may decide not to process a candidate’s exam if it notices problematic behaviour. For example, EPSO would exclude a candidate from the procedure if the recoding showed that the candidate worked with another person during the exam.

5) The possibility for candidates to take a test in a space that is not private and to use a corporate computer

EPSO’s representatives clarified that candidates are not required to sit the tests in a private residence. Candidates must find a space suitable for testing. EPSO does not check whether the space chosen is private or public. However, EPSO requires some privacy: there should be no outside noises and other people in the room. Privacy should not be confused with private space in the sense of a space located in a private residence. Candidates can sit the test in a private space located in a corporate building, for example. As long as the space is private, EPSO will not check whether or not the space is located in a private residence.

EPSO’s representatives also explained that it is possible to use a corporate computer to take the tests. Indeed, EPSO does not require candidates to own the computer on which they test. Nevertheless, EPSO informs candidates that certain technical requirements limit the use of corporate laptops. To launch the exam, candidates have to download the ProProctor application. During the exam, this application locks the candidates’ computers to prevent them from accessing windows other than the exam window. Most of the time, corporate computers have a firewall policy that conflicts with the function in the application for locking the computer. This would only become apparent once the application goes into that mode. Candidates can disable the firewall only if they have administrator rights on the computer. It is also possible for the owner of the computer (for example, the IT service of their employer) to authorise the ProProctor application. It is not EPSO’s role to contact the IT services of each employer to make these arrangements. Information concerning the use of corporate laptops is available on EPSO’s website since February 2023.

EPSO’s representatives added that most complaints on the use of corporate laptops come from EU staff with temporary contracts. They have difficulties understanding that they cannot take the tests on the computer that they use on a daily basis for work. Nevertheless, this practice is common: employers own corporate laptops and have the prerogative to decide what can be done with them.

6) The requirement to provide evidence of troubleshooting

EPSO’s representatives noted that the instructions provided to candidates describe the requirements concerning ‘incident tickets’. EPSO requires that candidates show evidence that they have tried to resolve their technical issue on the spot. When a technical issue appears, candidates must first go to Prometric and use the technical helpline, or the chat box. EPSO wants candidates to go to Prometric first for efficiency reasons. In this way, when EPSO assesses complaints, it does not start from scratch and already has information from the contractor. EPSO’s representatives clarified that it also accepts other types of evidence but that providing an incident ticket is the best evidence. 

Upon the inquiry team’s request, EPSO’s representatives confirmed that different formulations used, such as incident tickets, case tickets and Prometric tickets, all refer to the same thing but are not to be confused with EPSO's own tickets. It is always Prometric that provides the incident ticket number to the candidate. It is not possible for candidates themselves to generate a ticket number.

EPSO accepts other evidence than incident ticket numbers because it might happen that a candidate does not have a ticket number or was unable to use the chat box or the technical helpline. Candidates can make a screenshot of the chat box or use their phone to take a picture of the chat box before the start of the exam. EPSO will accept these pictures as evidence of troubleshooting. The bulk of technical issues happens before the exam is started and concerns the impossibility to launch the exam. At this stage, the application will not yet have locked the computer and prevent access to other windows, and it is possible to take a screenshot of the chat box. In sum, EPSO will examine any type of evidence and give candidates the benefit of the doubt.

EPSO’s representatives then clarified that there is no link to the chat box during the exam. If candidates experience technical issues during the exam, they have to exit the exam before they can ask Prometric to help. After exchanging with a chat bot, they can reach a human helpdesk agent. Candidates might experience two types of issues during the exam. First, their connection might temporarily drop. Second, candidates could experience slow performance, e.g. going from one question to the other is very slow. In this case, they cannot turn to Prometric on the spot but they have to submit their exam and later complain to EPSO that the slow performance impacted their results. EPSO has access to technical logs and can see, for instance, whether the exam lasted longer. There are also remedial measures when the server is slow. When the connection freezes for five minutes or more, the time of the exam also freezes.

Finally, EPSO’s representatives concluded that EPSO is aware that a candidate may not have an incident ticket number and that this is not necessarily the candidate’s fault.  Sometimes, Prometric is unable to provide a ticket number. Complainants are not excluded by default when they do not have a ticket number. In addition, if EPSO is aware of a more generic technical issue, it will issue a public message and give explanations about retesting plans.

7) EPSO’s access to ‘data forensics’

EPSO’s representatives clarified that for each “test event”, Prometric provides an initial technical report. The report contains only general information, including the number and identity of candidates who did not show up to the test or cancelled their test while taking it (when people cancel their test during the examination, there is a second check where they must confirm that they want to cancel their test). The report does not mention all technical problems that individual candidates may have encountered, such as slow performance or problems to launch the exam. However, on the basis of individual complaints, EPSO will ask Prometric to give access to information about the complainant’s test. For instance, Prometric could provide the chat logs of the candidate.

EPSO’s representatives clarified that Prometric provides a ‘data forensics’ service but only on demand. Prometric does not automatically give EPSO extensive information for each individual candidate’s problems during a test. When EPSO deals with a complaint, it first decides internally whether the complaint looks legitimate. If so, EPSO will launch an investigation and ask Prometric for specific information about the complainant’s test.

EPSO’s representatives also explained that EPSO deals differently with major technical issues. For example, an issue would be major when more than 5% of the tickets received by the operational team concern the same technical issue. In this case, EPSO will proactively contact Prometric because the problem might be on their side. Prometric could also inform EPSO if it notices a problem. For instance, in a selection procedure, Prometric encountered issues with its booking system. Many candidates were unable to book a test and complained to EPSO. This was a major issue.

8) The decision that an individual complaint deserves further investigation

EPSO considers complaints legitimate when they require an investigation. The Candidate Contact Service examines each ticket and sees if it requires an investigation. Typically, complaints merely expressing candidates’ dissatisfaction with remote testing will not require an investigation. In this case, the Candidate Contact Service will inform complainants that it cannot deal with their complaint. When needed, the Candidate Contact Service can contact candidates to request more information and later communicate the complaint to the operational team. More information on a candidate’s test can also be requested from Prometric. If the investigation reveals that the candidate is not providing accurate information, EPSO will inform them that it cannot deal with the complaint. For instance, EPSO received complaints in which candidates claimed that their connection dropped during the exam. After reviewing the technical logs, EPSO discovered that the candidates voluntarily terminated their exams. EPSO also received complaints where it is clear that the candidate failed to follow basic instructions, such as downloading the ProProctor application. In this case, the Candidate Contact Service will also reply to the complainant and not send the complaint to the Operations unit.

Conclusion of the first meeting

The inquiry team thanked EPSO’s representatives for their time and for the explanations provided. The inquiry team added that the second meeting to continue the discussion will soon be scheduled. The meeting ended.

 

Second meeting

Date: Thursday, 26 October 2023

Physical location at EPSO’s premises

Present

European Personnel Selection Office:

· Head of Sector, Legal Affairs;

· Head of Unit for Testing Operations

European Ombudsman:

· Tina Nilsson, Head of the Case-handling Unit;

· Valentina Stoeva, Inquiries Officer;

· Christophe Lesauvage, Legal expert;

· Zoé Lejeune, Inquiries Trainee

· Kevin Crespo, IT support

Purpose of the meeting

On 26 October 2023, the meeting continued. The purpose of the meeting was for the Ombudsman inquiry team to view a video demonstration and receive further information concerning the conditions of remote tests and how EPSO deals with technical issues experienced by candidates.

Introduction and procedural information

The Ombudsman inquiry team thanked EPSO’s representatives for the meeting.

The inquiry team explained that they would draw up a draft report on both meetings to be sent to EPSO to ensure that the contents were factually accurate and complete. No confidential information would be included in the report or otherwise provided to the complainant or any third party.

Information exchanged

EPSO’s representatives presented a video demonstration of remote testing in live proctored delivery mode. The demonstration was made in collaboration with their contractor, Prometric. It was originally prepared for a meeting with the European Data Protection Supervisor (EDPS).

EPSO’s representatives clarified that the delivery mode of the tests has changed since the demonstration was made. Nevertheless, EPSO’s representatives believe that the presentation remains useful for the Ombudsman, given that received complaints relate to the former delivery mode. EPSO’s representatives recalled that the main difference between the live proctored delivery mode and the automated proctored delivery mode is the absence of live proctors invigilating the exams.

Before launching the demonstration, EPSO’s representatives briefly explained the steps preceding the exam. Candidates receive the invitation to the exam on their EPSO account, which is the official communication channel between EPSO and candidates. Candidates can set up notifications from their EPSO account to their own mailbox. The invitation already contains information concerning technical requirements. Before sending the invitations, EPSO also uploads the test content on Prometric’s platform, as well as the list of candidates. This is to ensure that, when a candidate books a test via Prometric’s booking system, Prometric has access to the candidates who should be tested. Prometric then provides candidates with a confirmation number once they have booked a test. On the day of the test, candidates must use this confirmation number to enter the exam.

EPSO’s representatives further explained that, before launching the exam, candidates must pass a system check. It is possible to do the system check before the day of the exam. In the previous delivery mode, candidates had to do a more thorough Enhanced Readiness Check and a mock exam. If they did not take the mock exam, EPSO would not deal with their technical complaints. The goal was for candidates to anticipate technical problems before the exam and, if necessary, to seek the help of a specialist or find another computer. Prometric removed the Enhanced Readiness Check and the mock exam from the automated proctored delivery mode. EPSO’s representatives clarified that these two steps are important because they allow candidates to test in almost real conditions before the exam. EPSO has now asked Prometric to make significant improvements to the system, including reinstating the Enhanced Readiness Check and the mock exam.

1) The video demonstration

EPSO’s representatives showed the demonstration and provided a number of clarifications.

A. The launch of the exam

EPSO’s representatives explained that candidates are required to download the ProProctor application. Candidates must book and launch exams using the application. The application can be downloaded by anyone and appears as an icon on a computer’s desktop.

EPSO’s representatives noted that the ProProctor application goes into lockdown mode when candidates launch their exam. From that stage, candidates can no longer open other applications nor other windows and the application monitors the internet connection and activities on the computer. Prometric collects this data and EPSO can access them in case of a complaint. For example, the application will collect data about the computer used by the candidate. In the event of a complaint, EPSO will check whether the computer used met the technical requirements.

EPSO’s representatives clarified that they can also access logs showing which candidates took the system check and the mock exam. EPSO’s representatives mentioned a complaint in which the candidate had encountered technical problems during the exam. By checking the logs, EPSO saw that the complainant had carried out only part of the Enhanced Readiness Check before the test. The problem he encountered during the test was related to the part of the system check that he did not undertake.

B. The identification checks and the invigilation of the exam

EPSO’s representatives noted that, in live proctored delivery mode, proctors were performing the identity checks. In automated proctored delivery mode, identity checks are carried out by AI comparing a picture of the candidate to a picture of the candidate’s ID card. If the candidate fails the AI identity check, the candidate will still proceed with the test. However, the system will flag the problem and allow an EPSO staff member to check, at a later stage, the correspondence between both pictures. 

EPSO’s representatives added that AI in automated proctored ‘delivery mode’ can also flag issues during the exam. For example, the system detects voices, multiple faces, or candidates looking away. Recordings of the exams are placed on a review portal to which EPSO has access. EPSO is responsible for verifying the recordings and checking the flags. EPSO’s representatives also stated that, whenever a new flag is raised, a message temporarily appears on the candidate’s screen. There is also a count of the number of flags on the corner of the candidate’s screen.

C. The retention of records and data protection

EPSO’s representatives explained that the recordings are kept for 30 days. EPSO intends to extend the duration once operations resume. There are different retention periods. In the case of an administrative complaint (Article 90(2) of the EU Staff Regulations), EPSO keeps recordings longer. The administrative complaint could lead to judicial proceedings, which justifies the longer retention period. EPSO’s representatives clarified that EPSO only keeps recordings longer in the event that the candidate has lodged a complaint. In terms of the size of the data, it would not be possible to keep all recordings for more than 30 days. EPSO’s representatives explained that, in the live proctored delivery mode, candidates were also recorded and recordings were also kept 30 days. Exams were nevertheless invigilated by live proctors.

Concerning data protection, EPSO’s representatives stated that the EDPS carried out an audit of EPSO’s remote testing and is currently preparing its report.

D. The environmental checks

EPSO’s representatives explained that, in live proctored delivery mode, ‘readiness agents’ asked candidates to move their camera around and show their testing environment. These checks are called “environmental checks”. The readiness agents also asked candidates to show certain parts of their body, for example to roll up their sleeves. These are called “personal checks”. This aspect generated many complaints and candidates were very unhappy with the readiness agents. In addition, EPSO’s Management Board considered the personal checks too invasive. EPSO has stopped all checks – both personal and environmental - for the current tests in automated proctoring. EPSO intend to re-integrate basic environmental checks in the future, in particular as regards the room and the desk.

E. The duration of EPSO’s investigations

EPSO’s representatives explained that EPSO deals with technical complaints as fast as possible. Testing events always comprise two testing windows. EPSO aims to deal with all technical complaints before the start of the second testing window (the ‘contingency window’) to be able to schedule retests in the second testing window. EPSO tries to book all candidates who have encountered valid technical problems during the initial testing window in the second testing window or, when possible, at the end of the initial testing window. The normal duration of an investigation is a matter of days. At the end of a testing window, Prometric prepares an initial report with generic information. However, EPSO does not launch investigations based on the report but only based on complaints.  

D. Retesting possibilities

EPSO’s representatives noted that EPSO requires technical complaints to be filed within three days of the exam because this allows EPSO to retest candidates in the same testing window or in the planned contingency testing window. For the three selection procedures organised under the new testing model, EPSO has automatically booked a retest for all candidates who made a technical complaint. This is due to the high number of complaints. Approximately 36 % of the candidates who booked a test in the first selection procedure made a complaint and approximately 20-25 % in the second and third selection procedure. The normal rate of complaints should be around 7 to 10%. Out of those complaints, EPSO would reject a number of unfounded complaints. However, because the number of complaints was so high, EPSO booked a retest for all complainants. EPSO even moved the contingency window to later dates in certain selection procedures to allow EPSO to deal with all technical complaints before the start of the contingency window.

EPSO’s representatives further clarified their policy on retesting. Candidates who retest cannot keep their initial results. However, candidates can decide not to retest and keep their initial results. EPSO’s representatives also noted that some candidates are dissatisfied because they did not complain within the three-day deadline and subsequently learned that other candidates complained and were booked for a new test.

E. Security agents and proctors

EPSO’s representatives explained that in live proctored delivery mode, proctors invigilating exams could “escalate” an issue to a security agent. Only security agents could decide to end an exam. The security agents did not monitor candidates actively but were called by proctors in case of suspicion. Proctors had up to eight candidates to invigilate. EPSO’s representatives noted that the proctors worked for Prometric’s contractors based in Colombia and Morocco. This played a role in EPSO’s decision to move to automated proctored delivery mode because proctors based outside Europe had access to personal data from candidates.

EPSO’s representatives clarified that Prometric is an American company but that EPSO works with its Irish branch. Before the summer of 2023, there were data transfers to American servers, in addition to data transfers to proctors based in Colombia and Morocco. Since the new contract signed in December 2022, Prometric has worked to put in place a new “data loop” in the EU with all servers located in Ireland. This loop was activated in September 2023. EPSO’s representatives added that, from the beginning, the contract stated that Prometric had to comply with EU Regulations and use EU based servers only. When EPSO found out that Prometric did not fully comply with this, it asked for improvements which have now been implemented.

2) Questions

The video demonstration ended. The inquiry team asked a series of questions based on the previous meeting and recent complaints received by the Ombudsman.

A. The Operational systems and browsers

EPSO’s representatives clarified why the operational system MacOS Ventura 13.0 to 13.2.1 is not supported. Systems are continuously being updated. The ProProctor application is not always up to date with the latest updates. EPSO’s representatives acknowledged that the application is usually behind in relation to the latest MacOs updates. EPSO has asked Prometric to update its system for MacOS more regularly. EPSO’s representatives also clarified that candidates receive from Prometric the latest information concerning the operation systems supported when they book their test. Because of its limited resources, EPSO is not able to systematically check itself which systems are compatible with the application.

Upon the inquiry team’s remark, EPSO’s representatives noted that there might be some discrepancies between the instructions and EPSO’s website concerning the latest systems supported. It is likely due to an internal communication problem. Previously, only one team was responsible for communicating with the candidates and all communications with the candidates were made through the invitation letter and the link to instructions. Recently, EPSO opened another communication channel with the candidates by publishing information on its website to give future candidates the possibility to become more familiar with testing procedures, even before they apply to a competition. The discrepancies are likely due to a problem of coordination between the two teams now in charge of providing information to the candidates.

EPSO’s representatives also acknowledged that the instructions mention Google Chrome as the only supported browser. EPSO’s representatives explained that the tests do not take place on Google Chrome but on the application. They could not provide a definite answer to why using Google Chrome is part of the technical requirements but clarified that Google Chrome is the web browser recommended by Prometric for the best performance possible of the ProProctor application, which has to be connected via the Internet

B. The screen requirements

The instructions mention that graphics and text are designed for a single 19-inch or larger monitor. EPSO’s representatives explained that problems only arise once candidates enter the exam. In particular, some essay questions require candidates to open several documents. Because of the exam’s structure, it is difficult to read all documents on smaller screens. The scratchpad and the calculator may cause other difficulties in reading documents on smaller screens. EPSO’s representatives concluded that EPSO is aware of the problem, especially because laptops often have smaller screens. However, EPSO’s representatives noted that there are no plans to readjust the exam structure for smaller screens in the near future.

C. The possibility to reschedule a test

EPSO’s representatives clarified that, in the current situation, candidates are not provided with a choice of dates on their rescheduling time-slot, following technical issues. EPSO had so many candidates to reschedule that it provided them with a fixed date and time. EPSO is working with its contractor to revise the rescheduling procedure. At the moment, Prometric needs to create a new ‘eligibility number’ manually for candidates that are rescheduled. EPSO would like Prometric to use the same eligibility number as for the first test and to extend its validity to the contingency testing window. This solution would allow EPSO to quickly inform Prometric which candidates are going to retest. Candidates will then be able to book their own date and time in the contingency window. As in the initial testing window, candidates will be able to modify or cancel their booking up to 24 hours before their scheduled exam. EPSO is hoping to have the system in place for the contingency windows taking place in November[2]. However, EPSO cannot guarantee that the system will be ready.

EPSO’s representatives confirmed that rescheduling is allowed only once in cases of technical problems. The reason is that only two test contents are developed for each selection procedure: one for the initial window and one for the contingency window. The development of test contents takes months. The possibilities for rescheduling are therefore limited. If EPSO allowed rescheduling more than once, it would only be able to offer this to  candidates who were not exposed to the content of the test. EPSO’s representatives further explained that under normal conditions, only a handful of complainants experienced a technical issue in their rescheduled test. In this situation, candidates can complain to EPSO and EPSO's legal team or Selection Board can decide that the candidate needs to be retested a third time. The candidate will then be retested in an ad hoc manner, either at EPSO or remotely. The candidate will progress in a completely separate track because the selection procedure will not be halted. EPSO’s representatives noted that, should the problems concern a significant number of complainants, the selection procedure will be suspended. In particular, selection procedures steps involving ranking of candidates will have to be suspended.

Upon the inquiry team’s request, EPSO’s representatives clarified the differences between retesting a candidate and rescheduling a candidate. A rescheduling is organised for candidates having experienced technical issues before launching the exam. These candidates have not yet taken the test. A retest will be organized for candidates having experienced issues in the middle of the test. These candidates have been exposed to the exam. Nevertheless, the practical follow-up - rescheduling and a retest - is the same. EPSO’s representatives also acknowledged that EPSO staff members tend to use these terms interchangeably. 

D. The instruction to use another computer

The message that EPSO sends to candidates about their retest advises them to have at their disposal another computer. EPSO’s representatives clarified that, depending on the technical issue, it is possible that candidates taking the test for the second time experience the same technical issue, caused by their computer. EPSO thus advises them to use a different computer to increase their chances not to experience the same issue. EPSO’s representatives explained that one solution could be to allow candidates who encountered technical problems during their first test to be rescheduled to a test centre. This solution is not expected in the near future, as EPSO’s contract with Prometric does not require Prometric to provide test centres. EPSO is currently working on the preparation of a new call for tenders which will require the contractor to have a few test centres. The idea is that the contractor should have some capacity to organise physical tests in case of special needs.

The instructions also specify that, if candidates experience a technical problem when performing the enhanced system check prior to the day of the test and if Prometric cannot find a timely solution, candidates should make arrangements to take the test from another computer. EPSO’s representatives clarified that it is not a mandatory instruction. EPSO cannot force candidates to have two computers at their disposal. The goal is to inform candidates that, if they encounter a technical problem that cannot be solved with a particular computer, they will likely encounter the same problem on the day of the exam.

EPSO’s representatives recalled that the Enhanced Readiness Check and the mock exam are not part of the automated proctored delivery mode but that it has asked Prometric to make changes to include them in the new version. In the future, EPSO will recommend that candidates perform the Enhanced Readiness Check and the mock exam at least one week before the exam. If candidates experience technical issues related to their computer, they will be advised to try a different computer. EPSO is aware of the problems with the formulation of the instructions given to candidates. EPSO has already tried to make the instructions clearer, but acknowledged that there is still room for improvement.

E. The tolerance period for launching the exam.

The instructions mention in bold “do not launch your exam earlier than 30 minutes before the beginning of your exam”. Previous instructions specified that candidates who had launched their exam more than 30 minutes before the scheduled time would not be able to retest. EPSO’s representatives clarified that the objective of the booking system is to distribute candidates evenly during the day in order to avoid overloading the system. EPSO’s representatives clarified that they are not sure why the previous instructions mentioned that EPSO would not organise a retest for candidates who launched the exam more than 30 minutes before their scheduled time.

F. The testing environment.

The instructions requires that the testing location to be indoors 'walled – not glass walls'. EPSO’s representatives explained that this is part of the standard security features for remote testing. The rationale behind the prohibition of glass walls is that people could look into the exam content from outside the room. The instruction to cover the windows was intended to help with the quality of the lighting inside the room. If candidates face sunlight, the quality of the recordings will not be sufficient. EPSO intends to maintain these requirements because EPSO needs to be able to check the recordings and decide to exclude a candidate from the procedure if the recordings reveal issues. If some recordings are not sufficiently visible, equal treatment of candidates cannot be ensured. EPSO’s representatives further explained that EPSO uses standards that are common in the testing industry. EPSO already made some adjustments to the standards that it considered too strict. For example, EPSO stopped requiring candidates to empty their pockets or to show their ears.

EPSO’s representatives further explained that candidates can only have tissues and a transparent container for water on their working desk. This part of the instructions dates back from the live proctored delivery mode. Proctors were checking that candidates had only these items on their desk (except in case of special needs). The checks will soon be automated. Candidates will be required to show their desk and it will be recorded. EPSO staff could review the recordings and take a decision after the exam. These automated environmental checks are not in place yet but are foreseen in the improvements of the system. 

Candidates cannot have access to EPSO’s instructions on their desk in case of a technical problem. EPSO’s representatives explained that candidates can only access the instructions on their computer, and only if the computer is not in lockdown mode.

The inquiry team noted that it would be useful for candidates to have access to the instructions during the exam itself. The situation can be stressful for candidates and they might forget that they need to write in the chat box and later possibly reach the helpdesk. EPSO’s representatives noted the concerns but explained that Prometric’s application is also used by other companies. The room for EPSO to require a specific version of the application is therefore limited.

EPSO’s representative then detailed the requirements concerning the physical environment, that is, the room and furniture. Candidates cannot have cameras, other computers or smartphones that could record the screen. Items attached to the walls are not prohibited but EPSO advises to cover them. EPSO’s representatives acknowledged that the proctors tended to interpret the instructions too strictly. Candidates whose exams were terminated on the basis of a strict interpretation of the environmental requirements were retested. EPSO’s representatives further explained that EPSO intends to be reasonable when checking the flags of the tests under automated proctoring. Normally, candidates will be excluded from the selection procedure if a third party enters the exam room. EPSO’s representatives noted that EPSO intends to be reasonable and will not cancel an exam because a cat, toddler, or even an adult entered the exam room if the recording shows that it was an honest mistake.

G. Providing evidence of troubleshooting and obtaining ticket numbers

EPSO’s representatives confirmed that it would not be possible for the candidate to take a screenshot once the lockdown mode is activated. Nevertheless, the logs will show that the candidate wrote in the chat box and/or were in contact with a helpdesk agent. If candidates experience a problem before the lockdown mode is activated, they are able to take screenshots. Based on the inquiry team’s remark, EPSO’s representatives agreed that the instructions require better formulation on this point.

EPSO’s representatives also recalled that, in case there is no evidence of troubleshooting, EPSO will give the candidate the benefit of the doubt. If a candidate makes a generic complaint, EPSO will not look into it. However, if the complaint is sufficiently specified and the candidate explains the steps taken during the exam, EPSO will investigate. EPSO’s Candidate Contact Service receives all complaints. If a complaint mentions a description of a technical issue without any information, the Candidate Contact Service will ask for more information from the candidate. Once the Candidate Contact Service receives concrete information, it will issue a ticket to the technical team.

EPSO’s representatives also confirmed that the Prometric’s technical support automatically and proactively provide candidates with a ticket number when they report an issue. It is part of the protocol for the technical support agents. The inquiry team asked EPSO to provide the document describing this protocol. EPSO’s representatives agreed to provide it after the meeting.

H. The communication between EPSO and complainants

According to the instructions, ‘If, under exceptional circumstances, a test needs to be rescheduled or a candidate retested, the communication between EPSO and the candidate is handled directly by the test provider via email'. EPSO’s representatives confirmed that the test provider does not handle complaints on EPSO’s behalf. EPSO replies itself to complainants. Once the Candidate Contact Service has issued a ticket number, it will dispatch the tickets to the relevant teams within EPSO. If necessary, EPSO will contact Prometric to obtain more information. Based on the inquiry team’s remark, EPSO’s representatives acknowledged that the wording of the instructions is not entirely clear.

I. The duration of technical investigations  

EPSO’s representatives confirmed that EPSO handles all technical complaints before the second testing window starts. Complaints first arrive in the  Candidate Contact Service. Because most are of a technical nature, they are dispatched to the technical team. Technical complaints are dealt with as soon as possible and in all cases before the start of the testing contingency window. EPSO’s representatives added that two categories of complaints need to be submitted within three days: the technical complaints and the complaints concerning the content of the exam, for instance concerning mistranslation of an exam question. These complaints will also be dealt with as soon as possible and complainants will have the possibility to be reintegrated into the selection procedure. However, general complaints merely expressing dissatisfaction with remote testing will not be considered technical complaints to be dealt with before the second testing window. 

In relation to recent complaints received by the Ombudsman, EPSO’s representatives explained that they are aware that some candidates consider that it is unfair that other candidates had the possibility to retest while they do not have this possibility because they did not complain within three days. The retesting decision is issued after the three-days timeframe to submit a technical complaint. Late complainants will not be retested and will be informed that they failed to complain within the deadline. However, a candidate might launch an administrative complaint (under Article 90(2) of the EU Staff Regulations) against the decision putting an end to their participation in the competition. In that case, EPSO can still look at the logs to see if the person experienced technical issues and used the chat box and/or were in contact with the helpdesk. If the candidate’s complaint is substantiated, the candidate could be reintegrated in the selection procedure.

EPSO’s representatives also explained that the technical team has sent retesting decisions to candidates in batches. It is possible that candidates complained to the Ombudsman because they heard that others were being retested. By now, these candidates should all have received a decision on their complaints.

J. The Candidate Contact Service procedures

The inquiry team noted that EPSO sent to the Ombudsman a document named ‘Candidate Contact Service procedures’. The inquiry team understands that the document has been produced for the purpose of the inquiry and asked whether EPSO has an internal procedure or a protocol addressed to the Candidate Contact Service team members.

EPSO’s representatives replied that the Candidate Contact Service team leader produced the document for the purpose of the inquiry. However, they subsequently clarified that the Candidate Contact Service team has a well-documented SharePoint Online webpage, with guidance, templates, and instructions for the team to follow, as needed.

Conclusion of the meeting

The inquiry team thanked EPSO’s representatives for their time and for the explanations provided. The meeting ended.

Brussels, 26 October 2023

 

Tina Nilsson                                                                                                                 Valentina Stoeva

Head of the Case-handling Unit                                                                           Inquiries Officer

 

[1] Article 4.8 of the European Ombudsman’s Implementing Provisions.

[2] These tests have now been postponed to an unspecified point in time in the future.