Background to the complaint

1. The complainant participated in an EU staff selection procedure organised by the European Personnel Selection Office (EPSO) for the recruitment of administrators in the field of Information, communication and technology (EPSO/AD/398/22[1]). As part of the selection procedure, the complainant had to take remotely-proctored ‘reasoning tests’[2] monitored by an external contractor of EPSO.

2. On 10 May 2023, the complainant sat the tests.

3. Following the tests, after having discussed with other candidates, the complainant realised that the functionality of the calculator in the test application was limited during the tests she had taken. In particular, she could not use the keyboard to complete equations on the calculator. As she had used a computer with a different operating system to those candidates, she concluded that this technical problem was linked to the operating system of her computer.

4. On 13 May 2023, the complainant contacted EPSO’s contractor to raise the matter and received a ‘case number’.

5. The same day, the complainant submitted a complaint to EPSO. In her complaint, she claimed that the limited functionality of the calculator meant she had to use her mouse to use the calculator, which took a considerable amount of time, whereas other candidates had been able to use their keyboards. She contended this put her at a disadvantage compared to other candidates.

6. EPSO dismissed the complaint as it claimed that the complainant had not submitted her complaint in line with the instructions for remote testing, notably that she had not raised the issue with the contractor at the time of the test.

7. The complainant argued that she could not have known at the time of the test that the functionality of the calculator was limited compared with other candidates.

8. Dissatisfied with EPSO’s replies, the complainant turned to the Ombudsman.

The inquiry

9. The Ombudsman opened an inquiry into what proof EPSO required in terms of troubleshooting to process the complaint.

10. In the context of the inquiry, the Ombudsman also asked whether EPSO would consider reviewing the technical log of the complainant’s test.

11. In this context, the Ombudsman received EPSO’s reply on the complaint and, subsequently, the comments of the complainant in response.

12. In parallel, the Ombudsman conducted an own-initiative inquiry concerning EPSO’s remote testing[3]. The conclusions made in this context are relevant for this inquiry.

Arguments presented to the Ombudsman

13. The complainant argued that she had raised the issue with both EPSO’s contractor and EPSO itself as soon as she became aware of the issue. She received a case number from the contractor, which she provided to EPSO.

14. She could not have raised the issue with the technical support during the tests, as she became aware of the limited functionality of her calculator only after the test. Prior to the tests, candidates were not provided with instructions on how the calculator was expected to function.

15. In its reply to the Ombudsman, EPSO stated that it had dismissed the complaint as the complainant had not sought to raise the matter with the technical supporting during the exam, as indicated by the instructions.

16. While EPSO verified the technical log of the complainant’s tests in the context of the Ombudsman’s inquiry, it said that there were no technical means to verify afterwards that some features of the tools were indeed not working.

The Ombudsman's assessment

17. In response to the complainant’s concerns about the lack of certain functionalities, EPSO referred to the fact that the complainant had not raised the issue with the contractor’s technical support at the time of the test. It did not address the complainant’s argument that she could not have raised the matter with the technical support, since she became aware of the problem only after the test. However, this argument appears reasonable and, at the very least, EPSO should have replied to it.

18. In the context of another Ombudsman inquiry[4], EPSO stated that, when candidates experience an issue but were able to finish the test(s), it would consider it reasonable that they did not contact the technical support during the test. In such a case, EPSO stated that it would further investigate the technical logs of a case. This logic clearly appears equally valid in this case, where the complainant could not contact the technical support during the test, since she was not aware of the issue until subsequently. The Ombudsman considers that EPSO was wrong to reject the complaint merely because the complainant did not raise the issue with the technical support during the tests, and that how it handled the complaint amounted to maladministration.

19. However, in the context of the Ombudsman’s inquiry, EPSO reviewed the technical log of the complainant’s test.

20. EPSO stated that there are no technical means to verify afterwards whether certain functions of the calculator were not available.

21. While EPSO did not provide comprehensive arguments as to why it could not verify that some features of the tools were indeed not working, the Ombudsman notes that the complainant completed the tests. In the absence of definitive verification on the functionality of the calculator available to the complainant and other candidates, it is not possible to draw any conclusions about whether and to what degree the complainant was disadvantaged compared to other candidates, and the Ombudsman has therefore decided to close the case.

22. Regarding how EPSO handled the complaint, as EPSO consulted the technical logs in the context of the inquiry, the Ombudsman considers that it is not necessary to make a recommendation concerning this matter.

23. However, for future selection procedures involving remote testing, the Ombudsman considers that it is important that EPSO can verify, by consulting technical logs, whether there were problems with the functionality of tools within the test application. Only by doing so can it properly investigate complaints from candidates that experience problems during tests, and potentially provide them with redress. The Ombudsman is aware that EPSO is currently carrying out a procedure to award a contract for conducting its remote testing. In this context, the Ombudsman will make a suggestion for improvement aimed at addressing the above issue.

Conclusion

Based on the inquiry, the Ombudsman closes this case with the following conclusion:

There was maladministration in how EPSO dealt with the complaint about technical issues experienced by a candidate in a remote test.

The complainant and EPSO will be informed of this decision.

Suggestion for improvement

EPSO should explore whether it is possible to retroactively review all technical aspects of a candidate’s experience in a test application, including the functionality of tools, by consulting technical logs. If this is technically possible, EPSO should insist that this possibility is included in the future contract for the provision of remote testing services, which it is currently preparing.

 

Emily O'Reilly
European Ombudsman

Strasbourg, 08/07/2024

 

[1] https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv%3AOJ.CA.2022.076.01.0001.01.ENG&toc=OJ%3AC%3A2022%3A076A%3ATOC

[2] Tests are taken by candidates remotely, through a dedicated application .The reasoning tests consist in sets of questions in the field of verbal, numerical and abstract reasoning.

[3] OI/1/2023/VS  https://www.ombudsman.europa.eu/en/case/en/63317

[4] Case 1866/2023/MAG https://www.ombudsman.europa.eu/en/case/en/64925