The background to the complaint

1. This complaint concerns the alleged unequal treatment of candidates by the European Personnel Selection Office (EPSO) in staff selection procedures.

2. The complainant took part in Open Competition EPSO/AD/207/11, which aimed to select administrators in the field of European Public Administration.

3. In July 2011, EPSO informed the complainant that he had not been admitted to the next phase of the competition (the Assessment Centre phase) because he had not obtained the passmark of 70.447 in the admission tests (he had obtained 70 points).

4. The complainant then asked EPSO how it was possible that the passmark had been set at 70.447. In this respect, he noted that one correct answer was worth 1 point in the verbal, numerical and abstract reasoning tests and that the five aspects tested in the situational judgement test[1] were awarded points divisible by 0.125. As a result, he argued, the final mark of any candidate could only be a whole number or a number ending with the following decimals: .125; .250; .375; .5; .625; .750; .875.

5. In reply, EPSO clarified that it was possible to set the passmark at 70.447, given that certain questions in the admission tests had been "neutralised". It explained that "neutralisation" of questions works as follows. If a set of 10 questions is awarded 10 points (1 point for each correct answer), and one of the questions has to be neutralised (because, for example, of an error in the formulation of the question), the available 10 points are divided equally amongst the remaining 9 questions. This means that 1.111 points will be awarded for each correct answer and that a candidate who answers all 9 questions correctly will thus obtain 9.999 points for those questions.

6. The complainant replied by pointing out that the above method placed him at a disadvantage compared to candidates who were asked questions which were later neutralised. He stated that there were 20 questions in the verbal reasoning test. One point was awarded for each correct answer. If a candidate answered 18 questions correctly, he/she received 18 points, provided no questions were neutralised. If another candidate who took the same test was asked an erroneous question which was subsequently neutralised, each correct answer of that candidate was worth 1.052 points (20 divided by 19). This meant that even though that candidate answered 18 questions correctly, he/she received 18.95 points (18 x 1.052). In sum, while both candidates answered 18 questions correctly, they were awarded different points. The complainant also pointed out that calculations concerning a neutralised question in test D (situational judgement) were more complex due to the different values of the correct answers (each answer could be awarded 0.5, 1, 1.5 or 2 points). However, the same disadvantage would arise.

7. In the absence of any further reply from EPSO, the complainant turned to the European Ombudsman with the present complaint.

The subject matter of the inquiry

8. The Ombudsman opened an inquiry into the following allegation and claim:

Allegation:

The method used by EPSO to neutralise erroneous questions in Open Competitions leads to unequal treatment of candidates.

Claim:

EPSO should re-examine the complainant's result with a view to eliminating the unequal treatment of candidates.

The inquiry

9. On 31 January 2012, the Ombudsman asked EPSO to provide an opinion on the complaint by 30 April 2012. After having requested an extension of that deadline, EPSO sent its opinion on 24 May 2012. The opinion was forwarded to the complainant with an invitation to submit observations. The complainant did not submit observations.

The Ombudsman's analysis and conclusions

A. Allegation that the method used by EPSO to neutralise erroneous questions in Open Competitions leads to unequal treatment of candidates

Arguments presented to the Ombudsman

10. The complainant alleged that the method that EPSO uses to neutralise erroneous questions in Open Competition leads to unequal treatment between the candidates who were asked an erroneous question and those who were not.

11. In its opinion, EPSO first pointed out that, in line with established case-law[2], a selection board enjoys wide discretion when confronted with irregularities or errors which have occurred in the course of an open competition involving a large number of candidates, and which cannot, under the principles of proportionality and sound administration, be rectified by repeating the tests. Therefore, EPSO has recourse to a practice referred to as neutralisation, whereby it cancels an erroneous question and apportions the point(s) allocated to that question amongst the remaining questions.

12. When a question is neutralised, the selection board has to make sure that candidates are not thereby affected negatively. In other words, if a candidate obtains a total mark that is high enough to allow him/her to proceed to the next phase of the competition before neutralisation, but the total mark is insufficient in this regard after neutralisation, that candidate cannot be affected by neutralisation so he/she maintains his/her place amongst the successful candidates. This, however, did not occur in the complainant's case: his total mark was below the passmark both before and after neutralisation. EPSO added that none of the questions the complainant was asked had to be neutralised.

13. According to EPSO, the neutralisation of certain questions was irrelevant to the complainant because his total mark was insufficient for him to be admitted to the Assessment Centre phase both before and after neutralisation. The minimum total mark before neutralisation which a candidate had to obtain in order to be admitted to the Assessment Centre phase was 70.5. The complainant obtained a total mark of 70. After neutralisation, the lowest total mark that had to be obtained for candidates to be admitted to the Assessment Centre phase was that of 70.447. As mentioned above, the candidates who obtained this lowest mark could not be negatively affected by the neutralisation and therefore maintained their positions. Given that none of the complainant's questions were neutralised, and that he therefore maintained his initial total mark (70), he was not admitted to the Assessment Centre phase.

14. EPSO pointed out that, within the framework of an open competition, the principle of equal treatment is only breached when two categories of persons whose factual and legal situations are the same are treated differently or where different situations are treated in an identical manner.

15. EPSO also pointed out that the Notice of Competition stated that the number of candidates to be admitted to the next phase of the competition (the Assessment Centre phase) would be approximately three times the number of successful candidates indicated in the notice. It also stated that if a number of candidates were to tie for the last available place, they would all be invited to the assessment centre.

16. EPSO concluded by stating that the complainant was neither positively nor negatively affected by the method it used to neutralise erroneous questions in the competition in question. It therefore considered the complainant's allegation unfounded and that all candidates were treated fairly.

The Ombudsman's assessment

17. The Ombudsman notes that it is indeed correct to state that a candidate who has had a question which he/she answered neutralised may benefit from neutralisation. If candidate A in a competition answers 8 questions out of 10 correctly, and one of the questions that he answered incorrectly is neutralised (because of a defect in the question posed), candidate A's total mark for the eight correct questions will be 8 x 1.111 (8.888). If candidate B also answers eight questions correctly in the same tests, but none of his incorrect answers is neutralised, candidate B's total mark for the eight correct questions will be 8 x 1 (8). This constitutes a difference in treatment between candidates of equal merit. This difference in treatment is amplified by the fact that the passmark in the tests will be reduced to ensure that no candidate will be excluded from the competition as a result of neutralisation. Using the above example, this means that if the passmark had been set at 9, it would be reduced to 8.888, thus allowing candidate A, but not the equally competent candidate B, to be admitted to the next phase.

18. It is thus clear that having a question neutralised could be determinative as regards whether a candidate will be admitted to the next phase of a competition.

19. While it is likely that some candidates will benefit from neutralisation (namely, those candidates whose question answered incorrectly is neutralised), there is, of course, no means of knowing how any given candidate would have answered a neutralised question had that question been properly formulated in the first place. It is possible that the above-mentioned candidate A would have answered the defective question correctly had it been properly formulated in the first place, and would thus have obtained 9 points.

20. The problem outlined above would, of course, not arise if no defective questions were posed. The Ombudsman underlines, however, that while every effort should be made to ensure that no defective question is asked in staff selection procedures, it is difficult to guarantee that all questions posed to all candidates are error-free, particularly in a situation where questions are chosen from a database containing many tens of thousands of questions.

21. A selection board is always entitled to require candidates to take a new test when the first one is affected by a serious irregularity. As regards whether it would be appropriate for a selection board, using its broad margin of discretion, to require candidates who were asked defective questions to take a new computer-based test, the Ombudsman notes that prior to the introduction of computer-based testing, all candidates in open competitions were asked the same multiple choice questions. If defective questions were asked in those competitions, those questions were annulled by the selection boards in respect of all candidates[3]. Selection boards then proceeded to reallocate points for all valid questions answered (in sum, they 'neutralised' the defective questions). Some of the candidates who took part in this type of competition sought the annulment of the decisions excluding them from such competitions by arguing that neutralisation breached the principle of equal treatment[4]. The General Court stated, however, that a selection board has a large margin of discretion when faced with irregularities or errors in an open competition. Moreover, the Court held that, in light of the principles of proportionality and good administration, irregularities or errors in an open competition with mass participation cannot be corrected by repeating the tests[5]. The Court also noted, in this respect, that the neutralisation of questions has as its aim the objective assessment of the individual merits of (all) candidates in the competition, by excluding any unequal treatment due to an assessment of these candidates based on non-identical or incomparable criteria.

22. The Ombudsman understands the remark about the principle of proportionality, in the context of the ruling of the General Court in question, as referring only to the proportionate use of resources in an open competition with mass participation.

23. The position taken by the Court was entirely logical. If the selection board had opted that the tests be taken afresh, it would have had to invite all candidates to sit the tests anew. Inviting many thousands of candidates to take such tests again would not have amounted to a proportionate use of resources, or been in compliance with principles of good administration, especially since the alternative option, neutralisation of defective questions, was applied to all candidates equally (every candidate had the same question(s) neutralised).

24. Open competitions involving computer-based testing do not, however, pose the same logistical challenges in terms of tests taken afresh. While it is still the case that a large number of candidates take part in such open competitions, the candidates are asked different questions in the computer-based tests (the questions are chosen, at random, from a database containing tens of thousands of questions). A selection board that would opt to hold fresh tests with a view to ensuring that all candidates received valid questions would only need to invite those candidates who were asked defective questions. The new tests could moreover be limited to those candidates who, had they not been asked defective questions, would have had a mathematical possibility of reaching the passmark. In sum, there would be no useful purpose, and therefore no good reason, to invite to a fresh test (i) the candidates who would have reached the passmark even if the defective question had not been taken into account and (ii) the candidates who would not have reached the passmark even if a full point was awarded for the defective question. The number of candidates required to take such tests afresh should, assuming that the number of defective questions is very limited, be no more than a handful.

25. The Ombudsman is of the view that the logistics and costs of inviting such a very limited number of candidates to take the computer-based tests afresh would not necessarily always be disproportionate.

26. The Ombudsman has carefully examined the ruling of the Civil Service Tribunal in Case F-2/07 Matos Martins v Commission, which EPSO referred to in its opinion to the Ombudsman. The ruling relates to the neutralisation of questions in computer-based testing. It is only available in French and the relevant part reads as follows:

"191 À cet égard, il convient de rappeler qu’un jury de concours dispose d’un large pouvoir d’appréciation lorsqu’il est confronté à des irrégularités ou à des erreurs intervenues lors du déroulement d’un concours général ou d’une procédure de sélection à participation nombreuse qui ne peuvent, en vertu des principes de proportionnalité et de bonne administration, pas être réparées par une répétition des épreuves du concours (arrêt Giuletti e.a./Comission, précité, point 58).

192 En l’espèce, le requérant n’a pas établi que l’octroi d’un point pour chaque question neutralisée n’a pas été de nature à concilier le souci du jury d’assurer une égalité absolue de traitement entre les candidats et les exigences d’une bonne administration (voir, en ce sens, arrêt du Tribunal de première instance du 7 septembre 2005, Heinen/Comission, T-181/04, RecFP p. I-A-221 et II-1013, point 41)" (emphasis added by the Ombudsman).

27. The wording of paragraph 192 of the Matos Martins v Commission judgment implies that the Tribunal appears to recognise that neutralisation does not ensure absolute equality of candidates. However, the Tribunal takes the view that the need for absolute equality must be reconciled with the requirements of good administration.

28. The Ombudsman recognises this ruling of the Civil Service Tribunal, which implies that a selection board is entitled to choose to neutralise questions, rather than to request the very limited number of affected candidates to take the defective tests afresh.

29. In light of the above, and in particular in light of the ruling of the Civil Service Tribunal, the Ombudsman finds that no further inquiries are justified as regards the complainant's allegation.

30. As regards the complainant's claim that EPSO should re-examine the complainant's result with a view to eliminating the unequal treatment of candidates, the Ombudsman notes that, whatever the merits of the complainant's allegation, the complainant was not admitted to the next phase of the competition because he did not obtain the passmark of 70.447 in the admission tests. The Ombudsman notes that while certain candidates may benefit from having their scores increased as a result of neutralisation (see paragraphs 17-20 above), and that neutralisations may result in the previously established passmark being slightly lowered (in the present case, it was lowered to 70.447 from 70.5), neutralisation never results in the passmark increasing. Consequently, the Ombudsman notes, the complainant would have failed to reach the passmark even if neutralisation had not occurred. In light of these observations, the Ombudsman finds that no further inquiries are justified as regards the complainant's claim.

31. However, the Ombudsman insists, while the neutralisation of defective questions in computer-based tests is, in light of the ruling of the Civil Service Tribunal, one option which is always open to selection boards, EPSO should never assume that neutralisation is the only option open to resolve the problem of defective questions. In this context, the Ombudsman will make a further remark with a view to encouraging EPSO to examine the various options open to it in future competitions.

B. Conclusions

On the basis of the inquiry into this complaint, the Ombudsman closes it with the following conclusion:

The Ombudsman finds that no further inquiries into the allegation and claim are justified.

The complainant and EPSO will be informed of this decision.

Further remarks

EPSO should reflect on alternative ways of dealing with the problems of defective questions in computer-based tests. EPSO could consider:

1) the option of requesting the very limited number of candidates who were asked defective questions and who were thereby adversely affected to take the computer-based tests afresh. When organising competitions, EPSO should make sure that sufficient time and resources are allocated for such new tests, thereby ensuring that it can carry out the instructions of a selection board that requests such new tests.

2) the option of providing for replacement questions, at the beginning of computer-based tests, the answers to which would only be taken into account if the tests of individual candidates are found to contain defective questions.

 

Emily O'Reilly

Done in Strasbourg on 22 May 2014

---

[1] These were: resilience, prioritising, analysis and problem solving, working with others, and delivering quality and results. The awardable points were 9.375, 8.75, 9.375, 8.75 and 8.75 points respectively.

[2] EPSO referred to Case F-2/07 Matos Martins v Commission, judgment of 15 April 2010, not yet published in the ECR, paragraph 191.

[3] Errors often occurred only in certain language versions of the tests. However, if an error occurred in a question in one language version, the same question was annulled in all language versions (see joined cases T-167/99 and T-174/99, Carla Giulietti and others [2001] ECR SC-I-A-00093 and SC-II-00441, paragraph 44).

[4] See joined cases T-167/99 and T-174/99, Carla Giulietti and others, cited in footnote 3 above.

[5] See joined cases T-167/99 and T-174/99, Carla Giulietti and others, cited in footnote 3 above, paragraph 58.