The background to the complaint

1. The complainant, a Spanish national, participated in three EPSO open competitions (EPSO/AD/177/10, EPSO/AD/185/10 and EPSO/AD/206-207/11). He sat the computer-based tests (CBT) but did not obtain the mark required to be admitted to the next stage.

2. On 25 March 2011, the complainant contested EPSO's decisions not to admit him to the next stage of the competitions mentioned above. He also expressed concerns as regards (i) the margin of error of the CBT database in comparison with the number of candidates pre-selected for the next stage of the competitions, and (ii) the frequency of the neutralised questions.

3. Given that EPSO's reply of 4 April 2011 only partially addressed the complainant's concerns, he decided to turn to the Ombudsman.

The subject matter of the inquiry

4. In his complaint of to the Ombudsman, the complainant submitted the following allegation and claim:

Allegation:

EPSO failed properly to address the complainant's grievances set out in his letter dated 25 March 2011.

Claim:

EPSO should answer properly to the complainant's above correspondence.

The inquiry

5. With the aim of rapidly finding a satisfactory outcome to the present complaint, on 27 April 2011, the Ombudsman used the simplified procedure and asked EPSO to address the complainant's grievances. On 28 April 2011, EPSO replied, by forwarding to the Ombudsman's services a copy of its 7 April 2011 reply to the complainant which appeared not to have been delivered to him. Subsequently, the Ombudsman invited the complainant to submit observations on that reply. The complainant did so on 2 and 13 May 2011.

6. After a careful analysis of EPSO's reply and the complainant's observations, the Ombudsman considered that EPSO had still not taken a position on all of the specific points raised by the complainant. Consequently, the Ombudsman asked EPSO to provide an opinion on the complainant's allegation and related claim by 30 September 2011.

7. EPSO sent its opinion on 3 October 2011. The Ombudsman forwarded it to the complainant with an invitation to submit observations. The complainant did so on 30 November 2011.

8. On 1 and 18 December 2011, and on 8 and 13 February 2012, the complainant sent further correspondence relating to his complaint.

9. On 2 March 2012, the Ombudsman asked EPSO for further information regarding certain aspects of the case.

10. EPSO replied to the Ombudsman's further inquiries on 30 April 2012. This reply was forwarded to the complainant, who, on 27 June 2012, submitted his observations on it. On 9 and 30 July 2012, the complainant sent additional information.

The Ombudsman's analysis and conclusions

A. Alleged failure properly to address the complainant's grievances

Arguments presented to the Ombudsman

12. In his complaint to the Ombudsman, the complainant argued that EPSO's pre-selection process, consisting of multiple choice computer-based tests (CBT), is unfair and unfit for selecting good candidates for recruitment by the EU institutions. In his view,  "the inherent margin of error of questions in EPSO's CBT database is [...] 15-20%, which is much higher than the 'top' 1-2% which gets preselected through them".

13. The complainant stated that EPSO failed to acknowledge and to correct the errors contained in its CBT database. The CBT compulsory self-assessment test, introduced by EPSO in the registration process for 2011 Open Competition EPSO/AD/206/11, contained four erroneous questions out of a total of 20 questions contained in the part dealing with verbal numerical and abstract reasoning.

EPSO's reply in the context of the simplified procedure

14. In its reply to the complainant, provided in the context of the Ombudsman's simplified procedure, EPSO first outlined that the selection policy it implemented, including all the recent changes under its Development Programme, was defined and approved by all the EU institutions.

15. EPSO further pointed out that, since 2009, it has been monitoring the performance of all items used in CBT by means of appropriate psychometric analysis methods. An extremely limited number of items have been neutralised by selection boards.

16. As regards the compulsory self-assessment test, introduced in the registration process for 2011 Open Competition EPSO/AD/206/11, EPSO noted that the general aim is to familiarise potential applicants with the tests they will be expected to take in the first stage of the assessment (namely, verbal, numerical and abstract reasoning tests, as well as a situational judgement test). By offering a compulsory range of test questions, EPSO endeavours to help candidates to assess realistically the possibility of their being successful in an open competition. Thus, the test questions in the self-assessment test need to have the appropriate level of difficulty.

17. The items used in the self-assessment test were designed exclusively for this purpose and are not extracted from the databank. EPSO acknowledged that, as the complainant pointed out in his correspondence, one of the items used in the self-assessment test for the competition mentioned above displayed a wrong answer. However, this resulted from a data input error when the tests were uploaded on EPSO's website. EPSO further explained that this error was corrected within 24 hours after the self-assessment test questions were put on-line. EPSO expressed its regrets that the above situation might have confused some potential candidates. Nevertheless, it argued, this incident could not lead to the general conclusion that the whole database of questions was inappropriate.

The complainant's observations

18. In his observations on the above reply, the complainant argued that EPSO's failure to disclose the number of neutralised questions constituted evidence of noncompliance with the principle of transparency. EPSO acknowledged that one CBT question which had been uploaded on its website as part of the compulsory self-assessment test was wrong. The complainant pointed out that one wrong question out of 20 implies "5% of the questions being wrong". As regards the other three "clearly incorrect questions", the complainant noted that EPSO contended that they should be considered "difficult" questions. As a matter of courtesy, EPSO should prove that these three questions are correct. Moreover, given that the sample test was not extracted from the CBT database, "potential candidates have been confronted with a different CBT facility, [...] which did not necessarily correspond to the standards of the real CBT" (mainly, because four out of 20 questions were wrong and the test was unsupervised).

19. Finally, the complainant stated that EPSO failed to address his grievances relating to (i) the actual frequency of neutralised questions in competitions; and (ii) the inherent margin of error in the CBT database, which is not taken into account when selecting candidates.

EPSO's opinion

20. In its opinion on the complaint, EPSO submitted that, during the computer-based tests, each candidate receives a number of questions drawn randomly from a database. All questions have a similar level of difficulty, although the tests of each candidate do not necessarily contain the same questions[1].

21. EPSO stated that, in cases where candidates have had a question cancelled or neutralised, their marks are distributed on a proportional basis among the remaining questions, so that the score indicated in the notice of competition is not changed. In order to guarantee equal treatment of candidates and to avoid adversely affecting candidates, when questions are cancelled, EPSO makes a comparison between the situation 'before' and 'after' the cancellation. If there are any candidates who would have reached the threshold before the cancellation of questions but do not reach it because of the cancellation, those candidates will, in any event, be invited by EPSO to the next stage. In the complainant's case, EPSO did not cancel any questions in any of the three competitions in which he took part.

22. As regards the frequency of neutralised questions in competitions, EPSO outlined that, so far, the average percentage of questions cancelled compared with the total number of questions in the database tests used in various competitions accounts for only 0.22%. In particular, as regards the three competitions referred to by the complainant, the percentages of cancellation were as follows: EPSO/AD/177/10 - 0.24 %; EPSO/AD/185/10 - 0.23%; and EPSO/AD/206-207/11 - 0.33%. EPSO argued that the complainant's estimated 20% margin of error in the CBT database was based on a statistical extrapolation which is unfounded.

The complainant's observations on EPSO's opinion

23. In his observations on EPSO's opinion, the complainant noted that the use of "average percentage" of questions cancelled is misleading, given that the CBT database is the same for all recent competitions. Therefore, "the incidence of neutralisations is cumulative over each competition in which the questions have been previously used". For instance, as regards the three competitions in which he took part - the CBT process for the first competition (EPSO/AD/177/10) is influenced by the sum of neutralised questions for the competition itself (0.24%) plus those neutralised subsequently in competitions EPSO/AD/185/10 and EPSO/AD/206-207/11 (that is to say, 0.23% + 0.33%). That means that the margin of error was 0.80% for competition EPSO/AD/177/10.

24. He further noted that EPSO failed to state when and how many questions were neutralised. This information could allow a precise calculation of the incidence of neutralised questions in each competition to be made. The percentages given by EPSO allow, at least, for an estimate of the incidence of neutralised questions to be made. In this respect, the average cancellation percentage provided by EPSO (0.22%) should be multiplied by the number of competitions, at least 11, between March 2010 and October 2011. It follows that the share of neutralised questions is above 2.4% of the database.

EPSO's reply to the Ombudsman's further inquiries

25. In reply to the Ombudsman's request to explain what happens with the questions which are neutralised[2], EPSO stated that, when a selection board takes a decision to neutralise a question, this question is immediately blocked in the CBT database. The question is not removed from the database, but it cannot be used in any tests during the time it is blocked. Subsequently, the question is analysed and, if possible, it is corrected and fed back into the database. Questions that cannot be corrected are permanently deactivated in the database.

26. EPSO further pointed out that the complainant's arguments regarding the three contested questions in the self-assessment test were incorrect and inconclusive. Answering the Ombudsman's question concerning those three questions, EPSO reiterated that those questions were adequate.

27. In reply to the Ombudsman's request to provide statistics relating to the number of erroneous questions used in competitions and subsequently cancelled, EPSO indicated that, between 2006 and 2007, only 53 questions out of the several thousand available in the database used for admission tests were neutralised (only 0.65% of the questions in the database, which at the time contained over 12,000 items, in three languages). It added that the number of neutralised questions has considerably decreased since 2006. In 2007, only 14 questions were neutralised, while in 2008, only nine were neutralised. EPSO also provided detailed statistics from 2010 to the time of its reply regarding the number of neutralised questions per competition[3].

28. With regard to the Ombudsman's request for further information concerning the quality control procedures put in place to ensure the reliability of the CBT database, EPSO stated that the quality checks, including compulsory pre-testing on a control group, are carried out by contractors. They develop the questions prior to their delivery to EPSO. If required, a translation is made by DGT, and the translated version of the questions undergoes DGT's linguistic quality control process, before being delivered to EPSO. Upon receipt, EPSO staff carries out an internal quality check of the substance of each question. In addition, EPSO may conduct additional pre-testing of new test items, in order to improve their quality[4]. Finally, ex-post quality controls of test questions are carried out on a systematic basis.

29. EPSO pointed out that, following the rulings of the Civil Service Tribunal in Case F-35/08 Pachtitis v Commission and of the General Court in Appeal Case T-361/10 P Pachtitis v Commission, computer-based testing, including the control of test contents, has been placed under the authority of the selection boards. The selection boards carry out systematic in-depth ex-ante control of the questions in any given competition. In addition, systematic ex-post control is also exercised by the selection boards, namely, through examining candidates' complaints concerning specific test items, and through the neutralisation of items which, based on these complaints, are deemed to contain an error.

30. In reply to the Ombudsman's request for information concerning the scientific data supporting the validity and reliability of the CBT database and statistical analyses concerning the results obtained thus far, EPSO stated that, since 2009, all questions delivered have been subjected to an ex-post psychometric item performance analysis, which is carried out internally by qualified EPSO staff. Consequently, EPSO possesses, for each question in the CBT database and for all its existing language versions, a detailed item performance profile based on a specific scientific method (the Rasch model). This ex-post analysis is routinely used to prepare the pool of questions for upcoming competitions. If, on the basis of such analysis, it appears that one or more of the performance elements do not meet the expected standards, the question is removed from the database.

The complainant's observations on EPSO's reply

31. In his observations on EPSO's reply, the complainant contested the data provided by EPSO. He stated that the reference to the period 2006-2007 is irrelevant for the current CBT, because, at that time, all the candidates sat the test at the same time, had exactly the same questions, and could review them afterwards. Thus, erroneous questions had the same incidence on all candidates.

32. In his view, EPSO's calculations of erroneous questions are incorrect for the period 2006-2008[5]. Moreover, the data for the period after 2010 are incomplete, inaccurate and distorted because: (i) EPSO takes account of half of the competition cycles to date; (ii) it does not consider the cumulative damage of items which were only neutralised after being used in previous competitions; and (iii) it does not distinguish items which can be challenged such as ‘verbal’, ‘numerical’ and ‘abstract reasoning’,  from those which cannot, due to their subjective nature, such as ‘situational judgement’ (for AD competitions) and ‘professional skills’ (for AST competitions). According to the complainant's calculation, on the basis of the above incomplete, inaccurate and distorted data, the cumulative number of questions neutralised for the general competitions of 2010 was at least 6 to 43[6] out of 2540 items (which represents at least 0.24% to 1.69% of questions).

33. As regards the quality control procedures, the crucial issue regarding the neutralisation of questions is "the number of questions exposed to any form of independent verification". In his view, review of questions by candidates is the only legitimate means of quality control. Furthermore, EPSO cannot rely on contractors in terms of compulsory pre-testing. He also contested the ability of DGT to translate CBT questions because of their specific nature. The complainant argued that, in order to maintain the equivalence of difficulty standards, translating CBT questions from English into 22 other languages would require tremendous and comprehensive psychometric evaluation across these languages.

34. The complainant asserted that ex-ante verification does not serve its purpose because of the large number of CBT questions. In addition, in his view, a selection board cannot conduct psychometric standardisation pre-testing for lack of technical qualifications, means and time.

35. Finally, the complainant referred to the Rasch model. In his opinion, this model does not guarantee the objectivity of questions and it cannot replace ex-ante psychometric standardisation. He pointed out that the Rasch model only checks the input and, like any closed circuit of measurements, it only assesses in accordance to itself. Hence, the use of the Rasch model could bring fictive results.

The Ombudsman's assessment

36. The complainant was not satisfied with the quality of EPSO's replies of 4 and 7 April 2011 to his arguments concerning: (i) the alleged high frequency of neutralised questions in competitions; (ii) the inherent margin of error in the CBT database, which is not taken into account when pre-selecting candidates; and iii) the suitability of the quality checks performed by EPSO in order to ensure the reliability of the CBT database. He indicated that EPSO's replies did not allow him to understand whether, in substance, the current CBT system constituted an adequate means for pre-selecting candidates for EU recruitment.

37. In this respect, the Ombudsman points out that a distinction should be made between, on the one hand, the infringement of a procedural duty to provide information that is as exhaustive as possible[7] and, on the other hand, the infringement of that duty as an indicator of a substantive shortcoming in a decision.

38. The Ombudsman considers that, while EPSO's first two replies to the complainant were indeed not exhaustive, EPSO's opinion and its reply to the request for further information on the complaint were, taken together,  complete. In this way, EPSO complied with the procedural duty set out above.

39. A further question arises as to whether EPSO's opinion and further reply provide sufficient elements for assessing whether or not EPSO deals with the erroneous questions in a proper way.

40. As to the frequency of neutralised questions (under point (i)), the complainant argued that EPSO failed to acknowledge and to correct the errors contained in its CBT database. Moreover, in his view, the percentage of neutralised questions is higher than acknowledged by EPSO.

41. In so far as the complainant's conclusions are based on the CBT compulsory self-assessment test introduced for the registration process for 2011 Open Competition EPSO/AD/206/11, the Ombudsman regrets that EPSO, while agreeing with the complainant that one question in the self-assessment test was wrong, did not explain in detail why it considered the three other questions contested by the complainant to be adequate. However, given that all 20 questions included in the compulsory self-assessment test were not part of the CBT database used for EPSO competitions, the Ombudsman considers that no further inquiries are justified as regards this aspect of the case.

42. The Ombudsman further considers that, where the answer to one question out of 20 is incorrect, one cannot reasonably infer that the same is true for all sets of remaining questions, or even for only some of them. Furthermore, the fact that the complainant disagrees with EPSO's assessment of the contested items does not, as the complainant appears to argue, lead to the automatic conclusion that the three questions were erroneous.

43. As regards the frequency of neutralised questions (under point (ii)), EPSO provided detailed statistics, namely: (a) total figures concerning the neutralised questions for the competitions organised from 2006 to 2008; (b) the average percentage of neutralised questions in relation to the total number of questions in the database tests used thus far in the various competitions; and (c) overall numbers and percentages of neutralised questions for some of the competitions organised in 2010 and 2011.

44. Although the complainant is right in pointing out that, from 2006 to 2008, the current new system of competitions was not yet operational, the database of questions was already functioning for the then "admission tests", and the database was adopted for a new system of computer-based tests as such. Thus, the Ombudsman considers that EPSO was right in referring to these specific data in its opinion.

45. The Ombudsman finds that, in view of the complainant's interpretation of the statistics provided by EPSO and of his estimates concerning the number and/or percentage of neutralised questions, EPSO's reply is appropriate.

46. As a general remark, the Ombudsman notes that the total number of items in the CBT database is constantly changing. Furthermore, for each competition, the selection board draws a set pool of questions from the general CBT database. In principle, the pools of questions differ from one competition to another, and the overall number of items used for each pool also varies from one competition to another[8]. Moreover, some questions selected for a competition could be used again in other similar competitions.

47. Therefore, when determining the overall percentage of neutralised questions in relation to the total number of questions in the CBT database, one must consider only the total number of neutralised questions and the total number of items contained in the CBT database at a given time. For that reason, the percentage varies.

48. The reasoning set out above applies, mutatis mutandis, to determining the percentage of neutralised questions per competition. Consequently, on the basis of the total number of neutralised questions in a given competition, the percentage of neutralised questions for that competition is determined  in relation to the overall number of items (pool of questions) drawn by the selection board from the general CBT database for the competition in question.

49. For these reasons, the Ombudsman cannot agree with the complainant's method, which consists mainly of adding the percentages of neutralised questions for each competition (for which EPSO provided the statistics) in order to obtain the average percentage of neutralised questions in relation to the entire CBT database.

50.  In light of the foregoing, the complainant's argument that 'the incidence of neutralised questions is cumulative over each competition in which the questions have been previously used' cannot be upheld. Such reasoning would be accurate if the exact same questions from the CBT database were used over and over again in different competitions.  Even though one could reasonably assume that, given that the CBT database contains a limited number of items, some of the questions will be used in more than one similar competition, it does not follow that the selection process in a given competition is influenced by the sum of neutralised questions in the competition itself and of neutralised questions in similar, subsequent competitions.

51. Furthermore, the Ombudsman notes that, according to EPSO's statistics, the average percentage of questions neutralised in relation to the total number of questions in the CBT database used thus far in various competitions is very low and accounts for only 0.22%. The Ombudsman sees no reason to doubt the above overall figure provided by EPSO, a figure which appears to be corroborated by the low percentages of neutralised questions per competition[9].

52. Finally, in order to determine the impact of neutralised questions on the selection process, it is not enough to consider the percentage of neutralised questions per competition, but the steps taken by EPSO to ensure the equal treatment of candidates in such circumstances must also be considered. In this respect, the Ombudsman finds the explanations EPSO provided in its opinion (paragraph 21 above) reasonable, even if there might be other ways of dealing with that matter.

53. As regards the margin of error in the CBT database which is not accounted for when pre-selecting candidates and in light of his conclusions in paragraphs 47-49 above, the Ombudsman considers that the complainant failed to prove that the margin of error of questions in EPSO's CBT database is 15-20%. Therefore, the Ombudsman takes the view that no further inquiries are required into this aspect of the case.

54. With reference to the quality checks performed regarding the items in the CBT database, the Ombudsman is not persuaded by the complainant's arguments as to why these checks are not sufficient to guarantee the quality of the questions in that database. In this respect, he notes that EPSO provided a complete overview of the procedures carried out in order to ensure the quality of the CBT database.

55. The Ombudsman agrees with the complainant that different scientific methods could be used to assess the validity and reliability of the items in the CBT database. However, this is not sufficient to conclude that the method chosen by EPSO, namely, the Rasch model, is manifestly inappropriate or wrong. Finally, as EPSO correctly pointed out, according to the case-law of the Court of Justice, while the appointing authority has a wide discretion in determining the rules and conditions under which a competition is organised, the selection board has a wide discretion regarding the arrangements for, and the detailed content of, the tests within the framework of a competition. Both the choice and the assessment of the content of the questions set during a competition fall within the remit of the selection board.

56. In these circumstances, the Ombudsman concludes that EPSO properly explained its view. The Ombudsman also concludes that EPSO's procedures for dealing with erroneous questions and its entire system of supervision of the quality of CBT are adequate. The Ombudsman concludes that no further inquiries are justified as regards the complainant's complaint.

B. Conclusions

On the basis of his inquiry into this complaint, the Ombudsman closes it with the following conclusion:

No further inquiries are justified.

The complainant and EPSO will be informed of this decision.

 

P. Nikiforos Diamandouros

Done in Strasbourg on 17 January 2013

---

[1] EPSO referred to the case F-2/07 Matos Martins v Commission, in which the Civil Service Tribunal considered that computer-based tests (containing different questions for each candidate but with an equivalent level of difficulty) ensured objectivity of choice and equal treatment.

[2] The Ombudsman requested EPSO to explain (i) what happens with questions which have been neutralised, namely, whether they are removed from the CBT database or corrected and re-used for the next competitions, and (ii) why EPSO considered adequate the three questions, published in the compulsory self-assessment introduced in the registration process for the 2011 Open Competition, which were contested by the complainant. In addition, he asked EPSO whether it could: (a) provide information (namely, overall figures concerning the entire CBT database) about the number of erroneous questions which were used in competitions and which were subsequently cancelled; (b) explain in detail its internal quality control procedures set up to ensure the reliability of the CBT database (according to the complainant, only one EPSO staff member checks the questions in all EU official languages); and (c) provide information about (i) the scientific data (reports, studies, statistics) which support the validity and reliability of the CBT database for the EU selection process, and also (ii) statistical analyses concerning the results obtained thus far, as of the date the CBT system was established.

[3] For the 2010 AD Generalist cycle, only 6 questions out of 2540 used were neutralised (0.24%); for the 2010 AD Linguist cycle, only 8 questions out of 3526 used were neutralised (0.23%); for the 2010 AST cycle, only 8 questions out of 8898 used were neutralised (0.09%); for the 2011 AD Generalist cycle, only 21 questions out of 6364 used were neutralised (0.33%).

[4] This was the case, for example, with the situational judgement test, which was, at first, a compulsory non-eliminatory test in the 2010 AD and AST cycles, and, later, an eliminatory test in the 2011 competition cycles. Another example was the pre-testing, on a voluntary basis in CAST selection procedures, of the verbal and numerical reasoning test items translated into new languages for the purposes of launching 23-language CBT testing.

[5] According to the complainant, if the database of questions was the same between 2006 and 2008, the total number of neutralised questions was 53 (questions neutralised in 2006 and 2007) + 9 (questions neutralised in 2008) = 62. This represents 62 out of “over 12,000” questions, implying that 0.52% of the questions were wrong.

[6] That is to say, the sum of the questions neutralised in the 2010-2012 competitions.

[7] Article 12 of the European Code of Good Administrative Behaviour reads as follows: "when answering correspondence, telephone calls and e-mails, the official shall try to be as helpful as possible and shall reply as completely and accurately as possible to questions which are asked". (emphasis added)

[8] See footnote 3.

[9] Idem.