The background to the complaint

1. The complainant is a scientific researcher. In 2010, she applied for a European Research Council ('ERC') Advanced Grant and submitted a proposal entitled "Creating and Sustaining Scientific Authority: the Mathematical, Medical and Geographical Sciences in Four Islamic Societies (Ayyubids, Mamluks, Timurids, Safavids) 1200 - 1700".

2. On 29 November 2010, the complainant was informed that her proposal had been rejected.

3. The complainant challenged that decision by means of a request for redress submitted to the European Research Council Executive Agency ('ERCEA'). On 14 January 2011, she received an acknowledgement of receipt of her redress request.

4. On 14 February 2011, the ERCEA informed her of the final decision on her redress request. It stated that her case had been carefully reviewed by an internal Redress Committee, whose role was to consider whether there had been a failing in the evaluation process, and whether this failing was likely to jeopardise the decision reached in respect of her proposal. The Committee did not evaluate the proposal itself nor did it call into question the scientific judgement of appropriately qualified experts/reviewers. It considered that the complainant's request for redress questioned the scientific judgement of the review panel, which was outside its remit. The Redress Committee moreover found that no procedural error had been made in the evaluation process. The complainant's proposal had been evaluated by independent experts who were all relevant specialists in the field of expertise concerned. The latters' individual evaluations constituted their independent opinion. They were made prior to the panel meeting during which the panel thoroughly discussed her proposal. The final decision was based not only on the individual opinions of the reviewers but also on the outcome of a discussion within the panel and on her proposal's ranking against other proposals. The final evaluation reflected the consensus decision of the panel as a whole and was not necessarily in accordance with every opinion expressed by each individual reviewer. In view of the Redress Committee's recommendation, the ERCEA confirmed its initial decision on the complainant's proposal.

5. On 2 April 2011, the complainant turned to the President of the ERC.

6. In her reply of 28 April 2011, the President of the ERC stated that she was convinced that the complainant's suspicion that the comments of one particular reviewer had determined the panel's decision was incorrect. She pointed to what she referred to as fierce competition for funding and stated that the fact that the complainant's proposal had made it to the second round of the selection procedure confirmed that she is a top European scholar in her field of research. As regards the redress procedure, the President of the ERC explained that that procedure could not reverse the scientific judgement of a panel but instead was designed to verify compliance with procedural requirements.

7. In a further letter sent to the President of the ERC on 28 April 2011, the complainant pointed out that she did not seek to have the panel's decision reversed. Instead, she wished to draw attention to certain serious problems in the evaluation procedure and call on the President to remedy them for the future. The complainant stated that her assertion that the comments of one specific reviewer determined the panel's decision did not merely constitute a suspicion, given that the decision corresponded almost word for word to the comments of that particular reviewer. She moreover submitted that it was highly problematic that panel assessments could not be reviewed, even if they were inappropriate or even hostile, as the assessments concerning her proposal were. According to her, there should be a possibility to challenge such panel assessments. She also took the view that there should be a requirement for reviewers to have a professional background in the field of the proposals which they reviewed. Moreover, she referred to two other proposals which received funding but which were, in her view, significantly deficient. By way of conclusion, she submitted that the ERC's "principled decision" not to allow for any discussion of panel reviews was highly debatable.

8. In her complaint to the European Ombudsman dated 11 July 2011, the complainant pointed out that she did not receive a reply to her letter of 28 April 2011.

The subject matter of the inquiry

9. The complainant submitted the following allegation and claim in her complaint:

Allegation:

The ERCEA failed properly to respond to the complainant's request for redress.

Claim:

The ERCEA should ensure that its redress procedure deals adequately with complaints that reviewers: lack professional expertise in the field; misrepresented proposals; included personal attacks against applicants in their comments; and failed to review the actual proposal submitted.

10. The Ombudsman notes that, in her observations on the ERCEA's opinion, the complainant insisted that the ERCEA should replace the current redress system, which is limited to a review of procedural aspects, by a full review system. Since this claim did not form the subject matter of the opened inquiry, the Ombudsman does not consider that it is justified to widen the scope of the inquiry to include these concerns. In any event, there would be no need to do so because the Ombudsman has already had occasion to clarify that the fact that the Redress Committee cannot (re)-assess the scientific opinions of the panels and the individual reviewers constitutes a "limitation which is entirely common in appeal and review contexts, and is not in any way specific to applications concerning scientific issues. In other contexts – for instance, the recruitment or the evaluation of other types of applications – very similar 'standards of review' are applied, so as to respect the 'wide discretionary powers of' the initial assessors in question. In such contexts, ensuring full compliance with procedural rules is, for obvious reasons, of particular importance. If the substance cannot meaningfully be reviewed as such, the control organ (be it administrative or external, as in the case of an ombudsman or a court) is implicitly called upon to pay particularly close attention to the procedural aspects of the case"[1]. The Ombudsman does not see any reason, in the circumstances of the case at hand, to depart from this position.

11. In her observations, the complainant also claimed that, given that the Redress Committee "already investigates complaints against the qualification of reviewers, inappropriate reviews and other shortcomings of the reviewing process ..., it should make this part of the redress public..." The persons requesting redress should "receive a report about decision making of the Redress [Committee] and not merely a final verdict as is the case currently." According to the complainant, limiting redress to procedural aspects violates the principles of transparency and fairness. This claim, like the previous one, did not form the subject matter of the opened inquiry and the Ombudsman does not consider it justified to extend its scope. In fact, the complainant raised the issue of publicity for the first time in her observations and cannot therefore be regarded as having complied with the requirement of prior administrative approaches (Article 2.4 of the Statute of the European Ombudsman).

The inquiry

12. On 31 August 2011, the Ombudsman opened an inquiry and asked the ERCEA to submit an opinion.

13. On 25 November 2011, the ERCEA sent its opinion. On 6 December 2011, the ERCEA also sent its translation into German.

14. On 30 November 2011, the Ombudsman forwarded the ERCEA's opinion to the complainant.

15. On 27 December 2011, the complainant sent her observations on the ERCEA's opinion.

The Ombudsman's analysis and conclusions

Preliminary remarks

16. The present complaint concerns the adequacy of the response given by the ERCEA Redress Committee to the complainant's request for redress following the rejection of her scientific proposal.

17. At the outset, the Ombudsman observes that it is not materially possible for him to reassess the substantive, scientific assessment of the specific proposal submitted by the complainant to the ERCEA. Therefore, in accordance with his consistent practice[2], his review can focus only on procedural aspects and on verifying that the Redress Committee made no manifest error of assessment in the response it gave to the complainant's redress request.

A. Alleged inadequate response to the complainant's request for redress

Arguments presented to the Ombudsman

18. In support of her allegation, the complainant argued in her complaint that the review panel relied exclusively on the negative assessment of one reviewer and disregarded the positive assessments of the four other reviewers. The Redress Committee and the President of the ECR did not want to recognise this fact. As regards the negative assessment of one of the reviewers, he/she made incorrect assumptions about her proposal. Instead of evaluating her proposal, he/she effectively requested that the complainant submit another proposal with a different methodology. Moreover, the assessment contained disparaging and discriminatory statements.

19. In its opinion, the ERCEA explained that the review carried out by its Redress Committee is limited to procedural aspects. In fact, pursuant to Section 3.1.9(4) of the ERC Rules for submission of proposals[3], the Committee's task is to review the "CVs of the experts, the individual comments, and ER (Evaluation Report)". Accordingly, the Committee does not "evaluate the proposal". This role of the Committee is clearly explained in the ERC Guide for Applicants, which the applicants are invited to consult before submitting their application. Section 3.2.5 of the Guide states that the "redress procedure is not meant to call into question the scientific judgement made by the peer review panel; it will mainly look into procedural shortcomings." It further explains that the Committee "does not re-evaluate the proposal" but might send it back to the panel for re-evaluation only "if there is clear evidence of a shortcoming that could affect the eventual funding decision."

20. However, even if the Committee does not evaluate the proposal as such, it does review the individual comments of panel members who evaluated the proposal to the extent that it verifies whether an individual assessment and the related comment have been carried out and drafted in compliance with the established legal framework and ascertains that they do not present an error or shortcoming that could affect the final decision. Section 7.3 of the ERC Redress Guidelines provides typical examples of such shortcomings. These are (i) doubts over an expert's competence or impartiality, (ii) misinterpretation by the expert, (iii) factual errors, (iv) inappropriate comments. In the case at hand, the Redress Committee acknowledged that some of the comments of Reviewer 1 were "harsh and/or inappropriate" but considered that this "did not present shortcomings". The comment of Reviewer 1 amounted to a scientific judgement which could not justify re-evaluation. Moreover, the panel's decision was based on all the marks and comments of all the reviewers.

21. The ERCEA further observed that, before the Redress Committee's meeting, the Scientific Officer of the ERCEA, who is in charge of the supervision of the review panel, is requested to provide information to the Committee on the evaluation and on the way it is carried out. He is also requested to provide comments on the redress request. In the present case, the Scientific Officer observed in a note to the Redress Committee dated 25 January 2011, that the proposal had been "discussed intensively and all reviews" had "been included in the decision". He further confirmed that "all statements that the applicant disagrees with are a matter of scientific judgement and should not constitute arguments for sustaining redress".

22. The ERCEA submitted that the panel did not merely rely on the negative assessment of one reviewer and did not disregard the positive assessment of the remaining four reviewers. In fact, the final decision is based on the final mark given by the entire panel. That mark is usually based on an average of the individual marks awarded by all reviewers. As confirmed by the internal version of the evaluation report, in the complainant's case, her final mark was also based on an average of the individual marks awarded by all five reviewers. Thus, the view of Reviewer 1 as expressed in his/her comments was not decisive and the views of other reviewers were duly taken into account. Moreover, even if the comments expressed by Reviewer 1 may have appeared negative, the marks he/she awarded were not. In fact, he/she awarded 3 marks for the criterion 'Principal Investigator' and 2.5 marks for the criterion 'Research project' on a scale of 1-4, where 4 is the highest mark. Such marks are considered high in ERC evaluation procedures. Therefore, Reviewer 1 cannot be deemed to have been prejudiced against the complainant. The ERCEA stressed that the complainant received a positive assessment (6.3 marks out of 8) but other proposals were better. For the reasons outlined above, there were no grounds for the ERC President to uphold the complainant's allegation.

23. In reply to the complainant's claims, the ERCEA observed that, in general, panels are composed of high-level scientists who are experts possessing recognized professional experience in their field of research. They are established in order to cover the research areas of the Call. In the complainant's case, the reviewers who dealt with her proposal were experienced scientists in the specific field and no lack of professional expertise was ascertained during the redress procedure. According to the ERCEA, it must be obvious even to non-professionals that the reviewers' comments concerning the complainant's proposals showed a very good knowledge of the subject. Reviewer 1 in particular made an extensive analysis of the proposal whilst referring to current theories and alternatives. Although his comments may be considered inappropriate, they were certainly relevant to the subject.

24. Moreover, the reviewers' comments are based on their professional knowledge and experience and, although the complainant may disagree with them, those comments were judged correct by the Scientific Officer and the Redress Committee. Therefore, the complainant's proposal was not misrepresented.

25. In addition, although the Redress Committee considered the comments of Reviewer 1 to be "harsh and/or inappropriate", they did not contain any personal attack against the complainant. This is evidenced by the fact that he/she awarded 3 marks for the criterion 'Principal Investigator'.

26. In the ERCEA's opinion, Reviewer 1 clearly did not fail to evaluate the proposal. His/her comments clearly show that he/she did evaluate it. This was also acknowledged by the Scientific Officer and the Redress Committee.

27. In her observations on the ERCEA's opinion, the complainant pointed out that her experience shows that hostile evaluation is endemic to the humanities and social sciences. This is due to lobbies, networks, fundamental differences in methodologies, approach or other forms of belief and lack of specialist expertise. Acknowledging this fact implies the necessity to establish adequate redress procedures. This is all the more important in "small" fields such as the one in which the complainant is active, where expertise is rare and serious conflicts about working methods, criteria and judgements abound. As a minimum, one specialist in each of the main disciplinary affiliations of a project proposal should participate in the peer review. Rejections based on methodological or ideological reasons alone should not be possible, since they deny the rights of an applicant to pursue alternative or opposing research agendas. It should be possible to challenge the fact that the reviewers are not specialized enough or that their negative assessment is based on the fact that their research position is fundamentally different from the one of the applicant.

28. The complainant never contested that the numerical results corresponded to the average of the points awarded by the individual reviewers but criticized the fact that the final verbal summary of the evaluation of her project was a "more or less literal copy of the negative evaluation" of Reviewer 1. She added that, if it had merely been an issue of numbers, the panel could have simply said so.

29. The complainant went on to state that the ERCEA's statement that it was obvious even to a non-professional that the reviewers' comments show a very good knowledge of the subject is an oxymoron, since a non-professional lacks the necessary expertise to judge this point.

30. Finally, the complainant pointed out that her negative experience with the ERCEA's practices has been repeated in the context of another Call, launched in 2011, in which her proposal was also rejected under comparable circumstances.

The Ombudsman's assessment

31. The Ombudsman reiterates that the complainant alleged in her complaint that the ERCEA failed properly to respond to her request for redress. Although her allegation challenges the review carried out by the Redress Committee, her arguments in fact relate to the evaluation of her proposal by the panel. She argued that (i) the panel disregarded the positive assessment of four reviewers and exclusively relied on the negative assessment of one reviewer; (ii) the negative assessment made incorrect assumptions about her proposal; (iii) instead of evaluating her proposal, the reviewer effectively requested the submission of another proposal with a different methodology; (iv) he/she made disparaging and discriminatory statements; and (v) the panel members lacked expertise. The Ombudsman understands the complainant's position to be that she would only have considered the redress procedure to have been carried out properly if the Redress Committee had taken into account all her arguments when replying to her request for redress.

32. In so far as these arguments cover substantive aspects, the Ombudsman points out[4], as rightly stated by the ERCEA in the opinion, that, pursuant to Point 3.1.9 of the ERC Rules for submission of proposals "[t]he [redress] committee ... does not evaluate the proposal. Depending on the nature of the complaint, the committee may review the CVs of the experts, the individual comments, and [the Evaluation Report]."

33. On the basis of the above, it is clear that a redress procedure is not intended to repeat the assessment of the proposal itself. In fact, it is essentially limited to verifying that no procedural error occurred during the evaluation procedure. However, the Ombudsman has held that the Redress Committee should, without carrying out a full review of the scientific assessment and without exceeding the limits of the powers conferred on it, be able to correct any manifest errors of assessment[5]. In the present case, the Redress Committee apparently found that there was no manifest error of assessment. The issue arises, therefore, whether the Redress Committee came to this conclusion after having properly dealt with the complainant's arguments (i)-(v) summarised in point 31 above.

34. First, as regards the complainant's argument that, instead of evaluating her proposal, Reviewer 1 effectively requested the submission of another proposal with a different methodology (argument (iii)), the Ombudsman takes the view that the Redress Committee was right not to consider this argument to be grounded. In fact, the comments of Reviewer 1, as reproduced in the Evaluation Report which was sent to the complainant, clearly show that he/she did review the complainant's proposal, and indeed did so rather rigorously. His evaluation was factual and clearly reasoned. The fact that the reviewer was not satisfied with methodological aspects of that proposal cannot alter this conclusion.

35. Second, as regards the complainant's argument that the negative assessment of Reviewer 1 made incorrect assumptions about her proposal (argument (ii)), the Ombudsman finds that this argument is not supported by any evidence. In any event, it is not for the Ombudsman to call into question the scientific judgement of individual members of a review panel and there does not appear to be any manifest error of assessment. Nevertheless, as stated above, the substantive assessment is not within the Redress Committee's remit.

36. Third, as regards the complainant's argument that the assessment of Reviewer 1 contained disparaging and discriminatory statements (argument (iv)), the Ombudsman considers that this argument is not well-founded either. Although it is true that some comments may have been "harsh", as the ERCEA itself acknowledged, the Redress Committee is right to consider that, as explained by the ERCEA in its opinion, this does not constitute a shortcoming invalidating the final decision. In fact, the tone of those comments, though not courteous, does not automatically imply that Reviewer 1 was biased vis-à-vis the complainant, that she was discriminated against during the review process or that those comments constituted an attack on the individual concerned. In the Ombudsman's view, although other formulations would in fact have been preferable, the comments do reflect the style of a scientific judgement of an individual reviewer and still remain within the boundaries of academic or scientific freedom of expression. They are thus not disproportionate in the context of an evaluation procedure.

37. Moreover, the contested statements need to be read in conjunction with another part of Reviewer 1's comments, which depicts the complainant and her scientific activities in a rather positive way. ERCEA's submission that Reviewer 1 awarded the complainant three marks out of four for the criterion 'Principal Investigator' confirms this positive stance.

38. In addition, the Ombudsman does not consider that the conclusion of the Redress Committee was vitiated in any event by its decision not to refer the case to a new panel for reassessment, even though it concluded that the comments were "harsh and/or inappropriate". In fact, the Ombudsman cannot see how the manner in which certain comments were expressed by one reviewer could affect the panel's final, substantive, scientific assessment of the proposal.

39. Fourth, as regards the complainant's argument that the panel exclusively relied on the negative assessment of one reviewer and disregarded the positive assessments of the four other reviewers (argument (i)), the Ombudsman considers the Redress Committee's explanation that "the final evaluation result of the proposal reflects the consensus decision of the panel as a whole" to be correct. The conclusion that the overall assessment was not based on an individual assessment of one specific reviewer but on all the reviewers' comments is corroborated by the evidence submitted to the Ombudsman[6].

40. Although there are similarities between the panel's comments and those of Reviewer 1, the Ombudsman does not agree with the complainant that the former constitute an almost literal reproduction of the latter. Moreover, the Ombudsman finds it natural that, as explained in the comments, the critical remarks of Reviewer 1 were discussed by the panel. It is also worth noting that another reviewer, Reviewer 3, also expressed critical remarks with respect to the methodology of the proposal. In addition, it cannot be disregarded that the panel's comments also include positive statements.

41. Furthermore, the Ombudsman notes that the panel concluded that the proposal was of good quality and fundable but the main reason for its rejection was that it did not reach a sufficiently high position in the ranking order to be chosen for funding. In that respect, the Ombudsman finds convincing and satisfactory the explanation provided by the ERCEA that the final decision was based on the final mark given by the entire panel, which was based on an average of the marks awarded by all the reviewers. In the absence of evidence showing that there was a manifest error of assessment, the Ombudsman cannot examine whether an alleged incorrect assessment by one of the reviewers could have decreased the average mark and thus could have had a decisive impact on the final ranking of applicants. Thus, it is not necessary to examine whether the marks awarded by Reviewer 1, although already relatively high as explained by the ERCEA, should have been higher.

42. Fifth, as regards the issue of lack of expertise of the panel members (argument (v)), the Ombudsman has not been presented with any evidence in support of that assertion. Quite to the contrary, and although the Ombudsman lacks specific expertise in the scientific area concerned, he agrees with the ERCEA that the comments appear, at least at first sight, to have been made by persons possessing sufficient qualifications.

43. In this regard, the Ombudsman attaches weight to the ERCEA's explanation that the panels are composed of high-level scientists who are experts of recognized professional experience in their field of research and that they are organized in such a manner as to cover the research areas of the call. According to the ERCEA, the reviewers who dealt with the complainant's proposal are experienced scientists in the specific field and no lack of professional expertise was ascertained during the redress procedure. The complainant's argument that in very specialised areas of knowledge, a very small number of specialists exist and, therefore, personal animosities or other subjective factors may have a negative impact on the assessment of proposals in such areas, is not sufficient to call into question the ERCEA's assurances on the quality of the experts who evaluated the complainant's proposal.

B. Conclusions

On the basis of his inquiry into this complaint, the Ombudsman closes it with the following conclusion:

The Ombudsman finds no instance of maladministration corresponding to the complainant's allegation. Consequently, the complainant's claim cannot be sustained.

The complainant and the ERCEA will be informed of this decision.

 

P. Nikiforos Diamandouros

Done in Strasbourg on 29 May 2013

---

[1] See the Ombudsman's decision on complaint 485/2008/IP, of 16 December 2010, point 49.

[2] See the Ombudsman's decision on complaint 939/2009/DK, of 14 October 2010, points 27-28, and decision 2173/2011/OV, of 23 March 2012, point 18.

[3] Commission Decision C(2007) 2286 of 6 June 2007, as amended by Decision C(2007) 4429 of 27 September 2007.

[4] See the Ombudsman's decision on complaint 939/2009/DK, of 14 October 2010, points 37-38, and decision on complaint 705/2009/DK, of 7 February 2011, points 26-27.

[5] See the Ombudsman's decision on complaint 2173/2011/OV, of 23 March 2012, point 18.

[6] The Evaluation Report was marked confidential.