Background to the complaint

1. In October 2019, the Research Executive Agency (REA) launched a call for proposals, Blue Growth^[1], concerning the sustainable harvesting of resources from seas, oceans and inland waters.

2. In January 2020, the complainant submitted a proposal for one of the sub-actions under the programme, ’The Future of Seas and Oceans Flagship Initiative’^[2], which aimed to promote technologies for the observation of oceans. The complainant was the coordinator of a consortium composed of international partners. The proposal focused on developing technologies to research marine microorganisms up to 2500m depth.

3. The complainant’s proposal, together with three others, received scores above the minimum threshold for a project to be considered for funding. The REA ranked the complainant’s proposal fourth due to the application of a weighting linked to the award criterion “impact”^[3]. The REA selected the first two proposals for funding, meaning the complainant’s proposal was rejected.

4. In July 2020, the complainant requested a review of the REA’s decision not to fund the project. The Evaluation Review Committee, which dealt with the review, confirmed the result of the initial evaluation. Even though the committee found one error in the evaluation, it considered that this had no impact on the overall assessment of the proposal.

5. In November 2020, the complainant asked the European Commission to review the legality^[4] of the REA’s decision. On 3 March 2021, the Commission annulled the REA’s decision.^[5] It noted that there had been a factual error in the evaluation, which was undisputed by both parties. Furthermore, the total score of the complainant’s proposal was very close to the proposals selected for funding. According to the Commission, these two elements could indicate that the error might have affected the scoring and hence the final result of the evaluation and the ranking of the four proposals that received scores above the minimum threshold. The Commission thus stated that, as the Evaluation Review Committee is composed of administrative staff and not of expert evaluators, the decision of the Review Committee about the impact of such an error went beyond its expertise. The Commission concluded that, in order to safeguard the complainant’s right to a fair evaluation, the proposal should have been re-evaluated by experts.

6. Following the Commission’s decision, the REA re-evaluated the complainant’s proposal and the project received a lower score than in the original evaluation.^[6] On 7 May 2021, the REA therefore informed the complainant that the proposal could not be considered for funding because the two top proposals had scored higher and were consequently selected for funding.

7. The complainant turned to the Ombudsman in August 2021.  

The inquiry

8. The Ombudsman opened an inquiry into the complaint.

9. In the course of the inquiry, the Ombudsman received the reply of the REA and, subsequently, the comments of the complainant in response to the REA's reply. The Ombudsman inquiry team also sought clarifications from the project officer in charge of the complainant’s file and inspected the REA's file on the case, including the ‘individual evaluation reports’ (IERs) of the individual experts performing the evaluation and the subsequent re-evaluation, and the Commission’s internal guidance on how to conduct the evaluation of proposals^[7].

Arguments presented to the Ombudsman

Arguments by the complainant

10. The complainant argued that his consortium’s proposal should have been allocated funds following the re-evaluation, because it reached the minimum score for a project to be considered for funding (even though the score was lower than the initial evaluation). The complainant also expressed doubts about the objectivity of the re-evaluation, as the REA re-evaluated the proposal in isolation and did not re-evaluate the other proposals. Furthermore, the complainant contended that the REA’s evaluation did not adequately take into account, and score, the international character of the consortium.

11. The complainant also contended that, in practice, it is unlikely that a proposal (like that of his consortium) that was not initially granted funding has the same chances of being funded after a re-evaluation as the proposals initially selected.

12. The complainant argued that the REA’s feedback on the evaluation of the proposal - the Evaluation Summary Report (ESR) - was a compilation of expert comments, which were not the result of a decision taken by consensus. The complainant contended that the experts who evaluated his consortium’s project ignored the comments of other experts because the evaluators were not gathered in the same room and had to approve the report electronically.  

Arguments by the REA

13. The REA said that the evaluation process, as well as the re-evaluation, is not a comparative assessment of proposals. Project proposals are assessed individually against the award criteria. The task of the experts is not to decide about the selection of the proposal by comparison with other project proposals, but to evaluate each proposal on its own merits. The REA contended that, as the evaluation is done by experts, it does not judge or compare the independent assessment made by the experts within their professional competence.

14. The REA said that, as the Commission annulled only the evaluation of the complainant’s proposal, it did not re-evaluate the other proposals. This was because the Commission did not identify a procedural shortcoming that would affect all proposals. In line with the applicable rules, the REA awarded funding to the two proposals originally selected, as the re-evaluation of a proposal should not delay the process for selected proposals.^[8] The REA said that, if a re-evaluation results in the recommendation to fund the proposal, it may assign extra funding to allow the proposal to be funded in addition to those that were previously granted funds. The REA clarified that this has happened in the past.

15. The REA said that the initial evaluation of the complainant’s proposal had to be organised online (due to the COVID-19 pandemic) and took place in three phases, as per the standard practice. In the ’individual evaluation’ phase, each evaluator drafted an individual evaluation report (IER) for the proposal, with comments and scores for each award criterion. After that, the individual evaluators got together as a ’consensus group’ and reached a common position on the proposal, setting out the comments and scores for each award criterion in the ‘consensus report’. Finally, a panel review discussion took place to ensure the consistency of comments and scores given for all proposals evaluated and to discuss the draft ranking list of proposals. The purpose of the panel review discussion is to ensure that the evaluators have applied all criteria evenly. It therefore takes place only when there are multiple proposals evaluated by different groups of evaluators, and where more than one proposal has the same score so that the evaluators have to establish a ranking between those proposals.

16. The REA said that, in accordance with applicable rules, the re-evaluation of the proposal was carried out by evaluators not involved in the initial evaluation of the proposal of the complainant’s consortium.^[9] The REA appointed five expert evaluators, the same number of evaluators that evaluated the other proposals. These five evaluators had a diverse expertise, skillset, gender and geographical origin. Two experts were new and not previously involved in the evaluation of the call for proposals. Three experts were part of the initial pool of ten experts that evaluated the eligible proposals, but who had not evaluated the complainant’s proposal. During the initial evaluation, these three experts evaluated between four and five proposals (individual review and then consensus group), but not including the proposal of the complainant’s consortium. Of these three experts, one did not evaluate either of the two proposals that were finally funded, another evaluated one proposal that was funded, and the third expert was involved in the evaluation of the two proposals that were funded. In addition, these experts read the evaluation summary reports of some of the proposals from the initial evaluation, excluding the complainant’s, during the panel review to ensure the consistency of scores and comments in respect of all proposals. This ensured that, although the three experts were familiar with the topic and participated in the initial evaluation of the call for proposals, they had not reviewed the proposal of the complainant’s consortium and their participation in the re‑evaluation was thus objective. In addition, the new set of expert evaluators received the same briefing material as the original experts in relation to the evaluation procedure.

17. The REA said that it applied the same conditions throughout the process, aiming to ensure a harmonised approach between the initial evaluation and the subsequent re-evaluation of the proposal of the complainant’s consortium. The re-evaluation was based on the proposal as it was originally submitted and followed the same procedure as the original evaluation, with the exception of the final panel review stage. No panel review took place because there was only one proposal to be evaluated (and therefore there was only one panel of experts involved) and there was no need to rank the proposal. The project officer responsible for the complainant’s proposal ensured a quality check during the consensus phase by means of a second and third reading aimed at remedying factual inaccuracies.

The Ombudsman's assessment

18. The role of the Ombudsman in cases such as this is to assess whether there was a procedural error or a manifest error of assessment. It is not the role of the Ombudsman to re-evaluate a project proposal or to redo the scientific assessment carried out by experts. The aim of this inquiry was thus to assess whether the procedure by which the REA re-evaluated the complainant’s project complied with the Commission’s decision and was fair.

19. The applicable rules require that the proposals are treated fairly and equally, and that the evaluators agree a common view as to how the criteria are awarded and how the comments are made across all proposals. The rules also state that re-evaluations must be done primarily by evaluators not involved in the initial evaluation.^[10]

20. There is no specific procedure for conducting re-evaluations after the Commission has annulled a decision by an agency not to finance a project. However, it is reasonable that such re-evaluations follow the same procedure as for the initial evaluation and/or that that they provide for the same safeguards in terms of ensuring expertise, independent assessment and peer review, checking consistency, and preventing conflicts of interest.

21. In line with the applicable rules, the re-evaluation was done by evaluators who did not conduct the initial evaluation of the complainant’s proposal. The re-evaluation consisted of a thorough assessment, taking into consideration all the relevant elements of the proposal, and was carried out by experts with different skills and knowledge^[11], relevant to the call for proposals. Their names and fields of expertise are published in accordance with the applicable rules.^[12]

22. The REA provided adequate feedback to the complainant in the ESR and gave a comprehensive explanation as to why the project received the score it did. The documents in the file on the re-evaluation, which were inspected by the Ombudsman inquiry team, show that the scoring and wording of the ESR were the result of a genuine discussion between experts. The evaluators thoroughly debated all aspects of the evaluation and took all decisions by consensus. Through the online tool used by the experts to prepare the consensus report, experts could see and reply to the comments provided by other experts. There is no indication that the re-evaluation of the complainant’s proposal was flawed, for example that the experts misapplied the award criteria or that one expert unduly influenced other experts in an inappropriate or biased way.

23. The REA had no legal obligation to grant funding to all the proposals with a score equal to or above the minimum threshold for funding, and the available budget for The Future of Seas and Oceans Flagship Initiative was not sufficient to fund all such proposals. Consequently, the REA selected only the two highest ranked proposals for funding. The ranking was established based on the overall scores, as well as by applying a weighting to the award criterion ‘impact’. The particular importance attributed to the award criterion ’impact’ is in line with the nature of innovation actions, which focus on the deployment of a technology and its repercussions for society.^[13] In this case, as the re-evaluation resulted in the score for the proposal of the complainant’s consortium being reduced, there was no requirement to assess its relative ranking at the re-evaluation stage. Thus, the REA was not required to allocate extra budget resources to fund the proposal.

24. In the light of the above, the Ombudsman finds that, in the re-evaluation phase, the proposal was thoroughly and objectively assessed by a new panel of experts, who had the requisite expertise and conducted a peer review of their IERs to reach a consensus on the ESR. The re-evaluation is, per definition, a new evaluation, based on the same award criteria but carried out by a different panel of experts. The result may be, like in this case, that the proposal receives a lower score than in the original evaluation.

25. In this case, as the evaluators had to re-evaluate only the complainant’s proposal, they did not need a third tier of the evaluation (the panel review) to ensure consistency of the scoring. Also, given the score awarded in the re-evaluation, there was no requirement to rank the proposal against those who received funding. Furthermore, the experts who were involved in the initial evaluation of the proposal were not involved in the re-evaluation, thereby ensuring the necessary objectivity and independence of the re-evaluation process. The fact that three of the five experts involved in the re-evaluation were also involved in the initial evaluation of other proposals does not undermine their independence. On the contrary, this enabled those members of the new expert panel to rely on that experience in the re-evaluation process. 

26. The Ombudsman did not find any procedural error in how the REA carried out the re-evaluation and finds that the REA ensured a fair re-evaluation of the proposal, in accordance with the Commission’s decision annulling its initial evaluation. Therefore, the Ombudsman finds there was no maladministration in this case.

Conclusion

Based on the inquiry, the Ombudsman closes this case with the following conclusion.

There was no maladministration in how the Research Executive Agency (REA) re-evaluated the project.

The complainant and the Research Executive Agency will be informed of this decision.

 

Emily O'Reilly
European Ombudsman

Strasbourg, 13/09/2022

 

[1] Ref. H2020-BG-2018-2020, the Work programme is available at https://ec.europa.eu/research/participants/data/ref/h2020/wp/2018-2020/main/h2020-wp1820-food_en.pdf

[2] Topic BG-07-2019-2020, with an indicative budget of EUR 18 million.

[3] The project was rated 5/5 for excellence, but 4/5 for impact and 3.5/5 for quality and efficiency of implementation (12.5 points in total). In addition, according to the rules governing the Horizon 2020 programme, the ‘impact’ criterion was given an increased weighting. More information on scoring and weighting can be found in the General Annexes to the Horizon 2020 Work Programme: https://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-2020/annexes/h2020-wp1820-annex-ga_en.pdf

[4] The Commission’s powers in this respect are limited to verifying legal and procedural aspects, as well as the absence of a manifest error of assessment, but do not extend to the merits of the evaluation. See article 22 of Council Regulation (EC) No 58/2003 of 19 December 2022 laying down the statute for executive agencies to be entrusted with certain tasks in the management of Community programmes. The Regulation is available at https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32003R0058&qid=1657188000435

[5] Commission implementing decision of 3.3.2021 on the review of legality of an act of the Research Executive Agency pursuant to Council Regulation (EC) No 58/2003, C(2021) 1159 final.

[6] The project was scored 3.5/5 for excellence, 3.5/5 for impact, 3/5 for quality and efficiency of implementation (a total of 10 points).

[7] Contained in the Egrants vademecum.

[8] Article 16(6) of Regulation (EU) No 1290/2013 of the European Parliament and of the Council of 13 December 2013 laying down the rules for participation and dissemination in “Horizon 2020 - the Framework Programme for research and Innovation (2014-2020) and repealing Regulation (EC) No 1906/2006 available at https://ec.europa.eu/research/participants/data/ref/h2020/legal_basis/rules_participation/h2020-rules-participation_en.pdf

[9] Article 16(4) of the rules of participation.

[10] In accordance with Article 16(4) of the rules of participation.

[11] In accordance with Article 40(2) of the rules of participation.

[12] In accordance with Article 40(5) of the rules of participation. The expert lists are available at the follwing link, at the section Expert names (annual lists): https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/how-to-participate/reference-documents;programCode=H2020.

[13] See annex D of the work programme: https://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2018-2020/annexes/h2020-wp1820-annex-d-ia_en.pdf