Dear Mr P.,

On 1 April 2002, you made a complaint to the European Ombudsman on behalf of Virtual Development Association Limited (VDA) concerning the rejection by the Commission of your proposal "UVOZ" made further to the call for proposals "Energy, environment and sustainable development" (1998-2002).

On 30 April 2002, I forwarded the complaint to the President of the Commission. The Commission sent its opinion on 7 June 2002. I forwarded it to you with an invitation to make observations, which you sent on 19 July 2002.

I am writing now to let you know the results of the inquiries that have been made.

THE COMPLAINT

According to the complainant, the relevant facts were as follows:

The complainant, Virtual Development Association limited (VDA), a UK based association, submitted a proposal "UVOZ" further to the Commission's call for proposals for indirect RTD actions under the specific programme for research, technological development and demonstration on "Energy, environment and sustainable development" (1998-2002) (OJ 2000 C 324/11 of 15 November 2000). The complainant's proposal was rejected.

The complainant considered that the evaluation of its proposal was made in an illegal and biased way, because the Commission did not respect the evaluation procedure and the experts used illegal and wrong assessment criteria.

The complainant therefore wrote several e-mails on the matter to DG Research, asking for its proposal to be re-evaluated. The Head of Unit of the Key Action "Sustainable management and quality of water" replied that the evaluation of the proposal had been conducted in line with the rules of the Evaluation Manual and that a reconsideration of the proposal was not possible. Commissioner Busquin later confirmed this evaluation.

On 1 April 2002 the complainant made the present complaint to the Ombudsman, making the following four allegations and claim:

1. The Commission did not respect the evaluation procedure, as step 2 of the evaluation (blocks 2, 3 and 4) was not executed.

2. The experts used two illegal and wrong assessment criteria, which are not mentioned in the evaluation manual, namely 1) the fact that the complainant's partner is "not well known internationally" and 2) "the obligation to ensure, to provide equality or balance of efforts, funding, resources of proposers".

3. From the "general remarks" and the "weak points" contained in the evaluation of the complainant's proposal, the complainant deduces that the evaluation was biased.

4. The Commission should carry out a new independent evaluation of the proposal by new experts and other officials.

THE INQUIRY

The Commission's opinion

The proposal EVK1-2001-00277 (Development of a new cost-effective and environment-friendly UV technology for water treatment with wide applications - UVOZ) was submitted by the complainant to the Sub-Programme Environment and Sustainable Development (ESD), priority area "Sustainable Management and Quality of Water".

The Evaluation Manual describes that each proposal is to be evaluated by independent experts in three different steps, subdivided into various blocks or group of blocks: block 1 (scientific/technological quality), blocks 2, 3 and 4 (Community added value and contribution to EU policies, contribution to Community and social objectives, and economic development and S&T prospects), and block 5 (resources, partnership and management). After each step a threshold level has to be reached in order to pass to the evaluation of the following step(s). Failure to reach any threshold stops the evaluation after that step. However, the Evaluation Manual does not prescribe in which order those steps have to be evaluated, except that the evaluation logically has to start with the anonymous part of the proposal (block 1).

Since 1999, for reasons of practicality and strategic importance to the Programme, the ESD Programme has fixed the following order for all evaluations: Step 1- block 1; step 2 - block 5; and step 3 - blocks 2, 3 and 4.

Since the proposal UVOZ failed to pass the threshold of step 2, the evaluation stopped after this step and step 3 was not evaluated. This corresponds entirely to the evaluation rules as described in the Evaluation Manual. Hence, no violation of the evaluation procedure has taken place.

Five independent experts, selected on the basis of their specific expertise from the commission's Experts Database, evaluated the UVOZ proposal. They reached a consensus as set down in the Consensus Evaluation Report, which was sent to the proposal co-ordinator. In addition, the evaluation process has been observed by an independent observer panel which has not brought forward any critical remarks. The Commission official who moderated the consensus evaluation discussion among the experts had a neutral role and did not influence the conclusions of the expert panel. His role is restricted to act as mediator, in cases where no consensus can be reached among the 5 experts, but this did not apply to the consensus discussion on the UVOZ proposal. Hence, there was not unfair treatment by the Commission personnel.

Any evaluation of the resources, partnership and management of a proposal (Step 2) has to be based on information provided by the applicants concerning e.g. recent publications on the subject of the proposal, the CV's of the proposal partners (in order to judge their scientific/technological competence), the role of partners in the execution of the proposed programme of work, the description of how the proposed project will be managed (management structure) and how the various tasks and milestones have been set up. No additional evaluation criteria have been applied by the expert panel.

In the case of Partner 2, the evaluation report noticed the lack of a CV and of a list of recent publications which would have allowed to judge its quality. This led the evaluation panel to express "major concern about the role of Partner 2" and to state that this partner is "not well known internationally". The Commission cannot see in these statements any unfair judgement by the evaluation panel.

Through several oral and written communications between the complainant and the responsible Head of Unit of the Key Action of Sustainable Management and Quality of Water, the elements given above were explained in detail, but apparently the complainant could not be satisfied.

The complainant's observations

The complainant maintained his complaint and pointed to various infringements of the evaluation procedure:

The complainant indicated that nowhere in the Guide for Proposers (November 2000 edition) it is stated that the evaluation stops after any step. All steps of the evaluation must be executed.

The complainant also stated that, according to Annex K of the Evaluation Manual (edition of 14 July 2000), block 5 (step 3) must be evaluated at the end of the procedure. Common sense requires that all names and companies must be evaluated only in the last step.

As regards the lack of CV of Partner 2, the complainant observed that Partner 2 was a new company whose personnel invented the proposed technology. The complainant pointed out that all the CV's are excellent and that the scientists are well known internationally. The complainant further added that recent publications were not required by part C of the Evaluation Manual. All of the publications were supplied in part B of the proposal. Part C only requested short CV's.

The complainant thus confirmed that the judgement on block 1 and 5 were biased.

THE DECISION

1 The alleged failure to respect the evaluation procedure

1.1 The complainant alleged that the Commission did not respect the evaluation procedure, as step 2 of the evaluation (blocks 2, 3 and 4) was not executed.

1.2 The Commission observed that the Evaluation Manual does not prescribe in which order the steps have to be evaluated. It pointed out that since 1999, for reasons of practicality and strategic importance to the Programme, the ESD Programme has fixed the following order for all evaluations: Step 1- block 1; step 2 - block 5; and step 3 - blocks 2, 3 and 4. Since the proposal of the complainant failed to pass the threshold of step 2, the evaluation stopped after this step and step 3 was not evaluated. This corresponds entirely to the evaluation rules as described in the Evaluation Manual.

1.3 The Ombudsman notes that the evaluation procedure is contained both in the Evaluation Manual(1) (edition July 2000) and the Guide for Proposers with the former prevailing over the latter. The Guide for Proposers provides that "the evaluation of the individual proposal will be carried out in three steps each having a separate threshold and ending in a "go/no go" decision".

1.4 Annex K to the Evaluation Manual provides that stage 1 of the evaluation procedure (evaluation by the various panels) will be divided into 3 steps: "The step 1 evaluation will consist of evaluating scientific and technological value, including innovation (block 1). The identity of the participants will not be disclosed to the evaluators. At the end of this step, the evaluators will make a "go/no go" recommendation, depending if the threshold for step 1 has been reached or not. At step 2, the evaluators will assess "Community added value and contribution to EU policies" (block 2), "Contribution to Community social objectives" (block 3) and "Economic development and S&T prospects (block 4). The identity of the participants will be known. At the end of this second step, a "go/no go" recommendation will be made depending if the minimum threshold for step 2 has been reached. At step 3, the evaluators will assess "Management and resources" (block 5). At the end of this, a "go/no go" recommendation will be made depending if the minimum threshold for step 3 has been reached". Annex K concludes that "only the proposals which have reached the minimum threshold for steps 1, 2 and 3, i.e. those which received a go recommendation after all three steps, will proceed to stage 2".

1.5 It appears from the above that the order of evaluation, as communicated to the participants, is first step 1 (block 1), then step 2 (blocks 2, 3 and 4) and finally step 3 (block 5). It further appears that after each step a threshold level has to be reached in order to pass to the evaluation of the following step and that failure to reach any threshold stops the evaluation after that step.

1.6 The Ombudsman notes that nothing in the documents provided to the participants indicates that, as the Commission claims, the evaluation of step 3 (block 5) is carried out before step 2 (blocks 2, 3 an 4). In its opinion, the Commission did not point out where this rule, which substantially changes the order of evaluation - and thus the possible exclusion of participants - can be found. It was only a posteriori, namely after the complainant's proposal was rejected, that the responsible Head of Unit indicated to the complainant that it is more practical to evaluate first block 5, because most of the failures fall in this block and that, if a proposal fails there, there is no need to spend more time in evaluating the remaining parts.

1.7 On basis of the above, the Ombudsman considers that the order in which the Commission evaluated the complainant's proposal was different from the one communicated to the participants in the Evaluation Manual. It appears therefore that the Commission did not follow the rules of the evaluation procedure it had itself established and communicated. The Commission has however explained to the complainant the reasons for the change in the order of the evaluation. This change does not seem to have affected the outcome of the evaluation. No instance of maladministration was therefore found.

2 The alleged illegal and wrong assessment criteria

2.1 The complainant alleged that the experts used two illegal and wrong assessment criteria, which are not mentioned in the evaluation manual, namely 1) the fact that the complainant's partner is "not well known internationally" and 2) "the obligation to ensure, to provide equality or balance of efforts, funding, resources of proposers".

2.2 The Commission observed that, in the case of Partner 2, the evaluation report noticed the lack of a CV and of a list of recent publications which would have allowed to judge its quality. This led the evaluation panel to express "major concern about the role of Partner 2" and to state that this partner is "not well known internationally".

2.3 The Ombudsman has carefully considered the terms of the evaluation made by the experts, as well as further clarifications provided to the complainant by the responsible Head of Unit in his replies to the complainant's e-mails. With regard to the points raised by the complainant, the Commission indicated that the panel gave a mark of 2 for the evaluation of resources, partnership and management, because they felt that the resources were not balanced among the partnership, namely because if a single partner covers more than 60 % of the requested budget, the European dimension is very limited.

2.4 On basis of the explanations provided by the Commission, the Ombudsman has not found that any illegal or wrong assessment criteria would have been used by the expert panel. No instance of maladministration was therefore found with regard to this aspect of the case.

3 Alleged bias

3.1 From the "general remarks" and the "weak points" contained in the evaluation of the complainant's proposal, the complainant deduces that the evaluation was biased. The Commission stated that the complainant's proposal has been fairly assessed by five independent experts.

3.2 The Ombudsman considers that on basis of the evidence available to him, no indication of bias could be established. No instance of maladministration was therefore found with regard to this aspect of the case.

4 The claim for a new evaluation of the proposal

4.1 The complainant claims that the Commission should carry out a new independent evaluation of the proposal by new experts and other officials.

4.2 Considering the conclusions under points 1 to 3, there appear to be no grounds for the complainant's claim to have its proposal reconsidered by other experts and officials.

5 Conclusion

On the basis of the Ombudsman's inquiries into this complaint, there appears to have been no maladministration by the Commission. The Ombudsman therefore closes the case.

The President of the Commission will also be informed of this decision.

Yours sincerely,

 

Jacob SÖDERMAN

---

(1) Manual of Proposal Evaluation Procedures. This manual can be found on the cordis-website at the following address: http://www.cordis.lu/fp5/management/eval/r-em-01.htm#titre.