Entscheidung in der Sache 485/2008/(IG)(IP)PB - Fehlerhafte Überprüfung eines Antrags auf Rechtsbehelf

The background to the complaint

1. This complaint is about an Italian researcher who was unhappy with the way in which the European Research Council (ERC), an EU executive agency, handled his appeal against a decision to reject his research proposal.

2. Following an initial assessment, the ERC informed the complainant that his proposal had been rejected. The complainant lodged an appeal (a 'redress request') in accordance with the formal redress procedure. Following reminders and replies thereto, the ERC informed the complainant that his appeal was unsuccessful.

3. The complainant turned to the European Ombudsman, making the allegations set out further below.

4. To understand the present case, it is important to know that the ERC was launched only in 2007, and that the Commission continued to carry out some of the Agency's main tasks for a significant period of time. For the purpose of the Ombudsman's examination in the present case, the relevant institution is the Commission, more specifically the Commission's Directorate-General for Research. This Ombudsman's decision also refers, however, to 'the ERC' in several places, since much of the correspondence was formally exchanged with this Agency.

The subject matter of the inquiry

5. The Ombudsman opened an inquiry into the following allegations and claim:

Allegations:

(1) The decision concerning the complainant's appeal was late, in light of the promise of the ERC to conclude the process in early September, thereby causing the complainant to incur additional costs;

(2) The letter informing the complainant of the appeal procedure's outcome failed to respond to the specific arguments made by the complainant in his appeal, namely, that:

(a) Reviewer 3 erred by applying a criterion in relation to the experience as team leaders already gained by candidates; and

(b) Reviewer 3 applied stricter standards concerning the "quality of the proposed research project" and its "methodology" than those provided for in the Call and the ERC Guide for applicants, in particular, by making reference to the complainant's project not being sufficiently "precise" and "well-defined", despite the fact that applicants were only required to provide a "brief description" of scientific and technical aspects of the project proposal.

Claim:

The complainant should be compensated for the extra expenses he incurred during the period of delay concerning the outcome of his appeal, namely, from early September until 27 November 2007.

The inquiry

6. On 7 April 2008, the Ombudsman asked the European Commission to submit an opinion on the complaint. The Commission submitted its opinion on 9 July 2008. The opinion was forwarded to the complainant, who submitted his observations on 25 August 2008.

The Ombudsman's analysis and conclusions

A. Allegation of delay and related claim

Arguments presented to the Ombudsman

7. The complainant alleged that, taking into account the ERC's promise to conclude the process in early September, the decision concerning the complainant's appeal was late, thereby causing him additional costs. He made the related claim that he should be compensated for the extra costs he incurred because of the delay which lasted from early September until 27 November 2007. These extra costs were the result of the need to remain physically close to the place where a potential continuation of the evaluation procedure would take place, in the event the redress request proved successful. Both issues will be dealt with together in the following paragraphs.

8. In its opinion, the Commission noted that it received the complainant's redress request on 8 August 2007. An acknowledgement of receipt was sent on 9 August 2007. The final result of the redress procedure was communicated to the complainant on 27 November 2007.

9. The Commission acknowledged that there had been a delay in its processing of the appeal. In its opinion, the Commission apologised to the complainant for this delay. It pointed out that the delay was due to the large number of requests for redress and the thoroughness with which the Redress Committee examined each request.

10. With regard to the specific content of the allegation and claim here concerned, the Commission pointed out the following:

(a) The acknowledgment of receipt contained a reservation about the date when the results of the redress procedure would be available: "...and is expected to be available in early September 2007". The Commission did not find that this could be considered to constitute a firm commitment.

(b) There was an extensive exchange of correspondence between the Commission and the complainant during the intervening period. The Commission's correspondence of 7 September 2007 was particularly relevant. It clearly informed the complainant that, if the Redress Committee decided that his proposal should be re-evaluated, and if his proposal was then retained for the second stage, the standard deadline for that second stage (17 September 2007) would not apply. It also clarified that the applicant would be informed as soon as the Redress Committee had reached a decision.

11. Therefore, the complainant was at all times aware of both the status of the redress procedure (i.e., that it was delayed) and of the uncertainty of the outcome. At no point did the Commission raise any expectations as to its positive outcome.

12. In his observations, the complainant maintained his position.

The Ombudsman's assessment

13. The present part of the case raises two issues. The first concerns whether a delay occurred and the second concerns the possible consequences of such a delay.

14. As regards the first issue, the Commission fully recognised that a delay occurred, and it explained the reasons for it. The Commission also apologised to the complainant for the delay. The Ombudsman therefore considers that this issue has been adequately clarified and dealt with, and that no further inquiries are necessary.

15. As regards the second issue, the Ombudsman shares the Commission's view outlined above. In cases like the present one, financial compensation can only be paid if the actions of the Administration led the individual concerned to entertain relevant legitimate or reasonable expectations, or if there are compelling considerations of fairness. After examining the correspondence between the complainant and the Commission in the present case, the Ombudsman does not consider that the Commission induced the complainant to remain physically close to the place where the re-evaluation procedure took place, or, in the event the redress request proved successful, the place where the continuation of the proposal would take place. There is moreover no information to suggest that the nature of the redress procedure, or the way in which that procedure was presented to the applicants in general, could reasonably lead an applicant to entertain the expectation that such expenses would, in case of delay, be compensated. In light of the foregoing, the Ombudsman finds that the claim cannot be sustained.

B. Alleged inadequate reply to the redress request

Arguments presented to the Ombudsman

16. The complainant alleged that the outcome of the redress procedure failed to respond to the specific arguments he had made in his appeal. The specific arguments were, in summary, the following:

(a) Reviewer 3 erred by applying a criterion in relation to the experience as team leaders already gained by candidates; and

(b) Reviewer 3 applied stricter standards concerning the "quality of the proposed research project" and its "methodology" than those provided for in the Call and the ERC Guide for applicants, in particular, by making reference to the complainant's project not being sufficiently "precise" and "well-defined", despite the fact that applicants were only required to provide a "brief description" of scientific and technical aspects of the project proposal.

17. Specifically in the appeal, the complainant first stated that Reviewer 3 contradicted the principles of the Call, as set out in the ERC Guide for applicants. The complainant noted that Reviewer 3's comments concern (1) the "potential of the Principal Investigator" to be a team leader and (2) the "quality of the proposed research project". The complainant made remarks on these two aspects.

First aspect:

18. Concerning the "potential of the Principal Investigator" to be a team leader, Reviewer 3 stated that "[i]t is not clear that at this stage, he acquired all the needed skills to be a leader in a demanding project". The complainant noted that the ERC Guide for applicants stated that the "ERC Starting Independent Research Grant Scheme aims to provide adequate support to researchers at the stage at which they are intending to establish … their first independent research team". The complainant also noted that the ERC Guide for applicants stated that "researchers applying for an ERC Starting Grant must be able to demonstrate their potential to become independent research leaders". The complainant concluded, therefore, that the applicant was not required to have already acquired all the needed skills to be a leader in order to be eligible. The complainant also noted that the ERC Guide for applicants stated that, in order to be eligible for a grant, the "Principal Investigator must be at the stage where he/she is establishing independence (i.e. starting … their first research team …)".

19. The complainant also noted that the ERC Advanced Investigator Grant Scheme is aimed at providing support to "research projects carried out by leading investigators" and it "complements the ERC Starting Grant Scheme by targeting researchers who have already established themselves as being independent research leaders …" The complainant concluded that an applicant who has already acquired all the needed skills to be a leader (as stated in the report of Reviewer 3) should apply for the ERC Advanced Investigator Grant Scheme rather than the ERC Starting Grant Scheme.

20. In sum, the complainant argued that Reviewer 3 erred by applying a criterion relating to the experience as team leaders already gained by candidates, and that the appeal body erred by not providing a specific explanation in relation to this argument, which the complainant put forward in his appeal.

21. Concerning the "quality of the proposed research project", the complainant noted that Reviewer 3 stated that "[t]he proposal is interesting but it lacks showing in a convincing way a precise, well defined and original project. The relation between self-energies and more generally vertex corrections with cross-sections is certainly known. Relations with the proposed experimental quantities especially with systems having non-trivial orders like a nodal liquid in High-Tc superconductors is not obvious". (emphasis added)

22. Furthermore, the ERC Guide for applicants stated that one of the essential components for the application was a "brief description of scientific and technical aspects of the project proposal". The Guide for applicants also outlined that "the information provided should be sufficiently comprehensive". The complainant concluded that a sufficiently comprehensive brief description cannot be precise and well-defined at the same time. The complainant thus argued that, by referring to his project as not being sufficiently precise and well-defined, Reviewer 3 did not respect the conditions of the Call.

23. The complainant also noted that the ERC Guide for applicants imposed strict size, layout and format requirements for the applications. Without the opportunity to write formulas and attach plots, drawings and graphics in order to fulfil the stringent requirements, there was no opportunity for an in-depth discussion of the "relations" mentioned in Reviewer 3's report.

24. The complainant stressed that the ERC Guide for applicants allocated – for the "description of objectives and scientific and technical content of the project" – four pages at stage 1 and ten pages at stage 2. The complainant concluded that applicants were required to show "in a convincing way a precise, well defined and original project" and discuss in depth the "relations" mentioned in the report by Reviewer 3. This had to occur at stage 2 of the proposal submission, not at stage 1. The complainant also pointed out that the ERC Guide for applicants stated the following: "stage 2: describe the proposed methodology in detail … including any particular novel or unconventional aspects". The complainant concluded that the "not obvious relations" mentioned by Reviewer 3 must/should have been discussed at stage 2 instead of at stage 1.

25. The complainant also emphasised that the evaluation criterion of "methodology" at stage 1 required that the "outlined scientific approach ha[d] to be feasible", while at stage 2 "the proposed research methodology ha[d] to be comprehensive and appropriate". The complainant concluded that this demonstrated that the need to "show[ing] in a convincing way a precise, well defined and original project" together with a deep discussion of the "relations", which was mentioned in the report of Reviewer 3, was required at stage 2.

26. Finally, the complainant noted that, of the four reviews he received, only one was unfavourable. The ERC Guide for applicants stated that the overall scoring of the proposal would be based on the combined results of all reviewers. However, the complainant alleged that Reviewer 3's report overwhelmed the reports of the other reviewers.

27. In its opinion, the Commission provided the following background information, which is important for the examination and understanding of the case.

28. The complainant's proposal was submitted in response to the very first Call for proposals of the Specific Programme 'IDEAS' of the 7th Framework Programme of the European Union. In the framework of the Call for proposals (ERC-2007-StG), covering 'European Research Council (ERC) Starting Grants (StG)', a total of 9167 proposals were submitted. The ERC originally expected an upper working estimate of 3000 proposals. The Commission explained that ERC proposals are submitted by a principal investigator, with the support of a host organisation, which committed itself to engaging the principal investigator if the proposal is successful. The complainant was one such principal investigator.

29. Despite the overload of work resulting from the large number of proposals, the evaluation of proposals was completed on time. However, some procedures experienced a delay. This was, in particular, true of the redress procedure in which the complainant took part.

30. The evaluation process was governed by the 'ERC Rules for the submission of proposals and the related evaluation, selection and award procedures relevant to the IDEAS Specific Programme' (hereinafter 'the Rules').

31. Of the 9167 proposals, 559 applicants were invited to prepare a second stage proposal and submit it by 17 September 2007. A total of 8235 proposals were rejected on the recommendation of the evaluation experts. The remainder were rejected on eligibility grounds. The applicants were informed by the ERC by letter.

32. In the ERC evaluation process, panels of high-level scientists are responsible for the final recommendations on each proposal. They base their position on written assessments provided by independent reviewers. A specific feature of ERC evaluation reports is that they contain the detailed comments of each individual reviewer. This choice was made in the interests of transparency and in recognition of the fact that, in the assessments, differences of opinion are legitimate. The Commission takes the view that this detailed information is useful for the applicants. It is, however, important to recall that the ERC evaluation panels make their recommendations collectively.

33. The 8235 proposals that were rejected at the first stage (including proposals considered to be ineligible) gave rise to 245 valid requests for redress (3%). The complainant's request feel within this category. The redress procedure was carried out in accordance with the Rules, and involved a committee of six officials, chaired by an acting Director of the Commission's Directorate-General for Research. Each redress request was subject to an individual and in-depth examination of its merits.

34. According to the Rules, the redress procedure examines the evaluation process and its possible flaws. The redress procedure does not examine issues of scientific judgment or differences of opinion between reviewers. Consequently, for the 150 applicants seeking redress because they disagreed with the scientific judgment of the individual reviewers or the panel, the Redress Committee merely verified whether the reviewer in question possessed the appropriate scientific expertise, and if so, turned the redress request down. The complainant's redress request was amongst these 150 cases.

35. In contrast, the reviewers and panels carried out further examinations and even re-evaluated cases where there were proven or credible procedural errors. Examples of such errors include where reviewers confused proposals or made observations that were clearly not linked to the proposal examined. As a result of this approach, fifteen proposals were re-evaluated and one applicant was invited to submit a proposal for the second stage of the evaluation.

36. With regard to the specific allegations in the present case, the Commission stated that it disagreed with the complainant's view that the redress procedure failed to take proper account of his specific arguments. It made the following points.

37. Following a thorough investigation, the Redress Committee concluded that the judgments made by Reviewer 3 were "legitimate from the scientific point of view" and that Reviewer 3 did not make any factual or procedural errors. The Committee confirmed that the evaluation had been correctly conducted according to the rules. The Commission found that the Committee had therefore acted in line with its remit.

38. The Commission added the following points.

39. It is not true, as the complainant argues, that the evaluation result was "overwhelmed" (that is to say, primarily determined) by the views of Reviewer 3. The comments from the other reviewers were more favourable, but only moderately so. The proposal did not receive sufficient overall support to be amongst the 559 selected out of the 9167 proposals submitted.

40. Page limitations apply equally to all applicants. They are a normal part of the process[1].

41. Reviewer 3 did not, when applying the criteria here concerned, make any procedural or factual errors by stating the following: "[i]t is not clear that at this stage, he [that is, the complainant] acquired all the skills to be a leader in a demanding project". While it is true that applicants for the ERC 'Starting Grants' only have two to nine years of post-PhD research experience, they nevertheless clearly need to possess the qualities for leading a Starting Grant Project at the forefront of science. In this context, the comments of Reviewer 3 correctly addressed the evaluation criteria and reflected the use of scientific judgment.

42. In his observations, the complainant maintained his allegation. He emphasised that he never intended to appeal against the scientific judgment of the panel of reviewers, but rather wished to address what he saw to be a procedural error. He stated that the Commission's opinion contained some useful information, but he still considered that the decision on his appeal could have been more detailed.

The Ombudsman's assessment

43. Before examining the specific facts and arguments in this case, the Ombudsman considers it appropriate and useful to address some of the historical or general points which the Commission made in its opinion. This will help to understand better the context, and to provide a useful background for future examinations of the ERC's procedures for selecting research proposals. This appears, moreover, to be in line with one of the complainant's wishes, which is to help improve the ERC's procedures[2].

44. In the first place, the Ombudsman wishes to recognise the fact that the ERC and the Commission dealt with a very large number of applications submitted in response to the call here in question. According to the Commission's opinion, the number of applications received (more than nine thousand) was three times higher than the actual number expected. It is most satisfying to note that this amount of work was handled in an overall appropriate and timely manner. More generally, the Ombudsman considers that the high number of applications expresses a significant level of trust in the EU's procedures for handling such applications. Various positive aspects of the procedures, two of which are highlighted below, certainly help to reinforce such trust and the EU's ability to promote good research. In fact, the positive features highlighted here below appear to go hand in hand with the current aim of DG Research to improve its relationship with researchers[3].

45. In particular, the Ombudsman notes that the procedures here in question enhance the standards of transparency in one important way. As the Commission itself pointed out in its opinion, a specific feature of ERC evaluations is that they contain a detailed summary of each individual reviewer, and that this choice that was made in the interest of transparency and in recognition of the fact that differences of opinion are legitimate. The Ombudsman notes that the Commission does not appear to limit this reasoning to scientific evaluations, and there are indeed no reasons why that should be the case. Accordingly, the ERC evaluations may be considered to set an important new standard of transparency for the evaluation of EU calls for proposals. The Ombudsman applauds this development.

46. Given the novelty of the above-mentioned transparency feature, it is not surprising that, with the benefit of hindsight, it is possible to identify certain additional measures that could usefully have been introduced to accompany this positive development. For instance, it could have been useful to issue more precise instructions to the reviewers; and it could have been helpful to include an explanatory note to the applicants regarding the nature and relative importance of the individual reviewers' opinions. The examination of the present case further below serves to illustrate these points.

47. In addition to the above, the Ombudsman notes that, in order to deal with the very considerable number of redress requests, the Commission adopted at an early stage a coherent methodology for handling such requests. Specifically, the Redress Committee verified whether the initial reviewers who assessed the proposal in question possessed the appropriate scientific expertise, and, if that was the case, it turned the redress request down (including that of the complainant). Such a clear and coherent approach is in principle good administration because it enhances the transparency of the procedure and reduces the risk of arbitrariness. However, more detailed instructions could usefully have accompanied this positive aspect of the handling of requests. The examination of the present case further below also serves to illustrate this fact.

48. Turning to the facts of the present case, the Ombudsman recalls that the complainant specifically considered that the Redress Committee failed properly to address his arguments that Reviewer 3 erred by applying a criterion in relation to the experience already gained by candidates as a team leader.

49. The Commission rightly pointed out, in its opinion, that the Redress Committee could not (re)-assess the scientific opinions of the panels and the individual reviewers. The Ombudsman notes that this type of limitation is entirely common in appeal and review contexts, and is not in any way specific to applications concerning scientific issues. In other contexts – for instance, the recruitment or the evaluation of other types of applications – very similar 'standards of review' are applied, so as to respect the 'wide discretionary powers of' the initial assessors in question. In such contexts, ensuring full compliance with procedural rules is, for obvious reasons, of particular importance. If the substance cannot meaningfully be reviewed as such, the control organ (be it administrative or external, as in the case of an ombudsman or a court) is implicitly called upon to pay particularly close attention to the procedural aspects of the case.

50. In the present case, the complainant argued that, as regards the "potential of the Principal Investigator" to be a team leader, Reviewer 3 stated that "[i]t is not clear that at this stage, he acquired all the needed skills to be a leader in a demanding project". The complainant noted that the ERC Guide for applicants stated that the "ERC Starting Independent Research Grant Scheme aims to provide adequate support to researchers at the stage at which they are intending to establish … their first independent research team". The complainant also noted that the ERC Guide for applicants stated that "researchers applying for an ERC Starting Grant must be able to demonstrate their potential to become independent research leaders". The complainant therefore concluded that, in order to be eligible, the applicant was not required to have already acquired the needed skills to be a leader. The complainant also noted that the ERC Guide for applicants stated that, in order to be eligible for a grant, the "Principal Investigator must be at the stage where he/she is establishing independence (i.e. starting … their first research team …)".

51. The complainant also noted that the ERC Advanced Investigator Grant Scheme is aimed at providing support to "research projects carried out by leading investigators" and it "complements the ERC Starting Grant Scheme by targeting researchers who have already established themselves as being independent research leaders …" The complainant concluded that an applicant who has already acquired all the needed skills to be a leader (as stated in the report of Reviewer 3) should apply for the ERC Advanced Investigator Grant Scheme rather than the ERC Starting Grant Scheme.

52. The Ombudsman considers that, quite apart from the issue of whether the complainant's above arguments were accurate or not, they were sufficiently clear and reasonable to merit a specific and concrete response. Specifically, the complainant convincingly illustrated that he was addressing a procedural issue, namely, one that concerned the very applicability of a condition. The Redress Committee merely replied, however, that:

- "[a]s regards the comments presented by Reviewer 3, the redress committee states that there has been no factual error and an individual review differing from other reviews on the same proposal cannot be automatically qualified as unfair"; and

- "Reviewer 3 shows a judgment that is legitimate from a scientific point of view."

53. In neither of these statements did the Redress Committee specifically address the complainant's above-mentioned grievance. The first statement, if at all relevant, could only be understood to imply that it was a 'fact' that the complainant did not (already) possess the skills to lead a demanding project. And it contained no observations on the issue of whether such (existing) skills were formally required. The second statement, if at all relevant, suggests that the issue of the complainant's (existing) skills to lead a demanding project was a question of (natural) science. This statement as well made no observations on the issue of whether such capacity was formally required.

54. Neither of these statements were sufficiently pertinent to, or adequately addressed, the above-mentioned and apparently valid point raised by the complainant. Hence, the Redress Committee cannot be said to have properly dealt with that point.

55. Moreover, the complainant's arguments appear to have considerable substantive merit. The results of the evaluation support such an assessment. Out of four reviewers, Reviewer 3 was indeed the only one who referred to a presumed lack of "acquired" "skills" to be "a leader in a demanding project". Reviewer 2 did not make any comments on this at all , and Reviewer 1 and Reviewer 4 referred to the complainant's good publication record and concluded that he "shows significant potential". As the complainant argued, the nature of the call here in question did not appear to require "acquired skills" for leading a demanding project. "Skills" are something very concrete. It goes beyond a notion of "potential" or "capacity", and essentially suggests that the person concerned has already demonstrated a concrete ability to do a specific job.

56. The Commission's useful and frank opinion in the present inquiry contains indications as to why the above-mentioned oversight or error occurred. As previously noted, the large number of redress requests led the Commission to adopt a method whereby any kind of disagreement with the panels and their reviewers would be classified as disagreement on the 'scientific judgment' or 'facts', the latter apparently implying, primarily, gross errors such as evaluating the wrong application. This approach appears to have been overly pragmatic for the purposes of efficiency, by failing properly to take into account errors of procedure that are normally examined in such procedures, for instance, the possibly erroneous application of criteria.

57. The Commission pointed out that, in spite of the comments made by one reviewer, the final assessment was made on a collective basis. This is, formally speaking, accurate. However, it goes without saying that a panel should not base itself on assessments by reviewers who appear to have applied the wrong conditions in their assessments or committed other procedural errors. This is what the Redress Committee ought to have investigated in the present case. The concrete outcome of such an investigation should have been referred to in its decision on the complainant's redress request.

58. The second part of the complainant's second allegation stated that Reviewer 3 applied stricter standards concerning the "quality of the proposed research project" and its "methodology" than those provided for in the Call and the ERC Guide for applicants. In the complainant's view, Reviewer 3 did so, in particular when referring to the complainant's project as not being sufficiently "precise" and "well-defined" despite the fact that applicants were only required to provide a "brief description" of the scientific and technical aspects of the project proposal.

59. The Ombudsman's findings on the first part of the allegation also apply to this part as well. To present a "well-defined" project using a 'brief description' does, in fact, appear to be a contradiction in terms. The Redress Committee should have noted this, and identified it as a grievance concerning a procedural error (which must be addressed) and not a challenge to 'scientific judgment' or incorrect facts.

60. In light of the above findings on both parts of the allegation, the Ombudsman concludes that the Redress Committee wrongly refrained from investigating and addressing the issue regarding the incorrect application of criteria, or the application of irrelevant criteria. This was an instance of maladministration. The Ombudsman will make a corresponding critical remark below. The critical remark is followed by a related further remark.

61. With regard to the possible consequences of the Ombudsman's above findings, it is recalled that, as emphasised by the Commission, the conclusions on the complainant's application were made collectively by the relevant panel. Formally speaking, Reviewer 3 did not decide on the outcome of the process as such. Within the framework of the rules and the information available to the Ombudsman at this point, it is therefore not possible to conclude that the collective decision of the panel was manifestly tainted by a procedural error requiring the ERC to reassess the application here in question.

C. Conclusions

On the basis of his inquiry into this complaint, the Ombudsman closes it with the following critical remark:

The Redress Committee wrongly refrained from addressing, when answering the complainant's appeal, the issue regarding the incorrect application of criteria, or the application of irrelevant criteria. This was an instance of maladministration.

The complainant and the European Commission will be informed of this decision.

Further remark

The Ombudsman notes and applauds the Commission's success in handling a very large number of research applications in an overall appropriate and timely manner. He also notes that, by granting applicants access to the individual assessments of the independent reviewers, the procedures set an important new standard of transparency for EU calls for proposals.

With regard to his findings on the second allegation in the present case, the Ombudsman would be grateful to receive information on the instructions or guidelines now provided to independent reviewers and the Redress Committee on how to carry out their respective assessments and reviews of applications.

 

P. Nikiforos Diamandouros

Done in Strasbourg on 16 December 2010

---

[1] This remark refers to the complainant's points referred to in paragraph 25 above.

[2] E-mail of 19 March 2008 from the complainant to the Ombudsman.

[3] http://europa.eu/rapid/pressReleasesAction.do?reference=SPEECH/10/194&format=HTML&aged=0&language=EN&guiLanguage=en

http://ec.europa.eu/research/fp7/pdf/communication_on_simplification_2010_en.pdf