Learning in Development. Olivier Serrat

Чтение книги онлайн.

СКАЧАТЬ begin on a joint country assistance program evaluation in Bangladesh in partnership with the World Bank, the Department for International Development of the United Kingdom, and the Japan Bank for International Cooperation. This evaluation is expected to be finished in 2009.

      Validating Country Partnership Strategy Completion Reports. ADB is beginning to produce country partnership strategy completion reports. OED will pilot a new evaluation product in 2007 in the Maldives, the country strategy completion report validation, primarily for countries with small portfolios for which production of a full country assistance program evaluation would not be an efficient use of resources.

      Validating Project/Program Completion Reports. OED will change the way that individual projects/programs are selected for evaluation. The strategic objective is for the quality of completion reports to be sufficient to rely on this self-evaluation. Efforts to improve the quality of completion reports appear to have removed the upward bias of ratings at completion. The changes include the following: (i) moving away from the 25% random selection of projects and programs for which evaluation reports are prepared to a smaller, purposeful sample of 10 per year; (ii) OED would validate, based on a desk review, all ratings (with the validation assessment to be attached to the completion report) rather than commenting on draft completion reports—much of this work will be outsourced using funds that were previously spent on staff consultants for preparation of evaluation reports; and (iii) reporting success based on the combined completion and evaluation ratings as has been done in the more recent issues of Annual Evaluation Reviews. The projects and programs for evaluation will not be randomly selected. Selection triggers will include dispute over a rating, by OED or external stakeholders, and special interest.

      Evaluating Impact. OED agrees with the general conclusions of the debate in the international evaluation community (IEC) about impact evaluations: (i) more rigorous impact evaluations are desirable; (ii) the methodology will be determined by issues related to data availability, time, and resources; and (iii) impact evaluations will be undertaken selectively, largely in the social sectors. OED is undertaking its first rigorous impact evaluation on microcredit in the Philippines as part of an evaluation study on the effectiveness of ADB’s microcredit operations. This will be completed in 2007. One impact evaluation is programmed per year.

      Developing Evaluation Capacity. Evaluation capacity development is part of OED’s mandate. As of January 2007, OED had formulated 15 TA operations to this intent for a total amount of $4.35 million in Bangladesh, the People’s Republic of China, Nepal, Papua New Guinea, Philippines, Sri Lanka, and Thailand. Three phases can be distinguished in this assistance:

      • Phase 1: TA focused on building a postevaluation capacity within a central agency and providing the means for disseminating postevaluation findings for decision making.

      • Phase 2: TA aimed at establishing ADB’s project performance management system in central and sector agencies.

      • Phase 3: TA aimed at building more generic results in monitoring and evaluation capability.

      In the future, OED expects to work with evaluation units in DMCs to provide on-the-job evaluation experience and knowledge transfer, building on lessons learned from the evaluation of the 15 TA operations, not all of which were successful.

      Promoting Portfolio Performance. OED began to provide real-time feedback on portfolio performance in 2001. The 2005 Annual Report on Loan and Technical Assistance Portfolio Performance highlighted serious and fundamental corporate issues. At the DEC’s recommendation, ADB’s Management prepared an action plan to address these issues.

      Evaluating Business Processes. In connection with ADB’s reorganization of 2002, a working group on business process change was appointed in 2001 to review country strategies and programs and subregional cooperation strategies and programs; public sector loans (including project preparatory TA operations), private sector loans, and nonlending products and services (including advisory TA operations); and portfolio management. In addition to its reporting on portfolio performance, OED has included business process-related evaluation studies in its work program. Forthcoming evaluation studies will examine, for instance, the effectiveness of ADB’s loan processing system, its approaches to policy dialogue and reforms, and the quality of the design and monitoring framework.

       Box 13: Developing Evaluation Capacity in Developing Member Countries

      • Stability of trained staff, high-level support, and the existence of a mandate for evaluation by decree are factors that contribute to success.

      • More thorough preparation of future TA operations should ensure high-level ownership and commitment, and participation of key stakeholders in formulation and design.

      • If the conditions for public sector capacity building are not met, an assessment must determine whether the systemic or underlying problems should be addressed first.

      • Building DMC capacity requires a holistic approach, considering the needs at all levels.

      • The location of responsibility for evaluation within organizational hierarchies is also important.

      • During design and implementation of TA operations, care must be taken that performance evaluation systems do not become supply driven, complex, or too resource intensive to sustain.

      • Establishing performance evaluation systems is a means to an end—benefits are obtained when the results are used in decision making. The design of TA should include specific features to encourage, facilitate, and formalize the incorporation of evaluation results in decision making.

      • A case study approach is needed to develop staff competency and confidence to carry out evaluation.

      • For larger TA operations, a firm or institution should be recruited, rather than individuals.

      • The pace of TA should be driven by a sense of ownership and commitment in DMCs.

      • The introduction of computerized information systems is not a solution to poorly performing manual systems. Various institutional, management, and social factors need to be taken into account.

       Box 14: Perceptions of OED: Feedback from Members of the Board of Directors

      Interviews with Board members revealed the general perception of the mission and functions of OED as to provide independent assessment with a direct link to operations. OED is seen as collegial, dedicated, and professional. While OED has generally changed with the changing focus of ADB, there is an inevitable lag as evaluation activities adjust to new organizational thrusts.

      OED has been able to influence ADB’s operations at all levels by providing concrete recommendations based on solid and credible analysis. At the project/program level, the time lag between completion and evaluation is an issue, as evaluation findings can easily be dismissed as discussing the old way of doing things, while current practices may have changed. At the strategy and policy levels, the improved timing of country and sector assistance program evaluations has increased impact on the design of new country partnership strategies.

      In its knowledge management, OED faces several interface problems. Within ADB, OED should open up channels of communications, become even more specific about actionable recommendations, and delineate accountabilities clearly. The most difficult interface is with DMCs: OED should emphasize development of evaluation capacity. In the eyes of wider clienteles, such as NGOs, civil society organizations, and the general public, OED should not only be independent, but be perceived as such. It should produce concise and insightful summaries of its СКАЧАТЬ