Malawian experiences with Paris Declaration
This report presents the results of the Phase 2 Evaluation of the implementation of the Paris Declaration in Malawi. This is consistent with the comprehensive evaluation strategy developed by OECD-DAC to understand the Declaration’s achievements for aid effectiveness. The evaluation assesses the relevance and effectiveness of the Paris Declaration and its contribution to aid effectiveness and ultimately development effectiveness, including poverty reduction in Malawi.
The Paris Declaration (PD) on Aid Effectiveness is a joint international statement aimed to improve the quality of aid and its impact on development. Endorsed in 2005, the PD currently has over 120 countries and 30 international organisations as signatories, thereby committing themselves to the five norms of the Declaration, including ownership, harmonisation, alignment, managing for results and mutual accountability. The Declaration provides a practical roadmap for improving aid effectiveness, with an agreed set of monitorable actions and indicator targets to be met by 2010. Malawi signed the Declaration in 2005 and fully embraced and utilised the Declaration to strengthen its relationships with the development partners.
In order to understand what the Declaration has achieved for aid effectiveness, the OECD-DAC developed a comprehensive evaluation strategy. Phase 1 of the strategy focused on inputs, outputs and ways of improving the implementation of the Declaration. Eight countries participated in Country Evaluations and eleven Donor and Agencies conducted Headquarter studies. Phase 1 findings were presented at the High Level Forum (HLF) on Aid Effectiveness in Accra in September 2008. Phase 2 of the evaluation began in September 2009 and is being undertaken by a multinational team from International Organisation Development (IOD) Ltd., with focus on intermediate outcomes and development results. The findings of the evaluation will be presented at the HLF on Aid Effectiveness to be held in Seoul in 2011.
2.2 The Purpose and Scope of the Second Phase of PD Evaluation
This report presents the results of the second independent evaluation of the Implementation of the Paris Declaration in Malawi. The purpose of the Phase 2 evaluation is to assess the relevance and effectiveness of the Paris Declaration and its contribution to aid effectiveness and ultimately development effectiveness, including poverty reduction in Malawi. The evaluation design acknowledges the importance of country-specific differences including differences in policy, history, and resources both among donors and aid recipients. The evaluation is therefore expected to analyse results in context, taking into account preconditions or enabling conditions that may lead to or inhibit positive development results supported by aid. According to the Terms of Reference (ToRs), the specific objectives for the Malawi Evaluation included:
- To document the results achieved by implementing the Paris Declaration in Malawi;
- To enable country-based "partnerships, partner countries and donor/agencies to clarify, improve and strengthen policies and practices with the Paris Declaration in the pursuit of aid effectiveness and development effectiveness;
- To highlight barriers and difficulties that may limit effectiveness of the Paris Declaration and the effects and impacts - and how these barriers and difficulties may be overcome;
- To strengthen the knowledge base as to the ways in which development partnerships can most effectively and efficiently help maximise development results through aid in different development contexts - including various degrees of "fragility"; and
- To enable sharing and exchange of experience amongst stakeholders, countries and partnerships to facilitate reflection, lesson -learning and policy improvement.
2.3 Approach, Methodology and Limitation
2.3.1 Conceptual and Analytical Framework
The evaluation is a participatory qualitative assessment of three main aspects: a) the contextual relevance of the PD in Malawi and factors that positively or negatively affected the process of the implementation of the PD principles; b) the impact of the PD on efficiency of aid management, and c) the contribution of the PD to the achievement of sustainable development results.
The main focus of evidence gathering is on the behaviour changes within government and among development partners that have occurred (or are supposed to occur) and are expected to underpin subsequent improvements in aid effectiveness and the delivery of outcomes and impacts for Malawi. To break down the issues to be analysed and hypotheses to be tested into a series of questions to be asked when collecting information, the evaluation followed the specific questions in the ToRs and the Draft Operational Matrix for Country Evaluation provided by the Core Evaluation Team to guide country case studies during the Second Phase Evaluation of the Implementation of the Paris Declaration (Annex 3).
The qualitative assessment concerns the relevance of the operating environment of the PD in Malawi, the nature of behaviour change that has occurred among development partners and the development results that have been enabled or supported by this behavioural change. Generic approaches to qualitative data analysis was adopted in line with the qualitative nature of this evaluation. The generic approach provided ways of discerning, examining, comparing and contrasting as well as interpreting meaningful patterns or themes. The analysis was organised in three steps: (i) data reduction; (b) data display and (c) conclusion drawing and verification. The evaluation used multiple descriptors and sources of evidence to allow sufficient rigor and triangulation of emerging issues before drawing conclusions and recommendations. Throughout the course of qualitative analysis, the following questions were continuously asked by the Evaluators:
- What patterns and common themes emerge in responses dealing with specific items? How do these patterns (or lack thereof) help to illuminate the broader study question(s)?
- Are there any deviations from these patterns? If yes, are there any factors that might explain these atypical responses?
- What interesting trends/stories emerge from the responses? How can these stories help to illuminate the broader study question(s)?
- Do any of these patterns or findings suggest that additional data may need to be collected? Do any of the study questions need to be revised? and
- Do the patterns that emerge corroborate the findings of any corresponding qualitative analyses that have been conducted before? If not, what might explain these discrepancies in the context of Malawi?
A purposive sampling frame was adopted to ensure representation of all stakeholders in the sample. A sample size of 60 was adopted from government ministries and departments, development partners/donors, Non-Governmental Organisations (NGOs) and academic institutions. Because Malawi does not have many donors, an attempt was made to interview all donors.
The evaluators planned to carry out detailed analysis within two sectors of Health and Roads (Transport and Public Infrastructure). The two sectors chosen for the Malawi study were to provide a cross-sectional view of what is happening or has happened in sectors with a varied mix of donors and financing modalities. The sectors were chosen for case studies rather than as a representative sample of sectors in Malawi. While health has been the first sector to adopt a Sector Wide Approach (SWAp) arrangement, the choice of the road transport sector was in view of the substantive resources that are being channelled to the sector by "new and non-traditional donors. Consistently, donors working in these two sectors and the relevant arms of government executing programmes in these sectors were automatically selected for key informant interviews. Since a small number of donors operate in Malawi, all donors were interviewed. The United Nations Agencies were also included as they have made efforts towards achieving principles similar to the PD as part of the UN reform agenda. Given the tight timetable and likelihood of unavailability of some key informants within the donor group, the sampling frame remained wide to ensure that as many donors were available were included for interviews. A small sample of non-state sectors, mainly umbrella bodies of NGOs (CONGOMA) and the private sector (sampled through the Malawi Chamber of Commerce) was included to solicit views and perceptions of civil society and private sector in the broader sense of the PD itself and the associated behaviour changes in government or the development partners. Thus, the comprehensive list of interviewees in Annex 4 allowed the evaluators to broaden the spectrum of views and perceptions.
The sample included such donors as the African Development Bank (AfDB), World Bank, European Commission, Norwegian Aid, Department of International Development (DfID), Flemish International Cooperation Agency (FICA), Germany Development Corporation (GDC), Iceland International Development Assistance (ICEIDA), Irish Aid, Japan International Cooperation Agency (JICA), United States Agency for International Development (USAID), United nations Development Programme (UNDP), Food Agriculture Organisation (FAO) and the International Monetary Fund (IMF). Further, the sample included such government ministries and departments as Ministry of Finance, Ministry of Development Planning and Cooperation, Ministry of Agriculture and Food Security, Ministry of Health Ministry of Irrigation and Water Development, Ministry of Education, Science and Technology, Ministry of Transport and Public Infrastructure, Office of the Director of Public Procurement, Ministry of Gender, Women and UNDP, UNICEF, UNFPA, WFP, WHO, FAO, UNHCR, and UNAIDS Child Development, Ministry of Energy and Mines, Anti-Corruption Bureau, Roads Authority, Road Fund Administration, Accountant General and Auditor General. Finally, NGOs and private sector organisations were also included in the sample such as the Malawi Economic Justice Network (MEJN), World Vision International, Action Aid, Economic Association of Malawi (ECAMA), Malawi AIDS Network (MANET), Malawi Health Equity Network, CISANET, Gender Network, Centre for Human Rights and Rehabilitation, and Plan International. A full list of the institution and people interviewed is presented in Annex 4.
2.3.3 Data Collection
The evaluation used both qualitative and quantitative data collection procedures, including review of existing literature and secondary data (desk research), face-to-face semi-structured interviews as well as self and administered structured questionnaires.
Documentation reviews were undertaken to assist in the analysis of the Malawian context in relation to the implementation of the PD principles. The primary source of the literature review was the MoF, which provided the initial data on aid disbursement, public finance statistics, reports from government and donors and sector programmes.
Key Informant face-to-face interviews were undertaken with major stakeholders, which included developing partners, government ministries and departments, civil society, the media and private sector.
Finally, in order to carry out the data collection exercise in a timely and more focused manner, a number of structured questionnaires and interview guides were developed. In addition, self-administered questionnaires were e-mailed to about 20 organisations. Two interview guides were used in interviewing government and development partners on one hand, and the civil society, private sector and the press on the other. These tools are presented in Annex 3.
The main challenge faced during the data collection phase is the slow response from respondents and the tendency by stakeholders to schedule appointments in their own time. The data collection exercise was undertaken during September when most of the officials of the development partner agencies were away on annual summer vacation.
A number of self-administered questionnaires were not answered. Out of twenty questionnaires that were sent out only six were returned. The institutions that did not send back the structured questionnaires were followed up with telephone and face-to-face interviews. However, only 10 out of the 14 that did not return the questionnaires were interviewed. Nonetheless, this does not significantly affect the quality of data.
National and sector statistics presented challenges in terms of data availability and consistence over pre- and post-PD period. Finally, there is evidence of other developments apart from the PD that can explain development results and intermediate outcomes in Malawi. Thus, the assessment of development results is limited to possible linkages and contributions of the PD without following the route of attribution.
- Extracted from the report: A Country Evaluation of the Paris Declaration for Malawi, dated February 2011. Access the 111 page report here.