Sunday, August 31, 2014

How can UNESCO show results? By reporting them

UNESCO’s future depends on the willingness of Member States to support it. This willingness depends on UNESCO’s capacity to show Member States that it is useful and that it produces results. In an excellent evaluation report UNESCO evaluated whether its results-reports actually do provide (good) information about results. This blog is a summary of its findings (for everyone), followed by an Annex containing quotes from the report (for diplomats and experts who will discuss this next Tuesday).

[NB. Sometimes I had to simplify the conclusions of the report a little to make them easily understandable, but without altering the conclusions. I also added a personal conclusion about the meaning of this rather critical report.]

Problem: little to no information about the impact of UNESCO's work
Results are effects that UNESCO’s work has on reality. The main conclusion of the report is that UNESCO’s results-reports show very little information about these effects. Almost two thirds of the report describe UNESCO’s own processes and almost one third is about the “things” that are produced by these processes (meetings, workshops, policy guidelines, etc.). However what Member States need to convince their governments of UNESCO's usefulness is much less present in the report:
  • only one fifth is about effects that UNESCO's meetings etc. are supposed to produce, i.e. changes in the behavior of people and institutions who can change society; and
  • the report tells nothing about whether society actually did change thanks in part to UNESCO.
In this respect UNESCO’s results-reports are very different from the ones in other international organizations, as is shown in the picture below. This is a problem because UNESCO cannot be rewarded for impact it doesn’t report on.
UNESCO only reports about the work it produces while other international organizations report about the effects their work produce.

It would be easier to convince Member States of UNESCO's usefulness if its results-reports made a clear distinction between UNESCO's work (meetings, workshops, reports) and the positive effects of this work (better heritage protection, better education). To achieve this UNESCO could present two different types of results-reports that focus on different things:
  • An implementation report that only informs Member States if UNESCO’s work is done correctly: does UNESCO do the things right? Were activity targets met for example? Were the meetings, policy guidelines, global reports etc. produced in time, in the right quantity and within the given budget and quality requirements? This report would not look for causal effects, it would only verifiy whether the UNESCO work is done as planned.
  • A results-report that only informs Member States if UNESCO's work has positive effects on society: does UNESCO do the right things? For example: did the meetings and policy guidelines actually motivate countries to improve their education policies, practices and capacities? Did press freedom become more prominent on the international political agenda and did journalists and democracies benefit from this? This report would not provide a catalogue of UNESCO’s meetings and policy guidelines but focus exclusively on their (positive) effects on the world outside UNESCO. 
Does this study mean that UNESCO is useless?
Although the report concludes that UNESCO provides very little information about the effects of its work, this does not mean that UNESCO’s work doesn’t have effects! It only means that these effects are not made visible enough, that’s all. The list of examples of UNESCO's positive effects is too long to describe here. Let’s start with the protection of cultural heritage.

What are UNESCO’s effects in the field of culture?
Without UNESCO the awareness of and subsequent care for world heritage would simply not have existed. UNESCO literally created the “world heritage sentiment” in the late Fifties when it saved the Nubian monuments in Egypt (like the Temple of Pharaoh Ramses II) when they were about to be flooded due to the construction of the Nasser Dam. The effect of interventions like these is first of all that everyone in the world now are aware of the fact that they have some irreplaceable treasures in common. This awareness leads subsequently to better heritage protection policy and practices thanks to capacity building projects organized by UNESCO. To maintain, nurture and strengthen these effects UNESCO created several Culture Conventions that promote the protection of cultural heritage.

Is UNESCO still producing effects today?
Don’t think that these Conventions from the Fifties and Seventies are old laurels that UNESCO fell asleep on. Quite on the contrary: wherever cultural heritage is in danger UNESCO’s convening power is there to call the world around one large table in Paris. Like during the Mali crisis, where UNESCO called governments and experts together to plan action to assess the damage, to repair it and to raise the necessary funding for it. And UNESCO is not only saving cultural heritage, but also human lives. Thanks to UNESCO, tsunamis will not kill 230,000 people again like the Indian Ocean did in 2004 because UNESCO initiated and coordinated the creation of Tsunami Warning Systems. These are concrete examples of how UNESCO enables the world’s nations to concretely address challenges they would not be able to address on their own. And for a low price: UNESCO’s entire regular budget for one year is less than half of the annual budget of the University of Amsterdam in the Netherlands. 

What’s next?
The challenge this blog addresses is how to make all these results more visible in a report. This is what Member States will discuss next Tuesday 2 September 2014 in the Preparatory Group of UNESCO’s Executive Board. This Group is supposed to propose a new format for UNESCO’s results-reports. As the interested reader will imagine, the evaluation that was written by the Swedish consultant Kim Forss will come in very, very handy.


ANNEX: Quotes and excerpts from UNESCO’s “Formative Evaluation of UNESCO’s Results-Reporting"

·         Challenges:
o   No clear separation of activity and results-reporting: Activity/output-reporting is fundamentally different from reporting on expected results (outcomes). Outcome-reporting requires different data and research: info about target groups of beneficiaries, sample section, gathering of data from target and non-target groups, analysis of the data. UNESCO’s self-reporting on outcomes is too fragmented and weak for that. Consequence: in the C/3 there is no reference to impact at all and only 20% is about outcomes. 30% is about outputs and 60% describes only UNESCO’s own processes. “The [C/3] text describes what the Organization has done in terms of holding meetings. It presents which meetings were held, why and with whom. It does not say what happened after the meetings or whether any results were achieved.”
o   Biased self-reporting: Because of low levels of trust in the Organization self-reporting on results is likely to be biased. Consequence: “there is little suggestion in the C/3 or the EX/4 that anything has gone wrong or might have had negative results”.
o   Inefficient use of results reporting: Statutory reports (E/4 and C/3) are much thicker than in other International Organizations (1000 pages per biennium for UNESCO versus approx. 100 pages for other organizations). “Within the UNESCO results-reporting system, the self-reporting workload (due to the frequency of reporting and the small unit of analysis of reporting) is too high”. Conclusion: “value for money on self-reporting practices in the UNESCO system is rather low” because a lot of time and money is invested in it to produce an extremely large quantity of information “but still fails to answer basic questions about results (at outcome level)”. “The type of information requested by Governing Bodies and stakeholders is rarely produced in the system”.
o   Lack of focus: “Excessively high number of results and performance indicators
o   Lack of causal theory about how results should be attained: “need for a clear articulation of causal assumptions linking activities to results”: “EX/4 does not provide information on (…) analyzing change processes, discussing causality”. The report provides information on outputs but there is no information about impact, about a theory of change and about attribution of contribution to change. “There is little evidence to substantiate the claims to any outcomes and there is no explanation of how activities and outputs lead to outcomes”. “This evaluation did not find examples of outcomes which were substantiated by a theory of change explaining the links between the UNESCO activities and outputs and the hypothetical outcome”. “The question of how results were produced and whether they were caused by UNESCO alone, in conjunction with other agencies, or independently of UNESCO is not discussed in the EX/4 and C/3 reports analyzed here.” “It is not expected to have a detailed theory of change for every type of activity, output and outcome in the EX/4, rather what is needed is a theory of change or a causal framework that connects patterns of activities and outputs to higher level outcomes of interest (i.e. expected results).” What is needed: a complete picture of all levels of the process of change (convention>national policy>projects>effects) and UNESCO’s role in it. Good examples: World Heritage Convention, International Institute of Educational Planning (IIEP).
o   Poor measurement of results against indicators:Evaluations (…) do not systematically feed into the self-reporting processes and consequently into the EX/4 and C/3”. “If three reviews were planned in the C/5 [UNESCO’s programme], but only one was completed, this fact – reflecting poorly on implementation – does not show in the results report”. FAO and ILO: “the portfolio of the organization is presented in one chart, making it possible to see in which areas there is substantial progress and where it might be necessary to focus attention because targets are not met”.
·         What must be changed:
o   “Clear distinction between reporting on activities and output delivery and reporting on expected results (at outcome level)”
o   “Change in the frequency of reporting”
o   Determine which type of evaluation to use for what: self-evaluation and IOS or independent evaluations.
·         Suggested solutions:
o   Annual Programme Implementation Report (PIR): Is the production of outputs on track? No more analysis of results on activity level because too costly, time-consuming and not strategic. “The focus of this report should be a portfolio analysis of implementation, with programme performance and financial performance analyzed together”. “Aggregate statistics on implementation rates; comparisons between plans and actual implementation, traffic light system of on/off track, portfolio analysis of performance against targets”.
o   Quadrennial results report: What are outcomes? Impact? Difference for beneficiaries? Relevance? Comparative advantage? Effectiveness? Basis: evaluations by Sectors, by IOS and a quadrennial self-assessment exercise. Results analysis on programme level, not on activity level. The information in this report (on impact/outcomes) is the basis for Results Based Management, not the information on outputs in the PIR.
o   Continue SISTER [UNESCO’s online activity monitoring system] but simplify it to free human resources for reporting at higher, more strategic levels
o   Reduce number of results to 10-12 like in comparable organizations (versus 128 now)
·         Miscellaneous:
o   Comparison with other organizations: fewer strategic results (3 to 10 versus 128 for UNESCO), fewer pages (100 versus 1000 per biennium!), annual or biennial reports (versus 6 monthly for UNESCO), better analysis of outcomes (UNESCO reports only on activities and outputs (i.e. the work it does), while others report about global trends and outcomes (the CHANGES their work produce)).

No comments:

Post a Comment