UNESCO’s future
depends on the willingness of Member States to support it. This willingness
depends on UNESCO’s capacity to show Member States that it is useful and that
it produces results. In an excellent evaluation report
UNESCO evaluated whether its results-reports actually do provide (good)
information about results. This blog is a summary of its findings (for everyone),
followed by an Annex containing quotes from the report (for diplomats and experts who will
discuss this next
Tuesday).
[NB. Sometimes I had to
simplify the conclusions of the report a little to make them easily
understandable, but without altering the conclusions. I also added a personal
conclusion about the meaning of this rather critical report.]
Problem: little to no information about the impact of UNESCO's work
Results are effects
that UNESCO’s work has on reality. The main conclusion of the report is that
UNESCO’s results-reports show very little information about these effects. Almost
two thirds of the report describe UNESCO’s own processes and almost one
third is about the “things” that are produced by these processes
(meetings, workshops, policy guidelines, etc.). However what Member States need to convince their governments of UNESCO's usefulness is much less present in the report:
- only one fifth is about effects that UNESCO's meetings etc. are supposed to produce, i.e. changes in the behavior of people and institutions who can change society; and
- the report tells nothing about whether society actually did change thanks in part to UNESCO.
In this respect UNESCO’s results-reports
are very different from the ones in other international organizations, as is
shown in the picture below. This is a problem because UNESCO cannot be rewarded
for impact it doesn’t report on.
 |
| UNESCO only reports about the work it produces while other international organizations report about the effects their work produce. |
Solution?
It would be easier to convince Member States of UNESCO's usefulness if its results-reports made a clear distinction between UNESCO's work (meetings, workshops, reports) and the positive effects of this work (better heritage protection, better education). To achieve this UNESCO could present two different types of results-reports that
focus on different things:
- An implementation report that only informs Member States if UNESCO’s work is done correctly: does UNESCO do the things right? Were activity targets met for example? Were the meetings, policy guidelines, global reports etc. produced in time, in the right quantity and within the given budget and quality requirements? This report would not look for causal effects, it would only verifiy whether the UNESCO work is done as planned.
- A results-report that only informs Member States if UNESCO's work has positive effects on society: does UNESCO do the right things? For example: did the meetings and policy guidelines actually motivate countries to improve their education policies, practices and capacities? Did press freedom become more prominent on the international political agenda and did journalists and democracies benefit from this? This report would not provide a catalogue of UNESCO’s meetings and policy guidelines but focus exclusively on their (positive) effects on the world outside UNESCO.
Although the report
concludes that UNESCO provides very little information about the effects of its
work, this does not mean that
UNESCO’s work doesn’t have effects! It only means that these effects
are not made visible enough, that’s all. The list of examples of UNESCO's positive effects is too
long to describe here. Let’s start with the protection of cultural heritage.
What are UNESCO’s effects in the field of culture?
Without UNESCO the awareness
of and subsequent care for world heritage would simply not have existed. UNESCO
literally created the “world
heritage sentiment” in the late Fifties when it saved the Nubian monuments in
Egypt (like the Temple of Pharaoh Ramses II) when they were about to be
flooded due to the construction of the Nasser Dam. The effect of interventions
like these is first of all that everyone in the world now are aware of the fact that they have some irreplaceable treasures in common. This awareness
leads subsequently to better heritage protection policy and practices thanks to capacity
building projects organized by UNESCO. To maintain, nurture and strengthen
these effects UNESCO created several Culture Conventions that promote the protection of cultural heritage.
Is UNESCO still producing effects today?
Don’t think that
these Conventions from the Fifties and Seventies are old laurels that UNESCO
fell asleep on. Quite on the contrary: wherever cultural heritage is in danger
UNESCO’s convening power is there to call the world around one large
table in Paris. Like during the Mali crisis, where UNESCO called governments and experts together to plan action to assess the damage, to repair it and to raise the
necessary funding
for it. And UNESCO is not only saving cultural heritage, but also human lives. Thanks
to UNESCO, tsunamis will not kill 230,000 people again like the Indian Ocean did in 2004 because UNESCO initiated and coordinated the creation of Tsunami
Warning Systems. These are concrete examples of how UNESCO enables the
world’s nations to concretely address challenges they would not be able to address on
their own. And for a low price: UNESCO’s entire regular
budget for one year is less than half of the annual budget of the University of
Amsterdam in the Netherlands.
What’s next?
The challenge this
blog addresses is how to make all these results more visible in a report. This is
what Member States will discuss next Tuesday 2 September 2014 in the Preparatory
Group of UNESCO’s Executive Board. This Group is supposed to propose a new
format for UNESCO’s results-reports. As the interested reader will imagine, the
evaluation that was written by the Swedish consultant Kim Forss will come in
very, very handy.
--------------------------------------------------------------------------------------------------------
ANNEX: Quotes and excerpts from UNESCO’s
“Formative Evaluation of UNESCO’s Results-Reporting"
·
Challenges:
o
No clear separation of activity and
results-reporting: Activity/output-reporting
is fundamentally different from reporting on expected results (outcomes).
Outcome-reporting requires different data and research: info about target
groups of beneficiaries, sample section, gathering of data from target and non-target
groups, analysis of the data. UNESCO’s self-reporting on outcomes is too
fragmented and weak for that. Consequence: in the C/3 there is no reference to
impact at all and only 20% is about outcomes. 30% is about outputs and 60% describes
only UNESCO’s own processes. “The [C/3]
text describes what the Organization has done in terms of holding meetings. It
presents which meetings were held, why and with whom. It does not say what
happened after the meetings or whether any results were achieved.”
o
Biased self-reporting: Because of low levels of trust in
the Organization self-reporting on results is likely to be biased. Consequence:
“there is little suggestion in the C/3 or
the EX/4 that anything has gone wrong or might have had negative results”.
o
Inefficient use of results reporting: Statutory reports (E/4 and C/3)
are much thicker than in other International Organizations (1000 pages per
biennium for UNESCO versus approx. 100 pages for other organizations). “Within the UNESCO results-reporting system,
the self-reporting workload (due to the frequency of reporting and the small
unit of analysis of reporting) is too high”. Conclusion: “value for money on self-reporting practices
in the UNESCO system is rather low” because a lot of time and money is
invested in it to produce an extremely large quantity of information “but still fails to answer basic questions
about results (at outcome level)”. “The
type of information requested by Governing Bodies and stakeholders is rarely
produced in the system”.
o
Lack of focus: “Excessively high number of results and performance indicators”
o
Lack of causal theory about how
results should be attained: “need for a clear articulation
of causal assumptions linking activities to results”: “EX/4 does not provide information on (…) analyzing change processes,
discussing causality”. The report provides information on outputs but there
is no information about impact, about a theory of change and about attribution
of contribution to change. “There is
little evidence to substantiate the claims to any outcomes and there is no
explanation of how activities and outputs lead to outcomes”. “This evaluation did not find examples of
outcomes which were substantiated by a theory of change explaining the links
between the UNESCO activities and outputs and the hypothetical outcome”. “The question of how results were produced
and whether they were caused by UNESCO alone, in conjunction with other
agencies, or independently of UNESCO is not discussed in the EX/4 and C/3
reports analyzed here.” “It is not
expected to have a detailed theory of change for every type of activity, output
and outcome in the EX/4, rather what is needed is a theory of change or a
causal framework that connects patterns of activities and outputs to higher
level outcomes of interest (i.e. expected results).” What is needed: a complete
picture of all levels of the process of change (convention>national policy>projects>effects)
and UNESCO’s role in it. Good examples: World Heritage Convention,
International Institute of Educational Planning (IIEP).
o
Poor measurement of results against
indicators: “Evaluations (…) do not systematically feed
into the self-reporting processes and consequently into the EX/4 and C/3”.
“If three reviews were planned in the C/5
[UNESCO’s programme], but only one was completed, this fact – reflecting poorly
on implementation – does not show in the results report”. FAO and ILO: “the portfolio of the organization is
presented in one chart, making it possible to see in which areas there is
substantial progress and where it might be necessary to focus attention because
targets are not met”.
·
What must be changed:
o
“Clear
distinction between reporting on activities and output delivery and reporting
on expected results (at outcome level)”
o
“Change
in the frequency of reporting”
o
Determine
which type of evaluation to use for what: self-evaluation and IOS or
independent evaluations.
·
Suggested solutions:
o
Annual Programme Implementation
Report (PIR): Is the
production of outputs on track? No more analysis of results on activity level
because too costly, time-consuming and not strategic. “The focus of this report
should be a portfolio analysis of implementation, with programme performance
and financial performance analyzed together”. “Aggregate statistics on implementation
rates; comparisons between plans and actual implementation, traffic light
system of on/off track, portfolio analysis of performance against targets”.
o
Quadrennial results report: What are outcomes? Impact?
Difference for beneficiaries? Relevance? Comparative advantage? Effectiveness?
Basis: evaluations by Sectors, by IOS and a quadrennial self-assessment
exercise. Results analysis on programme level, not on activity level. The information in this report (on impact/outcomes) is the basis for Results Based Management, not the information on outputs in the PIR.
o
Continue SISTER [UNESCO’s online activity
monitoring system] but simplify it to free human resources for reporting at
higher, more strategic levels
o
Reduce number of results to 10-12 like
in comparable organizations (versus 128 now)
·
Miscellaneous:
o
Comparison with other organizations: fewer strategic results (3 to 10
versus 128 for UNESCO), fewer pages (100 versus 1000 per biennium!), annual or
biennial reports (versus 6 monthly for UNESCO), better analysis of outcomes
(UNESCO reports only on activities and outputs (i.e. the work it does), while others
report about global trends and outcomes (the CHANGES their work produce)).
