By Rod MacLeod
This blog is part of INTRAC’s season on shifting the power through monitoring, evaluation and learning (MEL). The season is part of our celebrations of INTRAC’s 30th anniversary.
‘One of our key principles is participation, which should be integrated into the research methodology’, stated a recent (and typical) Terms of Reference (ToR) for an NGO evaluation. All very commendable.
But an interesting question to pose is: who actually wrote the ToR? Almost invariably they have been developed by an M&E professional sitting in a head office in the global North. Already the purpose for the evaluation, the key evaluation questions, the outputs, and even aspects of the methodology will likely have been set without consulting partners and ‘beneficiaries’ at all. If participatory approaches have been overlooked in designing the ToR, then this sets the tone for the rest of the process.
This is not down to deep cynicism on the part of those who put out such tenders. They are usually well-meaning, but hard-pressed individuals with too many tasks and not enough time. Furthermore, they are under pressure from above – such as senior management and upstream donors – to get the evaluation satisfactorily completed. The next round of funding may depend upon it.
The need to show results
There is a perverse logic that perpetuates this system. The starting point seems entirely reasonable.
Donors want to know what exactly their investment has achieved. What changes has the project brought about? How many people have been reached and by how much have they benefited?
Recipient NGOs are under pressure from donors to provide ‘robust’ answers to these questions using ‘rigorous’ methods. Within government donors, the civil servants liaising with NGOs are themselves under pressure from Auditors General and Public Accounts Committees. When spending taxpayers’ money, donor civil society departments need to justify their decisions against a background where sections of the public, media and politicians are sceptical about aid expenditure.
Donors therefore make their grants conditional on recipient NGOs carrying out M&E which provides sufficient evidence. Quantitative data is usually an essential part of the mix – although it can be supplemented with more qualitative information. UK Aid, for example, wants milestones for each year of a project, which use absolute numbers – not percentages or proportions – and with a target for the end of the project to be compared with the baseline. Many donors seek to aggregate the outcomes from different projects to demonstrate their total impact, even to the extent of showing their combined contribution to the Sustainable Development Goals (SDGs).
Some will agree with such an approach on the basis that ‘if you can’t measure it, you can’t manage it’. Another view, often attributed to Albert Einstein, states that “not everything that can be counted counts and not everything that counts can be counted”. Important civil society work like advocacy, capacity strengthening, convening, or systems change, can prove elusive in terms of objectively proving outcomes and impact.
The result is an elaborate ‘dance of deceit’ in which CSOs make (often questionable) claims about what their work has achieved in a short timeframe, while donors pretend to believe them – and then report it upwards.
The power dynamic underlying the M&E system
You could argue that such is life, and if it keeps the show on the road, let’s all do the M&E jive. But there is collateral damage from such an approach.
Most immediately, it is notable that CSOs working in the global South have not come into this discussion at all so far. But they are the ones who ultimately must fulfil the donor requirements if working in partnership with an INGO. For all the talk of shifting the power and localisation to which donors and INGOs subscribe, these can seem hollow words if the M&E system (even to the extent of universal indicators) are imposed from above. While many emphasize the importance of learning as much as accountability, the latter often outweighs the former in practice.
What can be done?
How can we change this? Admittedly it is not easy, particularly when it comes to institutional donors caught in a chain of upwards accountability. Going back to the starting point, of course donors have the right to ask what difference their money has made. Civil society can seem arrogant if they demand funding with no strings attached.
Perhaps the way ahead is shown by some of the more enlightened foundations. They do not have the same constraints as public donors. As long as their boards are satisfied, this can enable more flexible M&E approaches and systems. This can then be linked to adaptive planning and funding, rather than tying recipients down to rigid lists of outputs. Greater emphasis on learning and improvement also opens up the possibility of more creative processes like using folklore.
Then M&E can become truly empowering rather than restricting.
Rod MacLeod joined INTRAC in March 2007 as programmes director and became a principal consultant in 2011. Previously he worked with Concern Worldwide in Cambodia, India, Sudan and Haiti, in a range of management positions including three at Country Director level. He has also worked at Y Care International as Director of Programmes and also as International Programmes Director for Progressio (formerly the Catholic Institute for International Relations), both based in London. Other roles have been at Save the Children Norway as an adviser covering East Africa and the Horn of Africa, and he has also worked as a freelance consultant based in Uganda.