Obsessed by assessment tools

By Rick James

---

This blog is no. 3 in a series of nine on the subject of capacity building:

No. 1: Calling our bluff on capacity building (March 2014)
No. 2: Cultivating capacity: returning to first principles (March 2014)
No. 3: Obsessed by assessment tools (April 2014)
No. 4: It’s (almost) all about leadership (May 2014)
No. 5: Cultivating character in capacity building (May 2014)
No. 6: Cultivating trust in partnerships (June 2014)
No. 7: Making it personal (June 2014)
No. 8: Letting go: the missing step in capacity building (July 2014)
No. 9: Strategic funerals in capacity building (August 2014)

As NGOs, we are besotted by tools. We treat them like magic bullets, as if suddenly the right tool could solve centuries-old dilemmas of human development.

Tools, like all models, are certainly needed to simplify complex realities. The problem comes when we take these tools too seriously. We believe a limited tool gives an accurate picture of a much more complex and changing situation. We see this particularly with organisational assessment (OA) tools.

Organisational assessment is core to capacity building. Assessing the difference between aspirations and reality is a precursor for change. Most effective capacity cultivation usually starts with some form of collaborative diagnosis. But most of our organisational assessment is not done particularly well. This is for two major reasons:

1. We use simplistic OA tools
2. We use the same OA tools and processes for multiple and conflicting purposes

There are hundreds of OA tools. Many of them cover the same core competencies of vision, leadership, governance, strategy, systems, structures, staffing, resources and networks. Some add a dynamic dimension with scores dependent on the maturity of the organisation.

The problem with almost all OA tools is that they take a mechanical approach to building capacity. Breaking down organisations into neat, measureable parts gives the false impression that capacity can be built in a logical, straightforward way. Dividing up into separate elements misses how these different elements relate to each other. We know too well that any organisation may be more (or less) than the sum of its parts.

Looking below the waterline

Many OA tools focus on the visible. It is easier to ask about the existence of documents rather than organisational behaviour.  We ask: ‘Does the organisation have a strategic plan?’ rather than investigate whether people experience the strategy as clear, focused, followed or even appropriate.

Much of how an organisation behaves is determined by intangible factors such as relationships, integrity and trust. Tools tend to focus on competences, not organisational character. Furthermore, most OA tools are blind to the context. They implicitly assume that all organisations should behave the same wherever they are in the world – usually following a North American/European template. They also suffer from being too comprehensive. By looking at everything, they fail to prioritise what’s most important.

Given these inherent limitations, perhaps it is not so surprising that we tend to only inflict OA tools on our partners – we don’t take our own medicine. As a starting point, perhaps we should make it a principle to only recommend OA tools we have used on ourselves and found to be helpful.

A Frankenstein monster?

Tools can be a bit like Frankenstein’s monster. We first create the tool, then the tool creates us. We begin to focus on only what the tool measures.

I recently read about what one international agency described as its ‘organisational development’ programme. In fact all it was doing was organisational assessment. The NGO confused assessment with change – as if doing a personality test could change your personality. With organisations, assessment is the easy bit. Change is exponentially more difficult.

Furthermore, any organisational assessment needs to be done with integrity. In a recent evaluation, we came across some European funders in Asia who were conducting their own secret OAs on partners – as if this would generate quality data on change.

Multiple and confused purposes

The quality of our organisational assessment is also undermined by using the same processes and tools for completely different purposes. OA can be used to:

1. Make a funding decision (for funders)
2. Catalyse and design an appropriate capacity building intervention (for the client)
3. Measure and attribute the extent of change (for funders)

Each of these may be valid but they are distinct and to a degree mutually exclusive. It is foolish to think that you can achieve purpose 2 by doing 1 or 3 (or vice versa). The key questions to ask are: Who wants it? What is the purpose? Any organisational assessment needs to be fit for purpose.

The recent obsession with OA tools may be partly driven by funders who are interested in using OA to measure change. They imagine that OA tools, administered before and after the capacity building intervention, will provide rigorous, measureable results. It’s intuitively appealing and certainly we must take the measurement of capacity building more seriously.

Unfortunately, pre- and post-test OA tools tempt us to falsely attribute rigour to spurious findings just because they can be quantified. In a later blog we’ll look at some of the limitations of such pre- and post-test approaches as well as more pragmatic and proportionate alternatives.

Organisational assessment is too important to do badly. Yes, we need tools, but let’s not go overboard. We can too easily become mesmerised by the tool and ignore what matters most in catalysing change.