Matthew Baumann Associates
Approaches to evaluation that work in a complex world
Understanding your challenges
Increasingly evaluative thinking is being used at the front end of programme and policy development. Existing theory can be used to frame and model complex behaviours. Participatory systems mapping can be used to understand the multiple perspectives on the causal drivers of a challenge across multiple domains, and at multiple 'levels' and to identify opportunities for intervention. Social research and behavioural insights can be used to gain deep understanding of what activities might be used to trigger and sustain change over time. Evidence maps can be produced to illustrate the knowns and unknowns of a problem and highlight where more evidence is needed. Taken together this body of evidence can help highlight more viable strategies for intervention.
Ex-ante evaluation
Ex-ante evaluation can be used to de-risk a policy. It helps by working through the many design and delivery issues and challenges before any implementation is undertaken. Central to this is the development and critical assessment of a theory (or theories) of change that explains how a programme is supposed to work, and how it will be implemented. I work with clients / stakeholders to draw out the key assumptions for a policy or programme and then use this process to assess the logic of 'how' activities are expected to lead to outcomes and impacts.
Deliverability assessment is another key part of ex-ante evaluation, by understanding the capacity and capability of actors on the ground to deliver the programme and what might enable or inhibit effective delivery. Theories of action can be used to underpin this kind of assessment.
Evaluation scoping and design
More complex evaluation projects require detailed scoping, ideally before they are commissioned so that commissioners and tenderers have a good understanding of the policy / programme and its context, the evaluation requirements and the feasible data and evaluation methods that are needed for effective evaluation commissioning. I work with clients and stakeholders to understand potential evaluation purposes and objectives, establish a shared understanding of the programme strategy or policy, establish evaluation criteria and key questions. From here I map existing and potentially applicable data sources, scope out new data collection, and clarify feasibility of different evaluative and analytical techniques. Key to this is a process of collaborative exploration and discussion about what is important and why so that the evaluation is focused and deliverable.
Evaluation of emergent programmes
Complex programmes tend to work in an emergent way towards multiple outcomes. They may have to overcome many barriers and obstacles to achieve their goals, over multiple 'stages'. Success may be dependent on the behaviours of multiple stakeholders who can only be indirectly influenced or incentivised to behave differently. In this context, the evidence base for how to achieve these goals is uncertain and clients need to actively manage delivery, changing activities over time to adapt to a changing environment. Actively monitoring and evaluating progrmames delivery and outcomes over time is critical in these contexts, and can provide regular, timely, actionable learning and evidence that can be used to strengthen programmes and policies. These kinds of evaluation frameworks need to be flexible, and will need to be reviewed regularly to ensure they are tailored to emergent requirements.