In this blog, Ciaran McDonald, Head of Evaluation and Impact at Rocket Science, considers what the Cabinet Office’s Test, Learn and Grow Playbook means for evaluation in practice. Drawing on our work, he explores why evaluation should be designed around the decisions it can inform, what different findings can reasonably tell us, and what enables organisations to act on what they learn.
Imagine a programme whose future funding will be decided in January, but whose evaluation isn’t due to report until March. The evaluation might be rigorous, the findings useful and the recommendations entirely sensible. But if the next year’s delivery arrangements have already been agreed, the opportunity to act on those findings may have passed.
Of course, the report could still provide valuable learning for future programmes. But it raises a question worth exploring at the outset of an evaluation: what decisions could the findings inform, and when will those decisions be made?
This is what interested me about the Cabinet Office’s new Test, Learn and Grow Playbook, published on 19 October. It provides practical guidance on testing assumptions, gathering evidence and adapting public services in response to what we learn.
One recommendation particularly stands out: agreeing early on what evidence would justify continuing, adapting, pausing or stopping an intervention.
It’s an important principle. And while the Playbook recognises the organisational conditions needed to put it into practice, I think it also presents a useful challenge for how evaluations are planned and delivered more widely.
In a previous blog on good quantitative evaluation, I argued that credible evidence depends on a clear connection between the question we’re asking, the data we collect and the claims we make. Here, I want to look at that relationship from another angle: the decisions the evidence is intended to inform.
Take a mentoring programme for young people at risk of involvement in violence. We might want to understand whether it reaches those who need it most, whether young people engage consistently, whether their confidence and wellbeing improve, or whether the intervention ultimately reduces offending.
These are all reasonable evaluation questions – but they serve different purposes.
If we’re uncertain whether the programme is reaching the right young people, examining referral patterns and speaking to those involved could help us improve how the service operates.
If we’re deciding whether to expand the programme across several areas, however, we’d need stronger evidence about its implementation, effectiveness and whether the intervention itself is responsible for the changes we’re seeing.
The Playbook encourages teams to focus on the assumptions that matter most to an intervention’s success, particularly where the supporting evidence is weakest. That’s a useful way to decide what to test. But when developing an evaluation, we also need to be clear about what decisions that learning could inform.
This means identifying the main decisions expected during the programme, when they’ll be made and who will make them.
That might sound obvious. In practice, it can expose some uncomfortable questions. Is the evaluation being asked to demonstrate impact before the relevant outcomes could reasonably emerge? Are we collecting data because it will inform a decision, or simply because it has always been part of the reporting requirements? Will the findings arrive in time to influence the next funding round?
The answers should influence how the evaluation is designed and delivered, not just how its findings are presented.
Even with well-defined questions, evaluation rarely provides the straightforward answers we’d sometimes like. A good example comes from our work at Rocket Science evaluating Plan B, a custody-based violence-prevention service in South Yorkshire.
We combined interviews and outcome measures with a quasi-experimental analysis comparing reoffending among participants with a matched group who hadn’t received the service.
We found no statistically significant overall differences in reoffending between the groups over the period examined. That didn’t establish that the service had no impact, but we couldn’t demonstrate an overall effect through that analysis.
At the same time, evidence from interviews and measures of wider protective factors suggested encouraging changes in areas including emotional wellbeing, behaviour and relationships.
We also found that people who completed the programme had fewer and less harmful reoffences than those who didn’t. That was worth exploring, but it wasn’t proof that completion caused those differences. People who complete an intervention may differ from those who don’t in ways that also affect their likelihood of reoffending.
Alongside the impact analysis, we identified important practical lessons about engagement, implementation and how progress could be better understood and measured. The commissioner and provider took forward recommendations relating to data-driven engagement and retention, more consistent use of outcome measures and the duration of support for clients.
These were practical opportunities to strengthen delivery, even though we couldn’t make a definitive claim about the programme’s overall impact on reoffending.
Had we focused solely on whether the matched analysis demonstrated a statistically significant reduction in reoffending, we would have overlooked useful learning about delivery. Equally, had we treated encouraging changes among participants as proof of reduced offending, we would have overstated what the evidence could support. Neither interpretation would have been particularly helpful to the people making decisions about the service.
Different findings can justify different decisions. We might have enough evidence to improve an aspect of delivery, while still needing stronger evidence before recommending substantial further investment. Sometimes the most useful conclusion is that an approach shows promise, but we don’t yet know enough to support a bigger claim.
For commissioners, understanding that distinction is important. For evaluators, communicating it clearly is part of our responsibility.
This is particularly relevant as test and learn approaches become more widely adopted.
There’s a lot to welcome about encouraging organisations to gather feedback, test ideas and improve services while they’re still being delivered. But moving quickly shouldn’t mean accepting evidence that isn’t strong enough to support the decisions being made.
Participants in a mentoring programme might report feeling more confident after receiving support. Monitoring data might show improvements in their school attendance over the same period. Both are encouraging signs of progress – but neither, on its own, tells us whether the intervention made the difference. That requires us to consider what might have happened without the intervention, drawing on experimental or quasi-experimental approaches where feasible and appropriate.
The distinction is recognised in both the Playbook and the 2026 Magenta Book guidance on Test and Learn. Test and learn is intended to strengthen, not replace, robust impact evaluation.
There’s another issue we need to consider when programmes are being adapted. If the intervention changes repeatedly, we need to document what changed, why and when. Otherwise, we risk reaching the end of an evaluation without being able to establish what was actually delivered or interpret the outcomes with sufficient confidence.
We should also consider future impact evaluation when designing and rolling out interventions, so that delivery decisions don’t unnecessarily limit opportunities for credible comparison.
For me, the key is proportionality. We don’t need the same strength of evidence to adjust an appointment-booking process as we do to justify expanding a major intervention.
This is where I think the biggest practical challenge lies. An evaluation might identify a problem with how people access support, or suggest that a particular element of an intervention needs to be delivered differently.
But what happens if the existing contract specifies how the service must operate? If funding is tied to activities that no longer appear to be the most effective? Or if making a change requires several organisations to agree?
More evidence won’t necessarily resolve those problems.
The Playbook recognises the importance of organisational culture, decision-making authority and flexibility. I think those considerations need to feature much more prominently in conversations about evaluation commissioning.
That doesn’t mean abandoning accountability or allowing programmes to change without appropriate oversight. It means thinking early about where there is flexibility, what changes would require approval, and how learning will be reviewed.
It also means being prepared for evidence that challenges our assumptions about who is benefiting.
An intervention might show encouraging outcomes among those who engage, while consistently struggling to reach people with greater or more complex needs. Expanding the existing approach might increase the number of people supported without addressing that problem.
And evidence that a particular delivery model isn’t working doesn’t mean the people receiving support no longer need it. Sometimes the appropriate response is to redesign provision or find another way of achieving the intended outcomes.
These decisions require input from the people using and delivering services, as well as commissioners and evaluators. They also involve costs, competing priorities and judgements about risk and fairness. Evidence should help make those choices better informed, rather than being expected to settle every question on its own.
That’s a commissioning and leadership challenge – not simply an evaluation problem.
It’s easy to talk about commissioners and delivery organisations needing to make better use of evidence. But I think those of us working in evaluation have some responsibility for how that relationship develops.
We should be ready to help clients sharpen an evaluation brief when the decisions it needs to inform aren’t yet clear. We should be honest when the proposed design or available data won’t support the conclusions a client hopes to draw. And we should think carefully about whether our approach will get useful findings to decision-makers while there’s still an opportunity to act.
That might mean agreeing review points at the outset, sharing emerging implementation findings before the final report, or helping commissioners distinguish between changes that can sensibly be made now and questions that require further investigation.
None of this means every evaluation needs to operate in real time. Accountability, longer-term impact assessment and informing future investment are all legitimate purposes. Nor is every intervention suited to continual adaptation.
A constructive conversation at inception about what an evaluation can realistically establish helps us agree the questions that matter most, design the work accordingly and be clear with clients about what the findings will – and won’t – be able to tell them.
That also means accepting that our advice won’t always be what a client wants to hear. Sometimes there will be encouraging evidence of progress but insufficient evidence of impact. Sometimes we’ll need to challenge assumptions about how an intervention works. And sometimes the appropriate recommendation will be to gather more evidence before making a significant investment decision.
Our job isn’t to make every finding sound conclusive. It’s to help organisations make sound judgements about what they know, what remains uncertain and what they should do next.
For me, that’s the real opportunity presented by the new Playbook.
It gives us a timely reason to think about how evidence is generated and used throughout the life of a programme, and how commissioning and delivery arrangements can support meaningful learning.
At Rocket Science, this thinking is already central to how we approach evaluation. We work closely with clients to understand the decisions they’re facing, what evidence would help inform those decisions, and how findings can be used to strengthen delivery and future investment.
I’d like to see this approach become more consistently embedded across the sector. The decisions an evaluation needs to inform should shape its design from the outset, including the questions we ask, the indicators we select and the methods we use.
What might the evidence tell us that would make us reconsider our approach? Who would be responsible for making that decision? And would they have the opportunity to act?
