Ensuring value for money in government policy transcript

Welcome to this webinar, which we are delighted to be bringing you in partnership with the UK Evaluation Society. This webinar marks the launch of
Verian's Centre for Value for Money and our recently published Verian Guide to VFM Evaluation. Our speakers today are myself, Alex Hurrell. I am the UK Head of Evaluation at Verian and I also lead our newly established Centre for Value for Money. The Evaluation Practice that I lead has a focus on delivering high quality, impactful evaluations drawing on broad capabilities and impact evaluation including RCTs, quasi-experimental designs and theory-based approaches.

Building on deep expertise and experience in large scale and complex primary data collection covering both surveys and core research. We have particular expertise in large scale evaluations, but we also have a broader objective to contribute to and facilitate knowledge sharing and building capacity across the evaluation community. This event is part of that initiative, bringing together external speakers to give us varying perspectives on key issues facing evaluation commissioners and practitioners.

Dr Julian King is a New Zealand based public policy consultant and a member of our newly established Centre for Value for Money. With a background in healthy mix and decades of evaluation experience, he is an international expert in VFM evaluation and has developed a new approach to VFM evaluation, the Value for Investment or VFI approach. He's done this through his doctoral research which he will introduce in this webinar.

Lucie Moore is the head of the Evaluation Task Force, a joint Treasury and Cabinet Office unit. She joined the Task Force at its inception in 2012 and prior to this role Lucie was an Evaluation Advisor at the FCDO.

The ETF have recommended the VFI approach to VFM evaluation as part of their Evaluation Academy training programme. Dr Kirstine Szifris is a criminologist by background and has spent the last decade working in evaluation in and around the criminal justice system. She recently joined the Crown
Prosecution Service as their Evaluation Manager, having come to academia and the third sector and took on the role of
President of the UK Evaluation Society in December 2023.

Following the presentations and reflections from the panel, there will be some time for Q&A. Please put forward your questions using the Zoom Q&A function.
This webinar is also being recorded. We'll share this soon after the session. We're also intending that today's webinar will be followed up by an in person event to further explore the key issues and questions emerging from the discussions. So if you are potentially interested in joining that, please do let us know.

In line with the UK Government's policy priorities and mission objectives, earlier this year Verian launched 7 Centres of Excellence to compliment our
existing teams and support our clients with the delivery of the
Government's national priorities.

The Centres connect across the UK teams and bring together variance, multidisciplinary methodological expertise, innovations, case studies and thought leadership, together with our policy and subject matter expertise and deep experience in citizen engagement.

Each Centre of Excellence is led by a member of the UK Leadership team and connects with the expertise of colleagues from across the UK and the globe.
In line with this, the purpose of the Centre for Value for Money is to bring together expertise, best practise and innovation in value for money evaluation from across the globe, with a focus on how to effectively assess what works in delivering policy impact, whether impacts justify programme costs and what improvements can be made in design or delivery to increase impact or value for money evaluation is critical for ensuring value for money.

Impact evaluation provides an impartial and robust assessment of whether a government policy is working and if so, value for money assessment can determine whether that impact is worth the cost. Impact evaluation tells us the extent to which a policy or programme has delivered its outcome objectives.
It tells us the impact that policy or programme has achieved relative to what would have occurred in the absence of that programme.

This impact provides value to society, or at least it should do if the programme is relevant and delivered effectively. And this value must be weighed
against the cost that was allocated to the policy or programme. So we compare the value to the money to assess whether the programme is a good use of
public resources to ensure this feeds into effective spending allocation decisions. Value needs to be considered broadly and holistically and not limited to just those impacts that can be readily monetised. This is in line with our recently published varying guide to VFM evaluation and will be covered further by Julian shortly.

A comprehensive impact evaluation should also assess how the policies impact can be improved either in it's design or in it's delivery. It can therefore also feed into an assessment of maximum potential impact and value for money which can be benchmarked against the assumptions and objectives set up and original economic appraisal and business case.

In other words, impact evaluation evidence will tell us if the policies achieved it's original intended VFM and how this could be improved at design
stage. Value for money can be substantially enhanced by bringing in external evaluation scrutiny into the programme design and appraisal process as
early as possible. It is increasingly expected that every policy should have a comprehensive theory of change which sets out the overarching policy objectives, the activities or outputs the policy will deliver and how these will deliver the intended outcomes and impacts.

A rigorous prospective evaluation of the policies design can ensure design is underpinned by logically sound impact pathways, existing evidence both from the UK and internationally. This can be done by evaluators reviewing and critiquing and policies underlying rationale. It's the programme value
proposition and it's theory of change in practise by probing into the logic theory and existing evidence for precisely why and how programme activity A
is expected to lead to outcome objective B, etcetera, etcetera. In other words, by front loading some of the evaluation effort, it would ensure that all programmes are explicit about how and to what extent they will successfully generate future public value. This would also maximise chances of ensuring the policy is successful from the outset and therefore requires less course correction or at the extreme, early termination of ineffectual programmes.

This in turn would ensure greater VFM in public spending overall, underpinning a right first time policy design and delivery culture. VFM can also be enhanced through delivery by ensuring that programme delivery is underpinned by robust monitoring, evaluation and learning or MEL systems in line with internationally accepted good practise.

An effective programme level mail system provides robust evidence and insight to enable adaptive management and course correction to achieve outcome
objectives and maximise value for money. An effective mail system should be internal to the programme delivery function I should set. It should be separate from any independent evaluation activities. An effective embedded malfunction should systematically track the intended inputs, outputs, outcomes and other specific contextual contextual factors critical to success as per the programme theory of change and overall VFM performance.

It should be aligned with and integrated with the any benefits realisation monitoring requirements, but it provides an enhanced function. An effective mail system will therefore provide programme managers with firstly high frequency data on whether their projects are on track to deliver their objectives and if not, how to fix this and secondly a systematic evaluation and learning plan with regular built in reflection points to ensure monitoring and evaluation data
and insights are fed through into in flight adjustments to programme implementation and updating the underlying programme design and theory of change.

Value for money is further promoted by routine and independent evaluation. Independent evaluation provides a robust, impartial judgement on how well a programme or policy is working in terms of achieving it's original objectives and intended value for money. This provides accountability to UK taxpayers as well as learning to improve the programme and inform back broader sector level policy.

I'm with international best practise. Independent evaluation should provide a holistic assessment of all aspects of performance, for example relevance, coherence, efficiency, effectiveness, impact, sustainability and also equity. In line with the Magenta Book and Green Book, guidance and evaluation will generally involve process impact and economic components. The evaluation will usually provide baseline interim and end line assessments.

The interim assessments can provide independent scrutiny of an ongoing performance assessment delivered by the internal mail system. Evaluation protocols, design documents and final reports should ideally be published in a timely manner and made publicly available to ensure accountability and maximise learning.

Ideally accompanied by a management response from the Commissioning Commissioner setting out how the programme will respond to the evaluation
findings to improve performance and value for money. Finally, value for money is maximised at a portfolio or departmental level by ensuring the evidence of impact and value for money of proposed policies and is systematically synthesised and reviewed and potentially this could also be complemented by sector level rapid evidence reviews. This provides a broad evidence based perspective on what types of programmes and interventions in each sector are demonstrating impact and value for money and a showing or a strong strong impact potential based on early results and previous relevant evaluation evidence. This should in turn guide public spending allocations with priority given to policy options that will deliver most public value.

So we know that evaluating value for money is crucial for identifying what programmes and policies will generate the most value to society and therefore
value for money assessment needs to guide public spending decisions. We also know that evaluation, and impact evaluation in particular, is critical for VFM
evaluation. It tells us if a policy is working and the impact it has delivered. But how do we make the link between impacts delivered and the value to society?

This represents integrating economic analysis and cost benefit analysis or CBA and doing so in a comprehensive and systematic way. I will now hand over to Julian who will introduce a new approach to value for money evaluation.

Thanks very much Alex and good afternoon everyone. Yes, it is as Alex was just eluding, we think that value for money and cost benefit analysis are sometimes used as if they're synonymous or interchangeable terms. And we think in fact, that the picture looks more like this. So cost benefit analysis can be and often is an important way of answering a value for money question, but it doesn't always provide a full answer to a value for money question. Analysis we can think of as a blender for values. So we take the the resources that are used in a programme, that's the value consumed and we can measure that and the currency of your choice. And we can also measure the impacts and value those in the same currency. So value consumed and value created.

And there's one extra secret ingredient which goes into the blender, which is the discount rate, which is adjusting the costs and benefits for timing. And when we whizzed that out, we can produce an indicator like a benefit cost ratio or various other indicators that tell us whether on the basis of the cost benefit test, whether a programme appears worthwhile. Just to give an example of what that can look like, consider drug courts. So drug courts treat addiction as a health issue rather than solely a criminal justice problem. And in doing so they're addressing root causes of drug related crime.

The outcomes that can happen as a result of a drug court are quite broad. As you can see if you if you scan your eye down this list, we might see some things that would be relatively simple to convert into monetary values. I think a lot of the potential outcomes at the top of this list—reduced drug use, reduced offending, decreased incarceration, cost savings to the criminal justice system. These are all things that we could endeavour to place monetary values on and include in a cost benefit analysis. On the other hand, there might be some that would be much more difficult to measure in money and might have to be left out of such an analysis. So it might not throw light on every the value of every outcome from the drug court.

Broadly there are, there are many ways of placing monetary values on things and we can categorise those as some things are already measured monetarily and that's convenient because then we can measure things like changes in income and changes in costs. So within our drug court CBA we could be looking at the direct programme spend. We can, we can look at budget figures for that with the actual spend figures on the benefits side, reduced costs to the wider system, police, prisons, hospitals and so on.

We could probably account for there are some other benefits that aren't automatically measured in money, but there are, there are market prices for some things. Some things are bought and sold in real markets and we can use those market prices as proxy values to estate things that are in our cost benefit model. So, for example, on the cost side, perhaps we want to account not only for the direct programme spend, but also for money that individuals are spending out of their own pockets to participate in the drug court programme.

We might not know exactly what everybody's spending on their transport costs, but as a proxy we could use something like average public transport fare and include that in the model. Similarly, on the benefits side, we may be able to estimate values for increased productivity, reduced social costs of crime and so on. Some things are not bought and sold in real markets, but we can set up pretend markets of various types, set up studies, value and games in which people reveal what they're willing to pay for various intangible, hard to monetize benefits.

An example could be improved quality of life. So we could place a monetary value on improved quality of life by by doing some form of willingness to pay study. And then perhaps there might be some benefits that, although very important, may have to be treated as out of scope because

Perhaps we don't have the data or we don't have the the time or the the money or skills to be able to place monetary values on it, on everything. And it's pretty routine for a cost benefit analysis to leave out some social and tangible outcomes that are just too hard to value in monetary terms. So perhaps strengthen family relationships, stronger social support systems could be one example of that.

So broadly we we face a trade off when we're going to design A cost benefit study. We could keep it fairly narrow. We could just look at the the costs and benefits that are measured in real money, or we could go a little bit broader and maybe use some market proxies for some additional costs and benefits. And we can probably keep that fairly accurate and precise. We may, on the other hand, want to make the study more comprehensive and include additional social value. And in doing so, we have to accept some trade-offs. It's going to be slightly less accurate, less precise, and we might need to decide where we're going to draw the line in order to have a sufficiently credible study that's that that stakeholders will find credible. And it could be that there are some things that we just have to treat as out of scope because they're infeasible. And that's just one of the reasons why Alex and I in this, in this document that we recently released, argue that CBA tells us something important. We should as as a field, as evaluators, we should be making more use of cost benefit analysis.

It's addressing an important aspect of value for money, which is the monetary value of impacts relative to costs. And it's kind of addressing a question loosely that's saying is society better off overall in the aggregate? The technical term for that is Keldor Hicks efficiency. So you can think of Keldor Hicks efficiency as one criterion. But as we know in evaluation, we often need to examine multiple criteria. We might want to look at relevance, distributive justice, productivity, acceptability and so on. So there are other aspects of value that that cost benefit analysis can't help us with. It can't help us with intangible benefits and costs, or at least the ones that we weren't able to estimate monetary values for. And there are various other limitations of CBA that are well covered in this document that I won't go into now.

We're not the only ones who think so. So they're for example, these two academics from the University of Oxford have argued that technocratic and and democratic approaches to public decision making should be combined and and in this approach, cost benefit analysis would be just one many inputs into decision making processes. Similarly, in the US, these two Ivy League laws who wrote a book called New Foundations of Cost Benefit Analysis, realised that CBA is not a super procedure. They used this term super procedure to refer to the idea of something that that could theoretically combine the insights from cost benefit analysis with all the other considerations, all the other criteria and evidence that we might want to bring into a decision. But they conceded in their book that they that they didn't know how that could be done. They just thought it would be a good idea if if, if that was possible, they should have asked an evaluator because we've known how to do this for 50 years. It's called the general logic of evaluation involves using criteria, definitions of aspects of performance and standards levels of performance to make sense of multiple streams of evidence in order to reach transparent judgements. We'll come back to that point.

So evaluation is the field that's responsible for determining the value of things. Values our middle name. And when I talk about value, I'm referring to the merit, worth, or significance that people and groups place on something. So things don't have value inherently. People place value on things. Some of the things that people place value on include impacts. Here I'm using a broad definition. Real differences in people, places, and things that are caused by organisational action. The avoidance of doubt that could include short, medium and long term outcomes, intended or unintended, relative to a counterfactual, but could be measured in in many different ways. I'm including all of that in the term impact. So people place value on impact. People also place value on the organisational actions, things like programmes, policies, products and services. How things done matters. Value gets placed on how things are done and those actions are fuelled by resources. And we place value on resources too. Money, yes, but not just money, not just money. People knowledge and skills, relationships and so on are resources that fuel organisational actions as well.

So those components, the resources that fuel organisational actions that cause or contribute to impacts that people place value on can all be placed in a diagram like this, like a value chain. A little bit like a theory of change, but a little bit broader than a theory of change. Theory of change will often focus mainly on actions and impacts. And here we're we're zooming out a little bit and we're thinking about the resources that are invested in the value that's created as well. And we can use this value chain to help us to answer questions about value for money. Or in other words, is our programme, is our policy good resource use? One of the things we can do with this value chain is analyse it in pieces. So there's a framework called the five ES and one of those ES as economy. And when we talk about economy, we're talking about good stewardship of resources. We're looking after the pot of money and are we being good stewards of the intangible resources as well That that that go into a programme?

The second E is efficiency and that's concerned with the productivity of our organisational actions. What ways of working will ensure that we get the best value from this investment. The third E is effectiveness, and that's concerned with impacts and outcomes. The 4th E actually starts with C and it's called cost effectiveness and that's concerned with whether a programme creates enough value to be considered worthwhile. That's the pace where cost benefit analysis can often give us really important insights. The 5th A is equity and it's important at every level of this value chain. So by analysing the value chain like this, it can help to give us some insights to to collect different types of evidence, different facts and figures that can help us to make a determination, but we still have to make a judgement as a good resource use. That's an evaluative question that we have to answer.

Perhaps it might help to if we produce some ratios. So we could look at the ratio of value over resources. That's called the benefit cost ratio. That comes out of a cost benefit analysis or equivalent, equivalently a social return on investment ratio and takes the same form. Something else we could do is look at a ratio of impacts to resources and health economics. We call that a cost effectiveness ratio. And at efficiency level, we could also produce various ratios of outputs to resources. So now we've given ourselves a constellation of numbers, and those numbers can help us to make judgements. But we still do have to make judgements because we're answering an evaluative question about good resource use. And that question sits at the intersection of two disciplines. Evaluation, the discipline responsible for determining how good something is, whether it's good enough. In economics, the study of resource use.

So when we answer value for money questions, we're answering an evaluative question about an economic problem. So This is why we think that we need to be drawing on both disciplines. The insights we can get from both disciplines to answer questions like how well are we using resources? Is the resource used creating enough value? How can we create more value from the available resources so to bring it back to evaluative reasoning? These good resource use questions can be addressed using the general logic of evaluation using criteria and standards.

We recommend developing rubrics as an intuitive, visual way of representing criteria and standards. So here, just as an example of a rubric, I have a matrix that has five criteria. I'm using the five ES that I just introduced before to illustrate, and I have 4 levels of performance or value, excellent, good, adequate, and poor. We don't have to have 4 levels. We could have more or less than that, and we could give them different labels. So it's just here to illustrate. And then within the body of the rubric, we start to populate this table with descriptions of what the evidence would look like at the intersection of each criterion and each standard. And that gives us a basis for making judgements transparently from multiple streams of evidence. And that basis for making judgements needs to be Co constructed and agreed.

It wouldn't be any good for me to sit in my office and make a rubric all by myself because that rubric would reflect my assumptions and values as as one person. 1 evaluator. We need to address this question of value to whom and ensure that the the values of people who affect a programme and those who are affected by a programme are are brought in appropriate ways into determining the basis which judgements should be made. And our approach to value for money hinges on this idea of develop developing transparent rubrics with stakeholders to make judgements from the evidence. So this approach we call value for investment.

It's not another method because as a field we already have all the methods and tools that we need. What we're contributing is a set of principles and a process to to help everybody align methods and tools to answer value for money questions. So to do this, we provided a sequence of eight steps that looks like this.

We have 4 design steps and then four evaluation or implementing the evaluation steps just for a quick fly over. Step one is to understand the programme. This is the sort of thing we would all do in the evaluations we conduct anyway, right. So align the evaluation to the context but but also to specify here a value proposition. And by that I really just mean describe the value of the programme in words. So coming back to drug treatment courts, and here's just AI won't go into the details, but we're just describing the value in words and we're thinking about to whom are drug courts valuable and how are they valuable? And we're specifying that because that brings clarity to Step 2 when we're defining our aspects of value for money, the aspects that we're going to look at. So here for convenience again, I'm using the five PS as a handy go to, not to be all and end all of of value for money criteria, but a good starting point into a conversation of of what aspects of good resource use we might need to look at. On the right hand side of the table, I've provided some examples of context specific sub criteria. So rather than using generic off the shelf definitions of economy, efficiency and so on, we're defining them in a way that has meaning in the context of our drug treatment court.

Step 3 then is to develop the standards, the other half of the rubric levels of value for money. Here's a generic set that comes from a guide that I wrote with Oxford Policy Management. And in this guide, we have generic definitions for excellent, good, adequate and poor so that we can align Each value for money framework we develop can be aligned to these generic definitions so that terms like excellent and good have consistent underlying meaning across multiple VFM assessments.

So we're bringing our criteria and standards together to make a rubric and we're describing what the evidence would look like at different levels of performance within the table. And just to give you one example, here's to illustrate how we might include the results. The outputs of a cost benefit analysis is a piece of evidence that we would use to make a determination about cost effectiveness based on the prospect, based on our judgement of the prospect of the programme breaking even, In other words, creating more value than it consumes. So if it breaks even beyond a reasonable doubt, we would call that excellent. If it breaks even more likely than not, we would call that good and so on. It's just there to illustrate.

Once we've completed our rubric, we can stand back and use that to determine what evidence we need to collect. And it becomes apparent very often that we're going to need multiple forms of evidence and some of the evidence might be quantitative, some might be qualitative and some might be economic. So here on the right hand side, I've just got some illustrative examples of the kinds of forms of evidence we might draw upon to evaluate the drug court.

Then at steps five and six we gather and analyse that evidence and at step 7 this is where the magic happens, where we can bring all these multiple strands of evidence together and use the rubrics to make sound, transparent judgements, ratings about each E and about value for money overall. This approach also gives us a clear structure for reporting our findings. So here we have again, illustratively the the 5E S and next to each criterion we can place a judgement meets the agreed transparent definition of excellent based on the following pieces of evidence. And here are some lessons and opportunities to improve. So terms like excellent, good, adequate and poor within the context of this evaluation, we can use those terms unapologetically because they've been carefully defined. Excuse me?

So this is the full process on one slide, and you'll see it's underpinned by 4 principles along the bottom. The first is that it's interdisciplinary. It's drawing on both insights from both evaluation and economics. Secondly, it's using mixed methods. It's at least recognising the validity of multiple forms of evidence and very often drawing on insights from multiple forms of evidence. It's underpinned by the process of evaluative reasoning that I've described, using criteria and standards to make judgements from the evidence. And it's participatory because we always need to address this question of value to whom and and ensure that those values are represented in the rubrics. The five ES I mentioned are just examples, so the criteria can can be adapted to fit to the context. So in addition to the five ES, there are other criteria out in the world.

The OECD has a set of criteria and there are many others. And criteria don't have to be specified in a top down manner from the organisations that that Commission evaluations. We also have some good examples of evaluations where we developed criteria with stakeholders from the ground up.

So these criteria are looking looking back at existing programmes to make ex post determinations. But we can also use this approach on an exam tape board looking basis to make judgements about potential programmes that have not yet been funded. And so one potential set of criteria that could be used for that is the Treasury's five case model that's used to structure business cases. And indeed that the New Zealand Treasury has something very similar that that it's using as well.

This approach can be applied to multiple levels. There are examples of whole organisational value for money or value for money of portfolios, portfolios consisting of hundreds or thousands of programmes and value for money judgements that are made across the portfolios and also of course of individual programmes.

This approach is, is getting out in the world and being used not just by ourselves but by others. And Alex and I've been providing value for money training, value for investment training workshops in London over the last couple of weeks. In a recurring thing coming out of those workshops is that many people are already using this approach, which is great to see. Just, I won't read them all out, but just to, to give you an idea of the breadth of sectors in which this approach is being used both in the International Development sphere as as well as domestically in various countries.

Why is it gaining this traction? We, we think part of the answer is that it's helping people to get clear answers to value for money questions. It's not telling people to stop doing what they already do. It's not replacing existing methods. Rather, it's adding a sense making framework to help people to define value for money in a context, to select an appropriate mix of methods, to make robust transparent judgements and to report them clearly. It can include cost benefit analysis within the mix. And we think that's really important. We think it's, it often should. However, we do come up against situations where it's not feasible for various reasons to do ACBA and this approach at least gives us a viable way through in those cases where CBA isn't possible.

A lot of this this approach has been written up in a series of Open Access resources. One of the places where you can download those resources is via my website. So I'm giving you the link there. And at this point I was. Thank you very much and hand to the next piece. Great, thank you, Julian.

So we will now hand over to Lucie Moore some reflections from her, her perspective. Brilliant. Thank you so much and thank you very much for inviting me to be here today. I'm delighted to be here. As Alex mentioned, I'm the head of the Evaluation Task Force. We're a joint unit established by the Cabinet Office and the Treasury, and we've been around since 2021. Our mission is to improve both the quality and the quantity of government evaluations and crucially, we want to ensure that these evaluations inform decision making. So our goal is simple but is very ambitious because we want to make sure that when government is making decisions about how to spend taxpayer money on policies and programmes, those decisions are based on the best possible evidence about what is good value for money.

So as Julian has said, ensuring value for money is not just about cutting costs, it's about maximising impact and it's about understanding the trade-offs and the opportunities, the opportunity costs involved in every decision. So for example, if we fund a job training programme, are we confident that it is actually improving employment outcomes Or if not, could we be, could that funding be better directed elsewhere so evaluations can help us answer those questions?

So why does evaluation matter? So as I've said, it matters because taxpayer money needs to be spent wisely. The demands on government resources are vast and they continue to grow. So whether it's addressing the cost of living crisis, investing in the NHS, tackling climate change, ensuring national security, government has to make really tough decisions about where to invest taxpayers money. And we want those decisions to be informed by the best possible evidence. So what works, what doesn't work and most importantly, how do we actually know?

So this is where evaluation comes in. So evaluation's not just an afterthought or an academic exercise, it's a crucial part of good governance. So it enables us to test whether our policies and programmes are actually delivering the intended outcomes and to help us to learn from our successes and our failures so that we can adapt when things aren't working. So by embedding evaluation and policy making, we want to make sure that public resources are spent wisely.

So the evaluation task force, we were set up to try and address a long standing challenge in government that too often decision makers are being forced to make really tough decisions with very little evidence. So our work is trying to change this. So we want to build a culture where evaluation is not just an optional nice to have for every programme, but it's an integral part of developing and implementing policies. We want every policy and programme and initiative that government does to have a clear plan for how it will be evaluated.

So this means we work with a wide range of people, we work with government departments to support them, to design evaluations upfront and ensure that they are rigorous and relevant. As of March of this year, the Evaluation Task Force had provided advice on over 350 programmes totalling 200 billion pounds of spend.

We also work really closely with the Treasury spending teams because we want to ensure that evidence is embedded into the spending review process so that funding decisions are informed by what we know works. And we have a big push on transparency and publishing evaluations because transparency and openness about what we find, including when things don't go as planned, is really critical for public trust. So we've made an evaluation registry. This is a website where one will be able to find all government evaluations in one place. And this will be publicly launched next year.

And finally, we invest in capacity and skills. So evaluation requires expertise and we want to equip civil servants with the tools they need to design and use evaluation. So we've created an evaluation Academy as Alex mentioned at the start, and the idea is to address across government skills gap. So The Academy Is made-up of 10 modules. It covers process impact and value for money. And the value for money approach covers the approach that Julian has outlined today. The academy's been run for two consecutive years and it's already led to over 2000 civil servants being trained on evaluation.

However, the challenge of improving evaluation across government is not something that the ETF, the evaluation task force. We are not going to solve this problem on our own. We need, we rely on partnerships, we rely on working with other people. So we work with policy professionals, analysts and delivery teams across government, but we also work with people outside of government, academics and external organisations, including the brilliant What Work Centres.

Collaboration is absolutely key to building the evidence base that we need to make better decisions and it's key to ensuring that not only the evidence is collected, but that is also used. So in closing, I just want to reiterate about ensuring evaluation value for money in government is not just about financial prudence. It's about improving outcomes for citizens. And it's about trust. It's about ensuring that the public knows that the money is being used responsibly and effectively. And it's about making sure that every decision we make is ground in the best possible available evidence so that we can invest in what works for those who need it the most. So thanks very much and looking forward to your questions. Thanks, Lucy.

So we've got lots of questions and we'll try our best to get through them all. So there were a few sort of processing ones. There will be just to clarify, there will be a recording of this presentation that we shared with everyone who registered afterwards. And we will also include instructions as to how to flag interest in the any, any follow up in person session on that.

So where to start? I guess we'll go from the top. There's a question here from Nikki, as well as front loading and upfront theory of change. How frequently would you suggest viewing this theory of change to test whether the initial theory of change is valid? I don't know whether, Julian, you wanna start with that and then maybe Lucie or Kirstine can come in too. The classic answer to any question about any evaluation ever. It depends. It depends on multiple factors, but I think that the principle of being able to review and update the theory of change to test and validate whether the theory holds is a really important one.

Christine or Lucy, anything to add on that? Yeah, I just agree series of change should be seen as living documents that need to be constantly updated. So it depends on what you're doing. Like it might be that you at the beginning, you do it quarterly, it might be that you move to an annual review, but it depends on how long the project and programme is that you're delivering really. Yes. And just to add on that, I mean, when I talked about mail systems and embedded mail systems earlier or monitoring, evaluation and learning systems, the learning plan should include that should specify that. So there'd be a process around it will often be annual reflection cycles, often aligned with review and reporting cycles around taking stock of the, the monitoring and internal evaluation and the learning and and then reviewing the theory of change and updating it so often it's annual doesn't have to be, but on a, on a big long term project that might make sense.

Next question from Owen Julian, Thanks for your presentation. I think we can all agree there are better ways to measure value for money than often done in government money. This may provide opportunities to influence Treasury's approach. I've certainly reached out to the to the office here, so hope to have a dialogue with them at some stage.

Thank you. Lucie, I don't know if you want to come in on that as well. I know you work closely with the Office for Value for Money. Yep, yeah, I mean, they're definitely aware of this approach. There's a couple of people from the evaluation task force on secondment into OVFM. So yeah, we're definitely sharing that learning. And I think that Julian's approach is is kind of compatible with existing HMT guidance. It's just provides kind of a process which can be followed to implement that guidance.

Great. We've got another question here. Thanks for the session. It's from Noriah. I've worked on various donor funded projects in Malawi. Some cases, in some cases evaluation should impact based on the theory of change log frame at the start. But over time there's no significant positive change on the lives of communities involved. How can we then ensure that, how can we measure value for money in the long run?

Julia, do you want to pick that up first? I think it is with any outcome or impact evaluation, it's important to pick criteria and measures that are realistic for the the time period in which we expect the the outcomes to play out. So we're looking for the right outcomes that we should realistically expect to see. That principle flows through into VFM evaluation as well. And ultimately when we're making assessments of cost effectiveness, the, the, the best way to use ACBA, if we're using it retrospectively is to be able to include the value of the long term impacts. And in order to do that, we'd have to wait long term till we've got those measurable impacts or we're doing a little bit of looking forwards and looking back and, and putting those two things together because CBA is inherently it's a modelling process. And so we, the intermediate results that we have to date, whatever those may be to be able to cast forward and make defensible assumptions and, and, and run some scenarios to suggest what the, the longer term value may be. And it's a very good tool for doing that.

And I guess, you know, the short answer to that is if they're genuinely aren't any, any sustainable longer term impacts, you know, if this starts off with a bang that Peters out, maybe it's not good value for money and that programme should stop, right? I think Lucie would agree. And I think what she said is that evaluation is about stopping things as much as starting things and allocating things where they're most effective.

We've got a question here from Barbara about the Green Book. So the guidance states that you should only use assumptions based on strong evidence and reference these evidence. So it's all transparent, but creating a robust evidence based and expensive and the normal approach when primary data gathering is expensive to use models. So, so do you have any suggestions about how we might address this very practical barrier to robust value for money evaluation? So I think that's about, you know, understanding the evidence base and plugging that into your initial economic analysis and appraisal in terms of does that that economic appraisal at the start. That's right. Those real world constraints always exist. And, and we want our value for money assessments to themselves provide value for money. And so we need to, we need to make the level of effort proportionate to the, the value we gain from the, the whole exercise. And that means that that to some, to some degree, that there may always be a missing piece of evidence that if only we had more resources, we could get better and better evidence. But we need to use the the evidence that we have and make the best use of it. And it comes back to this theme that that often that means some cost benefit analysis and some other methods together. So thinking of CBA within the context of mixed methods, rather than assuming that we all should rely on CBA alone.

And just to add on that, I think, you know, often I think the Green Book will, you know, an, an economic appraisal will be done and assumptions will be made and often those will be referenced and evidenced better than other assumptions. But one of the recurring themes of the sessions we've had over the last couple of weeks is that if there's a stronger link between the economic appraisal being explicit about the assumptions and then the evaluation and what the evaluation is assessing and testing. Then we can build a virtuous cycle whereby the evaluation can feed into the next economic with real numbers from a real impact evaluation that can plug into the next phase of a programme or a related programme in a, in, in a, in a, in the sector. So I think that all aligns very closely with what Lucy outlined. I don't know if Lucy wants to come in on that as well. Sure. Yeah, I think the question was asking a bit about the like the

cost involved in generating evidence and and how do you justify that. And I think I would just say so in the in UK government we spend a trillion pounds a year on public services. So spend a tiny fraction of that on evaluation. And if you can show that just, you know, if you can make improvements to a billion, you know, multi billion programme and you can find some savings and a more efficient way to do something, or you can show that something's not not good value for money and is a better way to achieve that outcome. The money to save from that can pay for evaluations for the rest of time.

Can I also just come in on a slightly different point? I wholeheartedly agree with what Lucie just said as well. It's about like making the case for the fact that you need the evidence to show what is and isn't working. But I think the other thing to think about as well, we talk a lot about assumptions. But the other thing, both with the last question and with this one to think about is like, what is why is it that we think that the activities are going to lead to the outcome? So when you're thinking about what you're going to measure, like like what what do we know about what doesn't doesn't work in terms of the how and the mechanisms and the like? Why is it that these activities that you're proposing like that you, that you what, what evidence can you use to justify that these activities that you're proposing will lead to the outcomes? And sometimes we spend a lot of time focusing on input and output kind of style evaluations. But there is definitely room for that sort of what, you know, what people often call the kind of unpicking that black box. But like having that real understanding of like, well, OK, fine. But why? Why are we, why is it that this is going to lead to that? And that can help you build the sort of justifications and the evidence base for it as well, I think. And it's sometimes not thought about sufficiently.

I think we've got a question, another question I think related to this concept of front loading some of this thinking about when you would bring in some of the the thinking about the value proposition, Julian. So at what point could you bring that in and as early as possible? I, I always think that's the, the ideal situation is when the evaluator is, is part of the, the programme design team from the very outset. So that that's everything from the programme design to the business case appraisal to, to, you know, designing the monitoring, evaluation and learning system can be, can be all part of one integrated whole activity and, and defining the value proposition as as just one piece of that. So I mean a specific, is it good to highlight the value chain, you know that slightly zoomed out version very early on and build on that with cost benefit analysis after approval or would it even be part of putting that all together, part of the process?

One of the big reasons that we put that eight step process together is that we can just keep coming back to it and say using it to navigate. And so yeah, I would say to just come back to the 8 steps. And, and at step one, we've got theory of change, we've got value proposition. Those, those items are there right at the outset because it's about working with stakeholders to develop a shared understanding and a shared language for talking about the the programme.

There's a few specific questions for you, Lucy, the one around if The Academy Is open to local government and one around if you could elaborate on the links between your task force and the spending reviews. Sure. So the Academy at the moment is only open to ministerial departments that we would love to open up further to arms lengths, bodies and to local government. It's definitely, there's definitely demand for that. So yeah, it's definitely something we are trying to achieve. So watch the space and then the links between US and spent. Was it the spending review process? Is that the question? So we kind of see ourselves as the evaluation advisors for the treasury spending teams. So the treasury spending teams have the unenviable task of trying to divvy up government budgets, but we see ourselves as providing them with advice on the quality of evaluation plans or the sorry, the quality of like underpinning evidence. So, you know, a department says we want to spend money on this particular programme, he is at the evidence that supports it and we can look at that evidence and say Yep, that's really strong evidence. We think this programme is very likely to work.

We will, you know, the contexts are comparable etcetera. And then we also advise spending teams, if we are going to invest in stuff which does not have a strong evidence base, which is the vast majority of government spending, then we say, OK, well, like if we're going to press green on spending this money, let's make sure that as we do it, we are collecting evidence on whether it is effective or not or good value for money.

Time for it, Few more questions. So there's one here, Evaluation and the nature of this framework is complex. How do we communicate or distil the complexity to senior decision makers who just want one singular return and they want it as early as possible? Julian, do you want to come in on that? And maybe Kirsten as well, from your perspective, one of the benefits of this approach and I showed on one of my slides is that we can cut straight to the chase and answer the value for money question. We can say this programme is good value for money or this programme is excellent value for money and we can explain exactly why. So in a similar way to, you know, the way the number that pops out of a cost benefit analysis has a lot of cut through. So the benefit cost ratio could say something. For every pound we spend in this programme, we create two 2 1/2 lbs of social value. Well, similarly, the VFI approach can provide something that's got similar cut through by saying this programme has good value for money and then to unpack the evidence and the reasoning behind that.

Great. There's one here, investment decision. Oh, sorry, Can I come in on that? Sorry, I was waiting for you. No, I'm I, yeah, Just to say, I think in terms of communicating evaluation findings, I think this is a massive weakness for the evaluation sector. I think we all could get a lot better at communicating the end of the evaluation. I think we're getting much better at communicating the starting point, which is the justification, the importance of evaluation. But this is something that we at the UK Evaluation Society are really starting to try to focus on that kind of communication of findings, like making sure that we're actually able to. Like, I think the problem with evaluators is we like to write