Challenge
Decades of research show that children who enter the care system in the UK experience worse outcomes later in life than those who do not.1 Some of the most common reasons children enter care include parental mental health problems, substance misuse, and domestic abuse within the household.
In response to concerns that children’s social workers lacked the resources to meaningfully identify and address these issues, the Family Safeguarding Model was developed by Sue Williams, then Assistant Director of Children’s Services at Hertfordshire County Council. The model aims to tackle the key issues that bring children and families to the attention of services by setting up multi-disciplinary and multi-agency teams to provide wider support. The model aims to safely prevent children from entering care by supporting parents at home.
Early evidence from local authorities adopting the model suggested positive outcomes.2 However, there has not yet been a robust evaluation of its long-term impacts involving the full sample of local authorities delivering the model. Verian has been funded by Foundations – What Works Centre for Children & Families, to conduct a full-scale evaluation of the model, to complement their own evaluation research of the model. This work builds on Verian’s previous experience evaluating interventions to support children and families, including the five-year evaluation of The Families First for Children Pathfinder and Family Network Pilot schemes.
Approach
The evaluation will estimate the impact of the Family Safeguarding Model on outcomes including the likelihood of children known to the local authority to enter care or being re-referred to children’s social care. The impact evaluation will use administrative data provided by the Department for Education (DfE). We are also conducting a full implementation and process evaluation, and a cost evaluation, to understand how the model is delivered and its potential economic benefits.
Since its introduction in Hertfordshire in 2015, the Family Safeguarding model has been adopted by 23 local authorities over the past decade. Since adoption across local authorities was not randomised, we are using quasi-experimental methods to estimate the impact of the model. These methods are used in policy evaluation to estimate causal effects when random allocation to an intervention is not possible. For example, local authorities may not be randomly assigned to implement a programme, instead adopting it based on local needs, readiness, or eligibility criteria. When the underlying assumptions of quasi-experimental methods are met and supported by rigorous sensitivity analyses and robustness checks, they can provide credible estimates of causal effects.
The quasi-experimental method we are using is called difference-in-differences, as used in one of our previous evaluations.3 This approach compares changes in outcomes over time between local authorities that adopted the model and those that did not. Specifically, it considers the difference between the change in outcomes from before to after adoption for local authorities that adopted the model, compared with the corresponding change in outcomes for local authorities that did not adopt the model. This difference between the changes in outcomes, or the difference-in-differences, is the key estimate of impact for the policy, programme or, in this case, model.
The key assumption underlying this method is that the change in outcomes for local authorities that did not adopt the model provides a reasonable estimate of what would have happened in Family Safeguarding local authorities, had they not adopted the model. This is known as the parallel trends assumption. The risk of this assumption not being met is smaller if trends in outcomes are similar between the two groups before the model was introduced. To estimate the impact of the model across key outcomes of interest, we will use the Children in Need and Children Looked After datasets, which form part of DfE's National Pupil Database.
Earlier this year, we published our evaluation protocol.4 It sets out in detail how we will conduct the difference-in-differences analysis. Importantly, pre-registering the analytical steps we will use for the impact analysis reduces the risk of ‘p-hacking’, where data or methodological decisions are manipulated or selectively reported to produce favourable or spurious statistically significant findings. Pre-registration of impact evaluations contributes to more reliable and reproducible findings and ultimately supports stronger evidence for decision-making, whether an evaluation uses an RCT or quasi-experimental design.
Impact
As we enter the analysis and reporting stage of the evaluation, we are following our pre-registered plan for the difference-in-differences analysis and are continuing to collect qualitative and quantitative data across the evaluation strands. The findings and lessons from the evaluation will help shape how local authorities and their safeguarding partners in England deliver children’s services in the future.
It has been an amazing opportunity for all of us at Verian to be part of this important evaluation, working alongside our colleagues at Foundations and the Centre for Family Safeguarding Practice.
Keep an eye out for our evaluation report, which will be published next year!
References
1 See findings across these studies: https://www.adruk.org/our-mission/our-impact/article/the-lifelong-health-and-wellbeing-trajectories-of-people-who-have-been-in-care/, https://learning.nspcc.org.uk/media/mjrls2ci/statistics-briefing-children-care.pdf
2 https://www.gov.uk/government/publications/family-safeguarding-hertfordshire-an-evaluation, https://foundations.org.uk/our-work/publications/strengthening-families-protecting-children-family-safeguarding/
3 The protocol for the previous evaluation can be accessed here.
4 Also registered to the Open Science Framework - https://osf.io/yeqxs/overview
Back
