The use of evidence in the policy cycle

Understanding a problem and its causes

The policy cycle begins with the identification and formulation of a problem that requires attention. Ideally, this step should be preceded or accompanied by a formal prioritisation process that considers the answers to the following questions:

The answers to the above questions will lead to the description of the problem, which can look different depending on the perspective. The understanding of the problem should therefore be underpinned by research evidence, the decision-making context, ethical considerations and discussions with key stakeholders.

The most difficult questions to answer in the diagnose stage are the latter ones, which is why we present potential solutions based on two case studies below. The first example illustrates the temporal challenges in the agenda-setting phase based on the election campaign decisions made by political parties. It is based on the experience of Verian in Germany. 

How long will a priority last in a constantly changing world? When to collect what type of evidence? When is evidence outdated?

Before elections, political parties have to make decisions about the candidates and the issues on which they want to base their campaign. 

The party committees usually opt for an evidence-based briefing before making their decisions and briefing the communications agency. 

The biggest challenge is the timing: when to collect what kind of evidence? 

Political moods can be very volatile. An early start allows for an early briefing with enough time to develop a good campaign. But it also carries the risk that many things can happen that change the perception and the impact of a campaign. The solution is to separate strategic and tactical objectives. 

Typical strategic research questions are: 

• Who are the accessible voter segments among eligible voters?
• Which topics and narratives mobilise potential voters and which topics demobilise political opponents?
• Where is our party vulnerable?

Typical tactical research questions are:
• What is the central campaign message in the final election camp ign phase that has the best potential to
maximise votes?
• How to deal with postal voters who might vote up to six weeks before the election date?

Evidence for strategic questions is usually collected 6 to 12 months before election day. Evidence for tactical questions is gathered about 4 to 12 weeks before election day. However, if unexpected events occur (e.g. a new conflict or a new opposing candidate) it is advisable to update the strategy based on new evidence collected after such significant events.

The second example illustrates the importance of using holistic diagnostic tools that incorporate multiple perspectives to identify the root causes of long-term social problems. Only then is it possible to identify levers for changing trends. It is based on the experience of Verian in Australia.

How can causes of a problem be identified, especially when they are not obvious?

Violence against women has a long tradition in Australia. Two out of five Australian women have experienced violence by a man after the age of 15. Every week, one or two women were killed by a current or former partner. The consequences of violence against women have cost the Australian economy 13.6 billion AUD annually. No wonder, the Australian government made reducing violence against women a national public health priority in 2015.

While there was extensive existing research on the subject, the vast majority had focussed on identifying the scale of the problem and the rational, conscious attitudes that can affect it. 

However, there was a lack of understanding of the core of the problem and its starting point, and therefore no idea of what could be done to reduce the violence. 

New research revealed that social norms and unconscious beliefs among adults and young people were promoting gender inequality, disrespect and aggressive behaviour. For example, there was a tendency towards victim blaming, when examples of aggressive or disrespectful behaviour between men and women were brought up (‘she must have done something’); as well as empathising with males (‘it’s very hard being a young boy, he probably just likes her’) and trivialising behaviour (it’s not that bad … it’s not like she ended up in hospital’). Due to the unconscious  nature of these engrained heuristics, many Australians become reconciled to their role in perpetuating the situation and easily dismiss responsibility.

This is a result of the perceived high cost of exerting influe ce and, as a result, many are quick to accept not to get involved when disrespectful behaviour between young men and women occurs. It is a common norm to view this experience as an important part of youth development. Influencers need to feel more confidence and self efficacy in order to respond appropriately.

The result of the initial evidence base was the development of a long-term national behaviour change campaign aimed at challenging and changing the engrained attitudes and beliefs at a societal level over time. The ‘Stop it at the Start’ campaign is a long-term investment, commensurate with the scale and complexity of the issue – because changing deep-rooted societal attitudes does not happen overnight. It is a primary prevention  campaign. Since its launch, it has been independently evaluated and is recognised as the most successful campaign ever launched by the Australian Government.

It has led to ‘new conversations’ about the links between disrespect and violence against women, a significant  reduction in the unconscious heuristics that unintentionally perpetuate the problem, and a significant reduction in violence-supportive attitudes among young people. And ultimately, the average number of Australian women  killed by a current or former partner has decreased significant y from one or two every week (2015) to one every 11 days (2022). 

What kind of evidence helped to uncover the cause of the problem?

Qualitative research to deconstruct the crux of the issue: to understand why, when most Australians condemned violence against women and children, the problem persisted in a pervasive way in society. Advanced qualitative techniques were utilised in combination with Verian’s proprietary Behaviour Change Framework.

Verian collected the perspectives of many different groups in different settings: Focus groups were conducted with mothers, fathers, parents together, older and younger siblings, grandparents and non-parent influencers. In-depth-interviews took place with more vulnerable target groups such as people with disabilities, members of the LGBTQIA community, and survivors of domestic violence.

Crafting and selecting solutions

Once the problem and its causes have been identified, the next tep is to develop and select the best solution for this problem. 

Depending on the nature of the problem and the degree of novelty, we may need an exploratory crafting phase for new solutions or a literature review for identifying best practice solutions. In  most cases, there will be a variety of possible solutions which usually require evidence to show which solution offers the highest impact and/or the best cost-benefit ratio. The following questions are typical of decision-making in this phase of the policy cycle: 

The first example below illustrates the challenges in the desig phase when searching for potential solutions. It is based on the experience of the Swedish National Council for Crime Prevention and Malmö University in Sweden, who accompanied the design phase with a process evaluation published in 2020.

How can we generate solutions for addressing the problem? How to adapt something that worked elsewhere?

Serious violent crime is one of the major societal problems in Sweden. The main source of this violence appears to be within and between criminal groups and is related to fire rms. Swedish research on what works to reduce serious violence is limited, and more knowledge is needed to provide the police and other crime prevention actors with the tools they need to reverse the trend.

To identify potential strategies for combating lethal violence, the Swedish National Council for Crime Prevention (Brå) looked at regions around the world that had previously tackled lethal violence. They found that a strategy called ‘Group Violence Interventions’ (GVI) had been successfully applied in Boston and other American cities in the 1990s.

To learn more about the requirements and practical implementation of this GVI strategy, representatives from Brå conducted a study visit to the United States and consulted with Professor David M. Kennedy, who led the Boston Gun Project in the 1990s and had become the Director of the National Network for Safe Communities in the United States.

The GVI policy that worked so well in Boston was developed by an interdisciplinary working group in a step-bystep process that helped the team identify and effectively solve problems that they may encounter (problemsolving model approach). The working group consisted of frontline practitioners from a variety of agencies as well as academics. Together, they combined qualitative and quantitative research with the experience of police officers and streetworkers.

The GVI approach sought to deter gang violence by convincing gang members that violence and gun use would have consequences, causing them to choose to change their behaviour. A key element of the strategy was to deliver a direct and explicit ‘deterrence message’ to a relatively small target audience, which spelled out the response to certain behaviours. Boston Police  Department made engaging in gang-related violence more costly for those involved. Communicating what had happened to others in the target audience was expected to deter further gang-related violence in the city.

After the successful implementation of GVI in Boston, the strategy has been implemented mainly in American cities, but also in a few European cities such as Glasgow and London. The conditions for being able to implement GVI in Sweden have been questioned with reference to the fact that the differences between the United States and Sweden are too great.

Therefore, it was decided that a pilot project would be set up in one Swedish city. This ‘real world’ trial was accompanied by a process and impact evaluation to understand the elements required to adapt such a policy to different context/areas. Malmö municipality became a pilot project for GVI in Sweden for two years. The American team was consulted for its knowledge based on its role in Boston’s previous GVI work. The impact evaluation of this trial, published in 2021, showed that the number of shootings in Malmö decreased during the period of the pilot project. However, it is difficult to ascert in statistically what factors are behind the reduction. Nevertheless, the results were promising enough for more Swedish cities to adopt and adapt this policy in their areas. 

Fine-tuning the implementation of a solution

Once the decision has been made as to which solution should be implemented for the problem, the next phase involves fine-tuning decisions during the implementation phase, including plans for communications. This phase is often neglected by political decision-makers, as they usually focus on the macro level of a political solution. However, the success and effectiveness of a solution often depends on well-designed implementation and communication in practice. To ensure that the policy works as efficiently and as effectively as possible, it is important that the implementation concept is carefully scrutinised and tested, ideally with all relevant intermediaries and target groups. Typical research questions at this stage are:

The first example below shows how resistance to policies and programmes can be dealt with through communication strategies. It is based on the experience of Verian in the Netherlands.

What could cause the measure to fail during implementation? How can a policy be fine-tuned with additional insights for a communications strategy?

The Netherlands is known as the “land of bicycles” and has the highest rate of bicycle ownership in the world. At the same time, the country has experienced a notable rise in cyclist casualties. A key factor in improving safety is the usage of bicycle helmets, but very few people in the Netherlands wear helmets during regular bike rides.

In response, the Ministry of Infrastructure and Water Management wanted to develop a comprehensive bicycle safety plan to reduce the number of serious injuries among cyclists. However, in order to be picked up by citizens who seemed predominantly opposed to helmet usage, research into what was driving attitudes, beliefs and behaviours of Dutch citizens on voluntary helmet usage for regular bike rides was required. 

Only with the knowledge of key driving factors of resistance against helmet usage can effective communications and implementation measures be designed and policy measures on helmet use actually pursued by citizens. As influencing factors can be hard to pin down for a topic whe e the prevailing social norm is so strong, Verian designed a two-stage research project. Interviews with parents of young children, commuters and seniors who cycle regularly uncovered that the main barriers to helmet usage were a low-risk perception, and the belief that helmets are unnecessary and affect their image negatively. 

Key barriers included social norms, stereotypes, appearance issues and inconvenience. However, the research also showed that acceptance could be increased if helmet use becomes more  common, risk perception rises, design and comfort improve, and personal values like responsibility and health consciousness become more associated with helmet use.

The quantitative survey showed that a considerable sub-section of non-helmet-wearers would be open to wearing a helmet in the future. To target those cyclists and others, the Ministry developed a communications strategy to strengthen voluntary helmet use among cyclists centred on those more implicit, emotional factors that do not only take rational cost-benefit calculations into account.

The research prompted the Ministry to adopt a holistic view of citizen decision-making when it comes to helmet use and ensure that emotional resistance does not override rational choice

The next example illustrates the challenge of ensuring stakeholder buy-in in the implementation phase. It is based on the experience of Ramboll in Germany.

How to ensure participation of intermediaries and target groups over a longer implementation period?

To improve teacher education during the course of study, the Federal Ministry of Education and Research (BMBF) launched a major funding and development program “Qualitätsoffensive Lehrerbildung”. The aim of the program was to optimise education structures at universities, develop relevant support services for students and ensure comparability of studies and qualifications across the federal tates. The federal ministry commissioned Ramboll Management Consulting and the University of Linz to carry out a comprehensive evaluation in order to assess the impact of this large-scale program.

The program involved 72 universities across all federal states of Germany with 91 very heterogeneous projects. Therefore, a large and diverse group of stakeholders had to be involved and motivated to participate over a relatively long evaluation period. The evaluators relied on stakeholder cooperation, and the program also aimed to achieve cross-state cooperation in order to ensure impact and comparability.

To maintain the willingness of all stakeholders to participate in data collection over the entire period (2016 – 2024), Ramboll set up a large-scale evaluation programme including online surveys for all projects, monitoring systems, expert interviews, interviews with funded and non-funded universities and group discussions with representatives of the education and science ministries of the federal states, as well as observation at events and document analysis. 

This wide variety of data sources ensured that all relevant data could be collected without putting too much burden on individual participants and that each participant was approached with a suitable research method. Very heterogeneous starting positions at the universities were considered through more extensive and less standardised monitoring instruments (e. g. more open response options).

The support of representatives of the federal states and a scientific advisory board underpinned the expertise  and authority of the researchers. Moreover, the transparency of the evaluation results, for example through annual thematic interim reports and workshops to discuss interim results, played a key role in engaging stakeholders by increasing trust and already providing some benefits to universities.

These measures ensured that the “Qualitätsoffensive Lehrerbildung” supported reforms and initiated new approaches to teacher education, as well as boosting research in teacher education. The projects and stakeholders were connected nationwide and supported in their work. The “Qualitätsoffensive Lehrerbildung” helped the universities to better manage the need for coordination and to become more resilient through networking.

The next example illustrates scaling challenges in the implementation phase after a successful pilot. The case study is based on the experience of testing simple behavioural interventions by the New South Wales (NSW) Behavioural Insights Unit to reduce the gender gap in recruitment for the NSW Government in Australia.

How to scale interventions that have been found to be effective during a pilot?

Since 2015, one of the Premier’s priorities was to increase the proportion of women in senior leadership roles in the NSW Government sector from 33% to 50% by 2025. A key challenge was that women do not apply for senior leadership roles at the same rate that men do (this was uncovered in the Diagnose phase). 

The Behavioural Insights team developed two simple, low-cost interventions designed to encourage women to reapply for senior roles after they narrowly missed out. The interventions were an email and a phone call with a recruitment manager that focused on how well the applicant had done and encouraged them to apply again. Both interventions leverage the behavioural tendency to judge an experience based on how we feel during its emotional peak and at its end. During the recruitment process, the emotional peak is often an interview, while the end for most candidates tends to be a generic email advising that their application was not successful. The intervention aimed to disrupt this final impression by prov ding candidates who were interviewed but not successful with an encouraging email or phone call. 

The interventions were tested in a pilot among a group of public sector agencies in 2019/2020. A randomisedcontrolled trial design with 1614 candidates was implemented. These were randomly allocated to receive either a ‘business as usual’ recruitment process or an encouraging email or an encouraging email plus a phone call.

Two weeks after receiving the intervention (or not), candidates were sent a survey to record their overall satisfaction with the recruitment process, intention to reapply and likelihood to refer a colleague to the NSW Government. Moreover, it was monitored whether candidates reapplied for another senior role in the NSW Government within three and six months later.

The pilot results clearly showed that both interventions were successful. They had a significant impact not only on the intention but also on the behaviour to reapply for senior roles. Moreover, the impact was significantly larger for women than men, so that the gender gap was significantly re uced to 12% (email only) and to 4% (phone and email) compared to the control group with a gender gap of 45%. 

It was calculated that if the email and phone call were scaled across the entire NSW Government, it could increase the proportion of women in senior roles by up to 0.5% each year and make a significant contribution to achieving equal gender representation in senior roles. Therefore, the Public Service Commission developed a toolkit to help recruitment managers implement the interventions in all clusters. Since then, gender diversity on NSW Government boards has further improved, although it has not yet been evaluated whether this progress is based on the roll-out of this treatment.

Evaluating the output, outcome and impact of a solution

Without any monitoring and evaluation after the solution has been implemented, you do not know whether and how the solution has worked. Collecting evidence during the implementation process will help as an early warning system if something goes wrong and needs to be corrected. Impact evaluations after implementation will give you the numbers that you may need to tell a success story, to understand whether it was value for money and to inform future policy decisions. The following research questions are typically asked at this stage:

The first example below illustrates the challenge of designing comprehensive and useful research programmes in the evaluation phase. It is based on the experience of Verian in Germany.

Has the selected option achieved the desired objectives?

The goal of the German education program “Bildungsprämie” was to increase participation in vocational training, especially for those who were hindered by financial barriers. y offering financial support, the aim was to make training affordable for those on low incomes, addressing both cost and other barriers, such as limited time and skills. 

To assess whether these effects were achieved, a research consortium consisting of Verian, Institut für angewandte Wirtschaftsforschung e.V and the Forschungsinstitut Betriebliche Bildung (f-bb) conducted a large-scale evaluation study with a holistic view of research aspects and questions.

While the evaluation focused on the standard question on effectiveness and goal achievement – “Did the programme increase participation in vocational training?” – a more comprehensive view was taken and also considered:

• Analysis and assessment of economic viability
• Analysis and assessment of educational success
• Identification of improvement potentials or fault controls in the process
• Development of actions and control measures for the federal program

The research also aimed to evaluate any long-term impact and insights for future training promotion programmes. In doing so, the research took a broader view and not only measured the continuing education and professional development behaviours and attitudes of participants, but also the broader relevance of subsidised training and its effect on social compensation, mobilisation and the reduction of unequal education participation. 

Consequently, it considered the embedding of the programme in the whole social context and further relevant developments. This broad focus helped to identify a range of possibilities for  improvement and for a sustained demand-oriented promotion of professional development, but also revealed limitations of the programme’s effectiveness. It showed that participants were more likely to attend higher quality training or additional training, and it especially seemed to encourage lower-income individuals. Also, the participants reported high satisfaction with the “Bildungsprämie”. 

However, further analysis showed that possible effects of the education premium seemed to be due to selection bias, with only more motivated participants taking part in the first place. The impact on economic and career advancement also was modest. Considering economic viability, it turned out that the programme made individual funding very expensive.

A well-designed evaluation study with diverse and extensive research questions therefore managed to produce a well-balanced and nuanced assessment of its overall effects.

The next example shows how monitoring systems can be useful in the evaluate phase to keep track of trends and new developments. It is based on the experience of Verian in the Netherlands.

How can you use monitoring & evaluation data to continually improve and engage and achieve sustainable change?

The Dutch National Sports Federation (NOC*NSF) aims to signific ntly increase sports participation in the Netherlands. To make sure all programmes are based on evidence and any new strategies are evaluated immediately, it commissioned Verian to conduct a monthly monitoring survey targeting a representative sample of Dutch citizens. 

Each month, the survey measures sports participation and levels of physical activity, as well as the most commonly practised sports and the demographic make-up of Dutch sports participants. It relies on a standardised set of questions based on a validated methodology to assess sports participation and physical activity. It can therefore produce a key indicator for sports participation that can be compared over time and across different contexts. As the data is collected monthly, a granular picture of trends and patterns emerges that allows for precise policy adaption. 

Moreover, the large sample sizes amassed over this period allow for detailed statistical analysis and sociodemographic comparisons. The monitoring survey also allows for ad hoc questions and special focus areas to obtain targeted data on emerging trends, seasonal activities or other particular areas of interest. The monitor therefore combines a standardised base module producing reliable and consistent outputs with the opportunity to flexibly adapt strategies and react to emerging opportunities in sports. 

A fixed data pipeline feeds data into easy-to-access dashboards just days after fieldwork has ended and makes any insights available to use immediately. Being very fle ible, the dashboard incorporates feedback from previous months’ data to make sure it is always up to date and produces timely and correct results. In general, the monitor is the basis for policy documents from NOC*NSF and the formulation of long-term strategies targeting the barriers identified in the survey for a diverse range of subsets of Dutch citizens.