Applied learning to build evidence

How do you “teach a man to fish” when it comes to evaluation? There’s a lot of scepticism about whether frontline service personnel can take responsibility for evaluation given the technical capabilities required. Furthermore, evaluators will typically say that workshops alone rarely lead to sustained evaluation practice.

At Kantar Public, we have a different story to tell. Last year we attended a symposium in Melbourne where we watched our training graduates present their own evaluation findings. They developed a Theory of Change and Evaluation Plan, designed and conducted surveys and interviews, ran t-tests and other analyses, and used data visualisation to engage the audience in their results.

These 20 individuals came from seven not-for-profit organisations that provide support services to family and friends of people who use alcohol and other drugs. We felt like proud parents because they had limited or no evaluation experience before we trained them, and most were program managers, clinicians, and administrators – not evaluators!

By attending our ‘Evaluation Bootcamp’ program they became evaluators.

What is Evaluation Bootcamp?

Evaluation Bootcamp is a program designed by Kantar Public to help organisations build evaluation capability, with the long-term aim to build evidence. Participants from diverse organisations are guided in real-time to design and deliver a real-world evaluation that meets their own learning goals and organisational needs. Participants advance and embed their skills in all key evaluation steps, regardless of whether they are starting with zero, some, or moderate evaluation expertise.

In our experience, the factors that typically prevent the translation of training into practice are lack of time, motivation, and a supportive environment. The program overcomes these barriers with three key approaches:

1. ‘Learning-by-doing’: Participants are guided to choose an appropriate evaluation design and plan for delivery within the Bootcamp timeline, with buffer to make calculated mistakes and change course. We provide a psychological safety-net with regular technical support and ad-hoc troubleshooting.

2. Project-based: Conducting an evaluation that solves an organisational problem motivates participants to develop the relevant systems and processes in their own delivery setting for ongoing use (e.g., workflow, data management, and ICT systems; and stakeholder engagement, knowledge management, and continuous professional learning processes).

3. Peer-to-peer learning: We recommend 1-3 representatives per organisation attend, with up to 10 organisations participating. This allows for small group work, break-out sessions, and sharing sessions so that participants can learn from (and be inspired by) their peers as well as facilitators.

Bootcamp delivers evidence efficiently

Beyond the obvious benefits to the individuals and organisations participating in the program, such as developing the skills and tools to report their impact to funders, Bootcamp is often a strategic investment for peak bodies working in the evidence space. The Alcohol and Drug Foundation funded the most recent Bootcamp we delivered, to address a knowledge gap in the published literature (and within the sector) about what works to support family and friends of people who use alcohol and other drugs.

In any area of science, the strongest evidence comes from multiple studies. Bootcamp enables multiple organisations to evaluate their own service at the same time to rapidly and cost-efficiently build an evidence base around a specific topic. In addition, by having the skills and tools in place to continue evaluating in the future, the initial investment keeps building knowledge over time.

Holding a symposium at the end of the training program to share learnings and normalise the challenges of evaluation generates new solutions to measurement and impact assessment, and promotes a learning culture throughout the entire sector.

Below is an overview of the program, showing delivery over 11 months. It includes 4x 4-hour interactive group training sessions, applied homework, 1:1 consultations and check-ins, and quality assurance on analysis and reporting. The content can be reduced or extended, and the timeline can be compressed or lengthened, depending on participant needs.

If Bootcamp is something your organisation is interested in participating in or funding, please get in touch with Kantar Public Australia.

This article was issued under our former global brand name: Kantar Public.