What does it take to be a good evaluator?

Many fellow evaluators tell me that when they go to a dinner party, they struggle to explain what they do. At times like this we may feel that our role is complicated to explain in comparison to traditional professions, e.g., a doctor or a teacher.

Oversimplifying our role to avoid confusion is usually the best approach with dinner party guests. “Hi my name’s Kizzy. I evaluate programs to find out if they are effective.” However, with our peers we should recognise the complexity of what we do because, when we produce good work, we inform policy and program decisions that improve people’s lives. Therefore, no matter our career stage we should regularly ask ourselves “what does it take to be a good evaluator?”

Maybe you’re thinking about starting a job in evaluation for the first time, or maybe you fell into evaluation and you’re not sure whether it’s for you. You might already be a competent evaluator, but you want to keep improving. Something that’s been on my mind is how evaluators will add value when generative artificial intelligence, like ChatGPT, can do much of an evaluator’s job.

In all of these cases, understanding what it takes to be a good evaluator can help you plan your career, seek out relevant skill development opportunities, make a positive social impact, and improve your job satisfaction. You’ll also be a better colleague and collaborator if you’re accountable for your own performance. At an organisational level, when leaders understand what it takes to be a good evaluator, they can develop an effective workforce strategy to enable evaluators to succeed.

1. Evaluation skills

Evaluators need a multitude of technical and soft skills. In the Australian evaluation team at Kantar Public, we use a capability wheel to help us focus our learning and development on the most important skills. The wheel includes four areas of technical skills: evaluation methods, project management, writing, and commercial acumen. It also includes two areas of soft skills: problem solving and relationship management.

2. Evaluation mindset

For most of our projects, we find that skills alone are not sufficient to deliver a successful evaluation. In a complex and changing delivery context, what it takes to be a good evaluator is also the mindset to prioritise what matters, be prepared, and stay positive. Let me explain with an example.

A member of our team led an evaluation of a program that is run in primary schools. The organisation that ran the program was under pressure to explain the value of their program after a previous evaluation took a very narrow cookie-cutter approach, examined outcomes beyond the program’s control, and concluded it wasn’t effective.

Our evaluator co-designed a Theory of Change and Key Evaluation Questions with the organisation. She considered:

• What outcomes are strategically important to measure given the organisation’s USP (unique selling point)?
• What outcomes are realistic and practical to observe given the design of the program and the timeframe for the evaluation?
• What operational issues could influence these outcomes and should therefore be accounted for in the evaluation?

This led our evaluator to propose a theory-based evaluation which examined how the unique elements of the program generated benefits for participants, and how the program could be adapted in the future to maximise these benefits.

The implementation of the evaluation coincided with the COVID-19 pandemic, causing unanticipated barriers to collecting data in schools. New approaches to participant recruitment had to be developed because of low take-up, and the evaluation design had to be changed to accommodate a smaller sample size. This in turn required further last-minute co-design workshops and user-testing to ensure the changes would be fit for purpose and meet stakeholder expectations.

Throughout the implementation period, our evaluator had the following mindset:

• Prioritise what matters: she communicated openly and regularly with everyone involved in the evaluation to design an approach that would deliver meaningful findings.
• Be prepared: she started with the end in mind (how the evaluation would be used), tested assumptions, and had a backup plan to efficiently adjust the implementation to the changing delivery context.
• Stay positive: pain is inevitable when implementing evaluations but suffering is optional, so she didn’t let setbacks get her down which helped our client stay positive too.

We are pleased to say that despite the limitations, the evaluation delivered new insights about the benefits of the program, which helped steer operational discussions about how the program could be developed in the future. Those involved in the evaluation said they valued our evaluator’s focus on solutions, her perseverance, and positive attitude. Because of her mindset, she also increased her skills in all areas of the capability wheel and felt a strong sense of achievement, boosting her job satisfaction and renewing her passion for evaluation work.

3. Fostering the right skills and mindset

At Kantar Public we know that this combination of skills and mindset, functioning in a virtuous circle, is what it takes to be a good evaluator. While AI models like ChatGPT can write a decent evaluation plan and analyse simple data, dealing with uncertainty, ambiguity, and adaptation with empathy for stakeholders is what sets apart a good human evaluator.

We also know that evaluators won’t develop the necessary skills and mindset to be successful without organisational support. Our workforce strategy integrates behavioural science and evaluation to ensure we create a culture that nudges people towards the right mindset and aligns with our capability wheel.

Wherever you are in your evaluation career, or if you lead an evaluation organisation, the first step to success is being clear about what good looks like. We hope this article brings you a little closer.

This article was issued under our former global brand name: Kantar Public.