This blog is the second in a series of short reflections on quantitative practice by Ciaran McDonald. The first argued that good quantitative evaluation is not defined by methodological sophistication, but by disciplined thinking that connects questions, data and claims. This piece looks at one of the most common places where that connection starts to break down: survey design.

Surveys are often the first quantitative method people reach for in evaluation. They are familiar, can be relatively quick to deploy, produce data that looks clean and easy to analyse, and often feel like a natural way of gathering evidence at scale.

However, a survey can be clear, carefully formatted and generate data that is straightforward to analyse – and still not do very much to answer the evaluation question.

That is because the main problem is not always poor wording, a weak response scale or a badly formatted question. Quite often, it begins earlier: the evaluation question has not been translated clearly enough into what we need to measure, so the survey ends up collecting information that is related to the topic, but not especially helpful for answering the question that matters.

In other words:

A survey does not answer an evaluation question just because it contains relevant-sounding questions.

It does so only when there is a clear line of sight between:

• What we are trying to understand
• What we decide to measure
• What we ask respondents
• What we hope to conclude from their answers

The problem often starts before the first question is written

Survey design is sometimes treated as a technical exercise that begins with drafting questions: keep items clear and concise, avoid leading language, choose sensible response options, and do not make the survey too long.

All of that matters. But before any of it, there is a more fundamental question to consider:

What exactly does this survey need to help us understand?

If that is not clear, it is easy for a survey instrument to become a collection of plausible questions rather than a purposeful piece of evaluation design. Questions get added because they sound relevant, because they have appeared in previous surveys, or because someone thinks the answer might come in useful later.

The result can be a survey that looks comprehensive, but does not have a strong enough connection to the evaluation itself. It gathers data, yes – but not necessarily the data our evaluation needs.

This is where the connection between questions, data and claims begins to break down.

A good survey is not one that asks a lot of reasonable questions. It is one that generates data capable of helping answer the evaluation question.

From evaluation question to survey item

A key discipline in survey design is to make the route from evaluation question to survey item explicit:

Evaluation question → outcome or concept → indicator → survey item

Each step does a different job, and so each step matters.

Imagine an evaluation asking:

To what extent has the programme improved young people’s feelings of safety in their local area?

The outcome of interest is feelings of safety. But even that phrase contains several possible meanings. It could refer to whether young people feel safe travelling to and from school, whether they avoid particular places, whether they feel confident accessing local spaces, or whether they know where to go for help if they feel unsafe.

That matters because each interpretation would lead to different survey questions.

Possible indicators could include:

• How safe young people feel travelling around their local area
• Whether there are specific places they avoid because they feel unsafe
• How confident they feel accessing local spaces, activities or services
• Whether they know where to seek help if they are worried about safety

Only then do we get to individual survey items, for example:

“How safe do you feel travelling around your local area during the day?”

or

“Are there places in your local area that you avoid because you feel unsafe?”

This progression is what makes the data meaningful. Without it, we risk jumping from a broad evaluation ambition straight to a question like “Do you feel safer?” and then struggling to interpret what the answer actually tells us. Safer where? In what situations? Compared to when? For which groups of young people?

The survey question is therefore not the starting point; it is the end point of a chain of reasoning.

When that chain is clear, the resulting data is much more likely to be useful. When it is not, we risk collecting responses that are hard to connect back to the evaluation question with any confidence.

Three common ways the survey design chain breaks down

1. We ask what is easy to ask, rather than what matters

Surveys often drift towards questions that are straightforward to ask and easy to report:

• Did participants enjoy the programme?
• Did they find it useful?
• Would they recommend it?

These are perfectly legitimate questions. They can tell us something about participant experience, delivery quality and perceived relevance. But they are not usually outcome measures.

If the evaluation question is whether people gained skills, became more confident, improved their wellbeing, or changed their behaviour, then a survey focused mainly on satisfaction is answering a different question.

This distinction matters because satisfaction can easily be mistaken for success. However, a participant may be very satisfied with a programme that has little effect on the intended outcome. Equally, an intervention that is demanding or challenging may be less immediately enjoyable while still effective in producing meaningful change.

Satisfaction is usually evidence about experience, not impact.

If we are not careful, the survey begins to encourage a stronger narrative than the evidence can bear.

2. We let ‘nice to know’ questions dilute the purpose of the survey

Another familiar problem is overloading a survey with questions that are potentially interesting, but not clearly linked to the evaluation.

These ‘nice to know’ questions usually creep in for understandable reasons. A commissioner is curious about one additional issue. A delivery team sees a chance to gather a bit more intelligence. An evaluator wonders whether a particular variable might prove helpful later.

Individually, each addition can seem harmless. Collectively, they change the shape of the survey.

Every extra question creates a burden for respondents. It also imposes a responsibility on the evaluator: it implies that the answer is worth collecting, analysing and interpreting. Each question should have a clear purpose, such as helping us to:

• Answer an evaluation question
• Make sense of an important finding
• Understand who has responded
• Inform a meaningful decision

If it does none of these things, it is worth asking whether it belongs there at all.

The fact that something would be interesting to know is not, by itself, a good reason to put it in a survey.

This is not about keeping surveys shorter for the sake of it; it is about keeping them sharper.

A strong survey does not collect everything we might be interested in – it collects what the evaluation actually needs.

3. We treat wording problems as the main issue, when they are often symptoms of weak conceptual design

Some survey problems are easy to spot:

• Double-barrelled questions
• Vague wording
• Inconsistent timeframes
• Response options that do not fit the question

For example:

“How satisfied were you with the quality and accessibility of the support?”

This asks about two different things at once. Someone may have found the support high quality but difficult to access, or very accessible but not especially valuable. The question gives them no clean way to make these distinctions.

These issues matter because they reduce data quality and make findings harder to interpret.

But they are often symptoms of a deeper problem: insufficient clarity about what is being measured and why. If we are not clear about the outcome or concept we are trying to capture, or the indicator we are using to capture it, it becomes much harder to write a precise and useful question. Weak wording often reflects weak thinking earlier in the design process.

Good question-writing is important, but it cannot rescue a survey that was never properly rooted in the evaluation purpose in the first place.

What better survey practice looks like

Good survey design begins by working carefully from the evaluation question to the evidence needed to answer it.

Before drafting items, it is worth being able to answer four things:

• What does the evaluation need to understand?
• What outcome, concept or aspect of delivery are we trying to capture?
• What would count as useful evidence in relation to that?
• What could we reasonably conclude from the responses?

Asking these questions improves survey design because it forces clarity before data collection begins.

It also helps reveal where a survey may not be the most appropriate tool, or when it is only part of the wider answer and therefore needs to be complemented by qualitative work, administrative data or other evidence.

For example, if we want to understand why participants disengaged from a programme, a survey may provide useful breadth, but qualitative work may be needed to understand the reasons in depth. If the evaluation needs to know how common a particular barrier is across a defined population, a survey may be very helpful.

The evaluation question should determine the role of the method, not the other way around.

A simple test

A useful test for every proposed survey question is:

What evaluation question will this help us answer, and what claim could we reasonably make from the response?

If the answer is unclear, the question may need to be revised, repositioned or removed.

This is a simple but powerful test. It exposes questions that do not really need to be there – and, just as importantly, questions that need to be sharpened because the evaluation is depending on them more than anyone has recognised.

Beyond ‘good questions’

Good survey design is not just about drafting well-phrased questions. It is about making sure those questions sit within a coherent route from evaluation purpose to evidence.

That means:

• Starting with what the evaluation needs to understand
• Translating outcomes or concepts into meaningful indicators
• Writing survey questions that capture those indicators clearly
• Resisting the temptation to collect data simply because it might be useful

A well-designed survey does more than gather responses. It strengthens the connection between the evaluation question, the evidence generated and the conclusions we are ultimately able to draw – and that is what makes it valuable.

In next month’s blog, Ciaran will explore why measuring an outcome is not always as straightforward as it sounds – and how stronger indicators can help evaluators generate evidence that is more meaningful and useful.

With thanks to Iona Nixon for her thoughtful input as this piece took shape, and for her careful comments on the final draft.