Skip to content

Guide

How to analyze survey results

To analyze survey results well, start from the question you set out to answer and let it guide what you look at. Always report the base (the number of people who answered, or n) and treat small bases as directional rather than conclusive. Read full distributions instead of relying on averages, segment your data to see where groups differ, and use the open-text answers to understand the why behind the numbers. Describe what the data shows as association, not proof of cause, and stay alert to who did not respond.

Start from the question you set out to answer

Before you touch a single chart, write down the decision the survey was meant to inform. A survey about a product launch answers a different question than one about workplace morale, and the same data can look encouraging or worrying depending on what you are trying to learn.

Let that question decide which results matter. It keeps you from cherry-picking the numbers that look good and from getting lost in secondary metrics that have nothing to do with the choice in front of you.

Report the base (n) before any percentage

Every percentage rests on a base: the number of people who actually answered that question. '80% agree' means one thing out of 500 responses and almost nothing out of 5. Always show the n next to the percentage so anyone reading it can judge how much weight it carries.

Treat small bases as directional. A handful of answers can hint at a pattern worth exploring, but it is not evidence you can act on with confidence. Wait for more responses, or frame the finding as a signal rather than a conclusion.

It's not just good practice: AAPOR's code of ethics requires publishing the sample size alongside results and providing, on request, the unweighted base each subgroup estimate rests on.

Read the distribution, not just the average

An average hides the shape of the answers. A satisfaction score of 3 out of 5 can come from a room full of lukewarm 3s, or from an even split of delighted 5s and angry 1s. Those two situations call for opposite responses, yet they share the same mean.

Look at the full spread: how many people chose each option, where the peaks are, and how many sit at the extremes. The distribution usually tells a clearer story than any single summary number.

Jamieson raised this in Medical Education when criticizing the routine use of means on Likert scales: these are ordered categories, not equal distances, so the median and the full distribution describe the responses better than the average.

Segment to see where groups differ

Overall numbers can flatten out real differences. Cross-tabulate your results by the groups that matter to you, such as region, role, plan, or how recently someone became a customer, and you often find that a single segment is driving the headline figure.

Segments only work if each one has enough base. When you split 200 responses into six groups, some will be too small to trust, so read those as directional. Live results help here: you can watch a segment fill up and wait until it is large enough before drawing a conclusion.

Cochran established this in Sampling Techniques when addressing domain estimation: the precision of a subgroup figure depends on the number of responses falling within that subgroup, not on the total sample size.

Find the why, and stay honest about cause

Closed questions tell you what is happening; open-text answers tell you why. Read a sample of the comments before you settle on any interpretation, because a single recurring complaint can explain a drop that the charts only hint at.

Two cautions keep your reading honest. First, a link between two things is association, not proof that one caused the other, because a hidden factor may drive both. Second, watch for non-response bias: the people who ignored your survey may differ from those who answered, and the happiest or busiest voices are often the ones missing.

And chasing a high response rate isn't enough on its own: the Groves and Peytcheva meta-analysis in Public Opinion Quarterly, which pooled 59 methodological studies designed to measure nonresponse bias, concluded that response rate does not generally predict that bias.

Frequently asked questions

How many responses do I need before I can trust the results?
It depends on how precise you need to be and how large your population is. A common convention is 95% confidence with a 5% margin of error, which for a large population works out to roughly 385 responses. Fewer responses can still be useful, but treat them as directional and widen your sense of the margin around each number.
What counts as enough base for a segment?
There is no single cutoff, but the smaller a segment, the wider its margin of error and the less you should lean on it. As a practical habit, report the n for every segment and treat any group with only a handful of responses as a signal to explore, not a fact to act on. If a segment matters to your decision, keep collecting until it is large enough.
If two groups answered differently, does that prove one thing caused the other?
No. A difference or a correlation shows association, not causation. A third factor you did not measure could be driving both patterns, so describe what the data shows and, if you need to establish cause, follow up with a controlled test rather than concluding from the survey alone.

Sources

  1. Jamieson, S. (2004). Likert scales: how to (ab)use them. Medical Education, 38(12), 1217-1218Why a mean over a Likert scale describes less well than the median and the distribution.
  2. Groves, R. M. & Peytcheva, E. (2008). The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis. Public Opinion Quarterly, 72(2), 167-189On why chasing a high response rate does not guarantee the absence of bias.
  3. AAPOR, Best Practices for Survey ResearchProfessional standard on disclosing the base (n), reporting results and not over-reading small bases.
  4. Groves, R. M. (2006). Nonresponse Rates and Nonresponse Bias in Household SurveysWhy who did not answer matters, and why a high response rate alone does not guarantee the absence of bias.

Build your survey with Camaleonic Survey

AI to draft it, your brand on every screen, and live results as responses arrive.

Start free