Reviewers and examiners see the same statistical errors over and over. Most are easy to avoid once you know to look for them. Here are ten of the most common — and the fix for each.
1. Choosing the wrong test
Running a t-test where you need ANOVA, or Pearson where the data is not normal. Fix: match the test to your question and data type before you start.
2. Ignoring assumptions
Tests assume things — normality, equal variances, independence. Fix: check the assumptions and switch to non-parametric tests when they fail.
3. Confusing correlation with causation
A relationship between two variables does not mean one causes the other. Fix: be precise in your language; correlation is not proof of cause.
4. p-hacking
Running many tests until something is "significant." Fix: decide your analyses in advance and report everything you ran.
5. Misreading the p-value
A p-value is not the probability your hypothesis is true, and p < 0.05 is not "proof." Fix: interpret it correctly and report effect sizes too.
6. Ignoring effect size
A result can be statistically significant but trivially small. Fix: always report effect sizes (Cohen's d, r, odds ratios) alongside p-values.
7. Too small a sample
Underpowered studies miss real effects. Fix: calculate sample size before collecting data.
8. Not handling missing data honestly
Silently dropping cases can bias results. Fix: report how much data was missing and how you handled it.
9. Over-interpreting the results
Claiming more than the data supports, or generalising beyond your sample. Fix: keep conclusions within the limits of your design.
10. Poor reporting
Tables without context, results that do not match the numbers. Fix: report clearly — test used, statistic, degrees of freedom, p-value, and effect size.
The common thread
Almost every mistake here comes from rushing the thinking, not the maths. Decide your methods in advance, respect the assumptions, and report honestly. Do that and your research will survive scrutiny that sinks weaker work.
How each mistake is detected and repaired
Naming the ten errors is the easy part. What makes them fixable is knowing what each looks like in real output and what to do once you find it.
| Mistake | How to detect it | The repair |
|---|---|---|
| Wrong test for the design | Paired data analysed with an independent test; 3 groups with multiple t-tests | Re-select from outcome type, group count and independence |
| Assumptions never checked | No Levene, no Q-Q plot, no VIF anywhere in the output | Run diagnostics; switch to a robust or non-parametric method |
| Correlation read as causation | The discussion uses "leads to" for observational data | Rewrite as association; state confounders explicitly |
| p-hacking | More analyses run than reported; subgroups appearing only in results | Pre-register; disclose every test performed |
| p-value misread | "There is a 3% chance the null is true" | Restate as evidence against the null under assumption |
| Effect size absent | A results table of p-values with no magnitudes | Add d, eta squared or odds ratios with CIs |
| Underpowered sample | No power calculation; small n with a null result | Report the detectable effect size; frame as inconclusive |
| Missing data mishandled | Sample size varies between tables with no explanation | Diagnose the mechanism; use multiple imputation |
| Over-interpretation | Conclusions broader than the sampling frame | Bound the claim to the population sampled |
| Poor reporting | Missing df, missing n, p = .000 | Apply a reporting checklist before submission |
The three that cost the most
Ignoring clustering. A survey that sampled 40 villages and then 25 households in each is not 1,000 independent observations. Analysed as though it were, every standard error is too small and every significance test overstated — sometimes dramatically. Nothing in the dataset flags this; you have to know how the sample was drawn. The repair is a mixed-effects model, clustered standard errors, or survey-weighted procedures. This is the most consequential error in Indian household survey work and the one least often caught in review.
Stepwise variable selection. Letting software choose predictors by p-value produces different models from near-identical datasets, inflates R squared, and yields p-values that no longer mean what they claim. Specify your model from theory and prior evidence, and report the model you specified.
HARKing — hypothesising after results are known. Running twenty comparisons, finding two significant, and presenting those two as the study's hypotheses. This is not a grey area, and it is the mechanism behind a large share of findings that fail to replicate. If a finding was unplanned, label it exploratory. That single clause preserves the credibility of everything else in the paper.
The mistakes specific to survey research
Beyond the general ten, field survey work has its own recurring failures.
| Mistake | Consequence |
|---|---|
| Analysing a single Likert item with ANOVA | Ordinal data treated as interval; use Kruskal-Wallis |
| Summing scale items without checking reliability | The composite may measure two unrelated things |
| Not reverse-scoring negative items | Alpha collapses; the finding inverts |
| Ignoring survey weights | Estimates biased toward over-sampled groups |
| Treating "don't know" as a neutral midpoint | Non-opinion contaminates the centre of the distribution |
| Comparing baseline and endline on different samples | Attrition mistaken for change |
| Chi-square with expected counts below 5 | The test is invalid; use Fisher's exact or merge categories |
| Reporting percentages from 15 interviews | Implies precision purposive sampling cannot support |
The attrition point deserves emphasis for anyone doing baseline and endline evaluation. If poorer households are likelier to migrate between rounds, the endline sample is systematically better off than the baseline sample — and the programme will appear to have raised incomes even if it did nothing at all. Always compare baseline characteristics of those who remained against those lost, and report the comparison.
A pre-submission audit
Fifteen minutes with this list catches most of what reviewers return work for.
| Check | Confirmed? |
|---|---|
| Every test justified by the design, not by habit | |
| Assumption checks reported, not just performed | |
| Every p-value accompanied by an effect size | |
| Confidence intervals present for the main findings | |
| Exact p-values, no p = .000 | |
| Degrees of freedom and n reported for every test | |
| Missing data mechanism and handling described | |
| Every analysis run is disclosed, including the failures | |
| Exploratory analyses labelled as exploratory | |
| Causal language used only where the design supports it | |
| Clustering, weighting or repeated measures accounted for | |
| Conclusions bounded by the population actually sampled |
The eighth row is the one people resist. Disclosing that you ran eleven comparisons and two were significant feels like weakening the paper. It is the opposite: an undisclosed multiplicity problem discovered by a reviewer is far more damaging than a disclosed one, and the disclosure often costs nothing more than a sentence in the methods.
Why these persist
Almost none of this is caused by ignorance of the correct method. Three structural pressures do most of the damage: publication systems that reward positive findings, deadlines that make a proper power calculation feel like a luxury, and software that runs any procedure you ask for without comment.
SPSS will not object to a t-test on clustered data. R will not warn you that your predictors were chosen by stepwise selection. The reasoning has to happen before the software is opened, which is the argument for settling the analysis plan while designing the study — covered in the five steps of hypothesis testing, how to choose the right statistical test and how to calculate sample size.
For reporting standards, how to present statistical results. Our statistical and research data analysis service applies the audit above as standard. The EQUATOR Network hosts the reporting guidelines for every major study design and is the best single resource for checking your write-up against field expectations.
Frequently Asked Questions
What is the most common statistical mistake in research?
Choosing a test that does not match the study design — most often analysing paired data as independent, running several t-tests where ANOVA is required, or treating clustered survey data as a simple random sample. All three inflate significance.
What is p-hacking and how do I avoid it?
Running many analyses, subgroups or model specifications and reporting only those that reached significance. Avoid it by fixing your primary outcome and analysis plan before collecting data, correcting for multiple comparisons, and disclosing every test you ran.
Why is correlation not causation?
Because an unmeasured third factor may drive both variables, or the causal direction may run the other way. Regression controls only for variables you actually measured. Causal claims need a design — randomisation, instrumental variables, or a credible difference-in-differences setup.
Do I have to report effect sizes?
Yes, and most journals now require it. A p-value indicates whether an effect is detectable; the effect size indicates whether it matters. Without it, a reader cannot distinguish a trivial difference in a huge sample from an important one.
What happens if I ignore clustering in survey data?
Your standard errors will be too small and your p-values too optimistic, sometimes by a large margin. A survey of 25 households in each of 40 villages carries far less independent information than 1,000 randomly selected households.
How do I know if my sample was too small?
Compute the effect size your design could detect at 80% power and compare it against effects considered meaningful in your field. If your study could only have detected an implausibly large effect, a null result is uninformative rather than reassuring — and should be reported that way.
Need help turning your data into defensible results? DSTechnoverse offers statistical and research data analysis services in SPSS, R, and Python — survey analysis, thesis and dissertation support, and business statistics. Get in touch to discuss your dataset.