You can run flawless statistics on a survey and still reach worthless conclusions — if the survey did not measure what you think it did. That is what reliability and validity protect against. Every serious researcher needs to understand them.
The classic analogy
Imagine a dartboard:
- Reliable but not valid: all your darts land tightly together, but far from the bullseye. Consistent, but consistently wrong.
- Valid but not reliable: your darts scatter around the bullseye. Right on average, but unpredictable.
- Reliable and valid: tight cluster on the bullseye. This is the goal.
Reliability: is your measure consistent?
Reliability is about consistency. Would you get the same result if you measured again? Key types:
- Internal consistency — do the items in a scale agree with each other? Measured by Cronbach's alpha (aim for ~0.70+).
- Test-retest reliability — does the same person score similarly on two occasions?
- Inter-rater reliability — do different raters agree?
Validity: are you measuring the right thing?
Validity is about accuracy — whether your instrument measures the concept it claims to. Key types:
- Content validity — do the items cover the whole concept? (often judged by experts)
- Construct validity — does the measure behave as theory predicts? (often checked with factor analysis)
- Criterion validity — does it correlate with an established measure or outcome?
Why you need both
- A reliable but invalid survey gives you consistent nonsense.
- A valid but unreliable survey gives you noisy, untrustworthy readings.
Only when a measure is both reliable and valid can you trust the results built on it.
How to check them in practice
- Pilot your survey before full rollout.
- Run Cronbach's alpha on each multi-item scale.
- Use factor analysis to confirm items group as expected.
- Where possible, correlate your measure with an established one.
- Get expert review of item content.
The full checking table
Each property has a specific diagnostic and a specific threshold. This is the reference to work through before a scale goes into the field, and again after the data comes back.
| Property | Question it answers | How to check | Acceptable |
|---|---|---|---|
| Internal consistency | Do the items hang together? | Cronbach's alpha, McDonald's omega | 0.70 and above |
| Test-retest reliability | Is it stable over time? | Correlate two administrations, 2 to 4 weeks apart | r above 0.70 |
| Inter-rater reliability | Do two enumerators agree? | Cohen's kappa, or ICC for continuous ratings | kappa above 0.60 |
| Parallel forms | Do two versions measure the same thing? | Correlate the two forms | r above 0.70 |
| Content validity | Does it cover the whole concept? | Expert panel, content validity ratio | Judgemental |
| Face validity | Does it look sensible to respondents? | Cognitive interviews during piloting | Judgemental |
| Construct validity | Does it behave as theory predicts? | Factor analysis, known-groups comparison | Structure matches theory |
| Convergent validity | Does it agree with related measures? | Correlate with an established scale | r above 0.50 |
| Discriminant validity | Is it distinct from unrelated measures? | Correlate with an unrelated scale | Low correlation |
| Criterion validity | Does it predict the real outcome? | Correlate with a gold standard | Depends on the field |
The row most often skipped is discriminant validity, and its absence produces a specific failure: a "job satisfaction" scale that correlates 0.86 with a general wellbeing measure may not be measuring anything distinct at all.
Reliable but invalid: the dangerous quadrant
The four combinations are not equally bad, and the ordering surprises people.
| Valid | Invalid | |
|---|---|---|
| Reliable | What you want | The dangerous case — consistently wrong |
| Unreliable | Impossible by definition | Obviously broken, at least it announces itself |
A scale that is unreliable is noisy and looks noisy. Your correlations attenuate, your tests lose power, and something visibly fails to work. That is unpleasant but self-announcing.
A scale that is reliable but invalid is a weighing machine calibrated five kilos heavy. It gives the same answer every time, the internal consistency is beautiful, alpha comes back at 0.91 — and every conclusion drawn from it is wrong. Nothing in the reliability output warns you.
This is why alpha alone is not evidence that a scale works. A high alpha proves the items correlate with each other; it says nothing about whether they measure the construct you named. Five items that all ask the same question in slightly different words will produce excellent reliability and poor content validity.
What Cronbach's alpha does not tell you
Alpha is the most reported and least understood statistic in survey research. Four things worth knowing:
It rises with the number of items. A twenty-item scale of mediocre items can reach 0.85 purely on length. Alpha is not a per-item quality measure.
It assumes one dimension. If your scale actually measures two things, alpha can still look acceptable while the mean score is meaningless. Run a factor analysis before trusting a composite.
Above 0.95 is a warning, not a triumph. It usually means items are near-duplicates. You have measured a narrow slice of the construct very precisely and missed the rest.
It is a property of the sample, not the instrument. A scale with alpha 0.82 in an urban pilot may return 0.61 in a rural sample with a translated version. Report alpha for your data, never cite the original paper's figure as if it transferred.
McDonald's omega is the better statistic and is now available in SPSS, jamovi and the R psych package. It does not assume all items contribute equally, which alpha does and almost no real scale satisfies.
Threats specific to translated and field instruments
Most of what undermines reliability in Indian survey work happens before analysis, in the questionnaire and the field.
| Threat | What goes wrong | Control |
|---|---|---|
| Translation drift | "Satisfied" and "content" carry different weight across languages | Back-translation by an independent translator |
| Reverse-worded items | Negation translates poorly and confuses respondents | Test reversed items separately in the pilot |
| Enumerator effects | Different interviewers elicit different answers | Randomise assignment; check reliability by enumerator |
| Response set | Agreeing with everything regardless of content | Mix item direction; check straight-lining |
| Ceiling and floor effects | Everyone answers at one extreme | Widen the response range in the pilot |
| Recall error | Long reference periods for irregular events | Anchor to local calendar events |
| Concept has no local equivalent | "Household head" varies by family structure | Define explicitly, apply consistently |
The enumerator check is under-used and cheap. Compute your key indicators broken down by enumerator. If one interviewer's respondents are systematically more satisfied, the difference is unlikely to be in the population. See how to reduce bias in field surveys and training enumerators for a quality field survey.
The practical sequence
For a new or adapted instrument, this order avoids nearly all the expensive mistakes:
- Define the construct in writing, before drafting any items.
- Draft items covering every facet of that definition — this is content validity.
- Expert review, then revise.
- Cognitive interviews with 8 to 12 respondents; ask them to explain what each question means to them.
- Translate and back-translate.
- Pilot with 30 to 50 respondents.
- Compute alpha and item-total correlations; drop or rewrite weak items.
- Run an exploratory factor analysis if the sample allows.
- Field the revised instrument.
- Re-check reliability on the full dataset and report it.
Step four returns more per hour spent than any statistic in this article. Asking a dozen people what they think a question is asking reveals ambiguities that no amount of post-hoc analysis can repair. For questionnaire construction see how to design a baseline survey questionnaire, and for what to do with the resulting scores, how to analyse Likert scale data. Our statistical and research data analysis service covers scale validation and factor analysis; the COSMIN checklist is the most thorough public standard for measurement property assessment.
The takeaway
Reliability and validity are not academic box-ticking — they are what make your survey findings mean something. Build them in at the design stage and verify them in analysis, and everything you conclude rests on solid ground.
Frequently Asked Questions
What is the difference between reliability and validity?
Reliability is consistency — the same measurement repeated gives the same answer. Validity is accuracy — the measurement captures the concept you intended. A measure can be reliable without being valid, which is the more dangerous failure because nothing in the output warns you.
What is an acceptable Cronbach's alpha?
0.70 and above for research use, 0.80 and above is good. Below 0.50 the items should not be combined. Above 0.95 usually signals redundant, near-duplicate items rather than an excellent scale.
Can a measure be reliable but not valid?
Yes, and this is the most serious measurement problem. A consistently miscalibrated instrument produces stable, repeatable, wrong answers. High reliability provides no protection against it — only validity evidence does.
How do I test validity in a survey?
Content validity through expert review against a written definition of the construct; construct validity through factor analysis and known-groups comparisons; convergent and discriminant validity by correlating with related and unrelated measures; criterion validity by comparison against a gold standard where one exists.
Do I need to test reliability if I use a published scale?
Yes. Reliability is a property of your sample, not of the instrument. A scale that performed well in its original population may perform differently in a different language, culture or age group. Report alpha for your own data.
What is inter-rater reliability and when does it matter?
It measures whether two enumerators or coders assign the same values to the same case. It matters whenever data collection involves judgement — observational assessments, qualitative coding, condition ratings. Cohen's kappa above 0.60 is generally considered acceptable.
Need help turning your data into defensible results? DSTechnoverse offers statistical and research data analysis services in SPSS, R, and Python — survey analysis, thesis and dissertation support, and business statistics. Get in touch to discuss your dataset.