Research Validity: A Field Practitioner's Guide
In this piece
Research validity protects two claims: a study measures what it says it measures and its conclusions hold under scrutiny. Reliability is consistency and validity is accuracy. A perfectly reliable instrument can still measure the wrong thing and return stable, wrong answers.
Key Takeaways
- Each validity type maps to a different failure: internal to confounding, external to generalizability and construct to conceptual accuracy. Knowing which type is at risk shapes every design decision.
- Qualitative validity rests on Lincoln and Guba's trustworthiness criteria. Field practices like triangulation, member checking, negative case analysis and consistent follow-up criteria protect those criteria in practice.
- In multi-market studies, a question can translate accurately and still measure a different concept in each market. Cognitive interviewing before fielding catches that drift.
- A validity audit trail is built during design and fieldwork. Researchers who record design rationale and link every theme to its source evidence defend findings more credibly when a client challenges the conclusions.
The Core Types of Research Validity
Each type flags a different way a study can fail.
Internal validity asks whether an observed relationship reflects real cause and effect, free from confounds and design errors.
External validity asks whether findings generalize to the population or context the decision requires.
Construct validity asks whether the questions measure the psychological or behavioral object they claim to measure.
Content validity asks whether the instrument covers the full domain of the construct, usually as judged by qualified reviewers.
Criterion validity asks whether scores correlate with an established external measure.
Face validity asks whether the instrument appears, on its face, to measure what it claims. It is usually judged by respondents or other non-experts.
Statistical conclusion validity asks whether the statistical test and sample are appropriate for the inference being drawn.
Validity in Qualitative Research
Qualitative studies face the same underlying challenge under a different vocabulary. Lincoln and Guba set out the standard framework in their 1985 book Naturalistic Inquiry as four trustworthiness criteria. Credibility asks whether the findings accurately reflect respondents' realities. Transferability asks whether they apply in comparable contexts. Dependability asks whether the process would produce consistent results under similar conditions. Confirmability asks whether the conclusions can be traced to the data.
Several field practices protect these criteria. Lincoln and Guba's own techniques include prolonged engagement, triangulation across data sources, member checking, negative case analysis and an audit trail. Applied teams add saturation before fieldwork closes and control of interviewer effects. Interviewer effects matter because one moderator's style can skew the pattern of responses across a whole study.
Consistent follow-up criteria are where platform design intersects with validity. Enumerate's AI Probing reads each open-ended answer while the respondent is still present. It follows up only when the answer misses the researcher's criteria, up to a maximum the researcher sets. Because the same criteria apply to every respondent in the study, one moderator's habits can't shape the results.
Validity Across Markets
Multi-market studies add a construct validity problem of their own. A positioning study can run in several markets with translations that are grammatically accurate and still measure a different concept in each. "Convenience" might anchor to transit time in one market and to price and proximity in another. Cognitive interviewing with a few respondents in each market before fielding catches that conceptual drift. Our piece on multi-market research explains why native-language responses should enter analysis as primary data, so a researcher can check each theme against the original wording.
Documenting Validity Decisions
Validity decisions made during design and fieldwork need to survive scrutiny at the readout. Keep a short audit trail recording why each question was worded as it was, which construct it operationalizes and which alternatives were rejected. That record often decides whether a client accepts a contested finding or quietly sets it aside. It also maps directly to confirmability, because it lets a reviewer follow the chain from researcher assumption to design choice to evidence.
Every theme should connect to the exact transcript passage, recording or clip behind it. Enumerate keeps codes, themes and summaries linked to the response or passage behind them, so a researcher can open a contested finding and check its source. Redaction choices belong in the audit trail too. Settle PII handling before the first transcript is exported so redaction doesn't break the chain from a theme back to its passage.
See how researcher-set probing criteria and evidence-linked themes work on a live study. Book a demo with Enumerate.
Frequently Asked Questions
The main types are internal, external, construct, content, criterion, face and statistical conclusion validity. Internal and external validity concern whether a result is real and whether it generalizes. Construct, content, criterion and face validity concern whether the instrument measures the right thing. Statistical conclusion validity concerns whether the analysis supports the inference drawn from it.
Internal validity asks whether an observed relationship reflects real cause and effect, free from confounds and design errors. External validity asks whether the finding holds beyond the study, in the population or context the decision is about. A tightly controlled study can be strong on the first and weak on the second.
Reliability is consistency: the same instrument administered twice under the same conditions should produce the same result. Validity is accuracy: the instrument needs to measure the right construct in the first place. You need both, but a reliable invalid measure reliably delivers the wrong answer.
The standard framework is Lincoln and Guba's trustworthiness criteria: credibility, transferability, dependability and confirmability. In practice that means triangulating across data sources, running member checks, looking for negative cases and reaching saturation before fieldwork closes. It also means keeping an audit trail that links every conclusion to its evidence. Applying the same follow-up criteria across interviews matters too, because findings that reflect moderator variation more than respondent reality fail the dependability test.
Related reading
High Frequency Panelists: The 4% Problem in Survey Research
4% of panelists account for 30, 50% of all completed surveys. Here's the economics that produced professional respondents, and what it means for your data.
Read more
Qualitative Researcher Career Paths: 3 Roles That Will Thrive
Three researcher roles will thrive as AI reshapes qual. Here's which career paths hold value, and where the pressure is most acute.
Read more
Research Quality Assurance: A Practical Guide for Insights Teams
Research quality assurance separates reliable insight from expensive noise. Learn the frameworks that keep your studies defensible at scale.
Read more