Definition and research guide
This page combines a MeSH definition, publication statistics and selected research abstracts. It is a starting point for reading the literature, not a medical textbook or a systematic evidence review.
Content available: 4 complete abstracts (857 words), 10 references. Full article text is not stored or reproduced here; follow the PMID, DOI or PMC links to read the original sources and their access terms.
Definition
The statistical reproducibility of measurements (often in a clinical context), including the testing of instrumentation or techniques to obtain reproducible results. The concept includes reproducibility of physiological measurements, which may be used to develop rules to assess probability or prognosis, or response to a stimulus; reproducibility of occurrence of a condition; and reproducibility of experimental results.
Definition: NLM Medical Subject Headings (MeSH) scope note (Public domain (U.S. National Library of Medicine); attribution requested). View the MeSH record.
Also known as: Reproducibility Of Result; Of Result, Reproducibility; Of Results, Reproducibility; Result, Reproducibility Of; Results, Reproducibility Of; Reproducibility of Finding; Finding Reproducibilities; Finding Reproducibility.
Where it fits in the MeSH classification
- Broader topics
- Epidemiologic Research Design; Evaluation Studies as Topic
- Narrower topics
- Data Accuracy; Dimensional Measurement Accuracy
Research landscape
PubMed indexes 523,076 records tagged with the MeSH heading “Reproducibility of Results” (counts retrieved 2026-10-29). 121,694 of them (23.3%) were published from 2020 onward, and the busiest period so far is 2010-2019 with 214,070 records.
Reviews account for 30,858 records (5.9%) and meta-analyses for 3,754. Randomized controlled trials number 9,409 (1.8%) as a share of all records on the topic. Randomized trials make up only a small share of this literature, so much of what is known comes from laboratory, observational or descriptive work.
Show the data
| Period | Records |
|---|---|
| through 1999 | 44,906 |
| 2000-2009 | 142,406 |
| 2010-2019 | 214,070 |
| 2020 onward | 121,694 |
Publication-type chart data
| Type | Records | Share |
|---|---|---|
| Reviews | 30,858 | 5.9% |
| Meta-analyses | 3,754 | 0.7% |
| Randomized controlled trials | 9,409 | 1.8% |
Method: PubMed query "Reproducibility of Results"[MeSH Terms] with publication-type and date filters, retrieved 2026-10-29.
Reading the evidence in clinical research
Research methods determine how much a result can be trusted. Randomized trials guard against confounding, systematic reviews and meta-analyses combine many studies, and observational designs are valuable but can show only association. Reporting standards, trial registration and pre-specified outcomes reduce bias. When reading about a method, ask what problem it is designed to solve and which problems it does not solve.
Evidence from open-access studies
These open-access studies come from PubMed’s best-match results for this heading (papers whose main topic it is), ranked here by citation count, with one recent addition. Each abstract is reproduced in full under its recorded open licence so that you can read the authors’ own words. They illustrate the literature; they are not a systematic summary of it. Citation counts measure attention, not reliability or clinical benefit.
| Study / authors | Year | Source label / design | Europe PMC citations | Licence / source |
|---|---|---|---|---|
| Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA | 2023 | Research article Design not recorded in structured metadata | 383 | cc by PMID 36917576 · DOI · PMC10013894 |
| Evaluating CloudResearch's Approved Group as a solution for problematic data quality on MTurk | 2023 | Research article Design not recorded in structured metadata | 74 | cc by PMID 36326997 · DOI · PMC10700412 |
| Assumptions and analysis planning in studies with missing data in multiple variables: moving beyond the MCAR/MAR/MNAR classification | 2023 | Research article Design not recorded in structured metadata | 56 | cc by PMID 36779333 · DOI · PMC10396404 |
| The fundamentals of eye tracking, Part 7: Determining data quality | 2026 | Research article Design not recorded in structured metadata | 2 | cc by PMID 42230437 · DOI · PMC13230380 |
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA [1]
Abstract reading summary — source excerpts
- Opening excerpt
- With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots).
- Closing excerpt
- 00, yet took the longest to collect the data.
- Interpretation limits
- Verbatim excerpts from the authors’ abstract; the closing excerpt may describe a finding, limitation or recommendation. Design, population, effect size and uncertainty have not been independently extracted or appraised. Read the complete abstract and linked paper before interpreting the findings.
Read complete abstract (161 words)
With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher's question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.
Abstract reproduced from the open-access article under the recorded cc by licence; © the authors. PMID 36917576 · DOI · PMC10013894
Evaluating CloudResearch's Approved Group as a solution for problematic data quality on MTurk [2]
Abstract reading summary — source excerpts
- Opening excerpt
- Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers.
- Closing excerpt
- We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.
- Interpretation limits
- Verbatim excerpts from the authors’ abstract; the closing excerpt may describe a finding, limitation or recommendation. Design, population, effect size and uncertainty have not been independently extracted or appraised. Read the complete abstract and linked paper before interpreting the findings.
Read complete abstract (240 words)
Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch's vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.
Abstract reproduced from the open-access article under the recorded cc by licence; © the authors. PMID 36326997 · DOI · PMC10700412
Assumptions and analysis planning in studies with missing data in multiple variables: moving beyond the MCAR/MAR/MNAR classification [3]
Abstract reading summary — source excerpts
- Opening excerpt
- Researchers faced with incomplete data are encouraged to consider whether their data are 'missing completely at random' (MCAR), 'missing at random' (MAR) or 'missing not at random' (MNAR) when planning their analysis.
- Closing excerpt
- In this manuscript we outline an approach for deciding which method to use to handle multivariable missing data in an analysis, using directed acyclic graphs to depict missingness assumptions and determining the implications in terms of recoverability of the target estimand.
- Interpretation limits
- Verbatim excerpts from the authors’ abstract; the closing excerpt may describe a finding, limitation or recommendation. Design, population, effect size and uncertainty have not been independently extracted or appraised. Read the complete abstract and linked paper before interpreting the findings.
Read complete abstract (244 words)
Researchers faced with incomplete data are encouraged to consider whether their data are 'missing completely at random' (MCAR), 'missing at random' (MAR) or 'missing not at random' (MNAR) when planning their analysis. However, there are two major problems with this classification as originally defined by Rubin in the 1970s. First, when there are missing data in multiple variables, the plausibility of the MAR assumption is difficult to assess using substantive knowledge and is more stringent than is generally appreciated. Second, although MCAR and MAR are sufficient conditions for consistent estimation with specific methods, they are not necessary conditions and therefore this categorization does not directly determine the best approach for handling the missing data in an analysis. How best to handle missing data depends on the assumed causal relationships between variables and their missingness, and what these relationships imply in terms of the 'recoverability' of the target estimand (the population parameter that encodes the answer to the underlying research question). Recoverability is defined as whether the estimand can be consistently estimated from the patterns and associations in the observed data without needing to invoke external information on the extent to which the distribution of missing values might differ from that of observed values. In this manuscript we outline an approach for deciding which method to use to handle multivariable missing data in an analysis, using directed acyclic graphs to depict missingness assumptions and determining the implications in terms of recoverability of the target estimand.
Abstract reproduced from the open-access article under the recorded cc by licence; © the authors. PMID 36779333 · DOI · PMC10396404
The fundamentals of eye tracking, Part 7: Determining data quality [4]
Abstract reading summary — source excerpts
- Opening excerpt
- Understanding the quality of eye-tracking recordings, often characterized using accuracy, precision, and data loss, is crucial for the interpretation of eye tracking data.
- Closing excerpt
- We hope that this article enables any researcher to determine, critically evaluate, and report on eye-tracking data quality, and that it spurs researchers to adopt a data quality perspective in all their future eye-tracking studies.
- Interpretation limits
- Verbatim excerpts from the authors’ abstract; the closing excerpt may describe a finding, limitation or recommendation. Design, population, effect size and uncertainty have not been independently extracted or appraised. Read the complete abstract and linked paper before interpreting the findings.
Read complete abstract (212 words)
Understanding the quality of eye-tracking recordings, often characterized using accuracy, precision, and data loss, is crucial for the interpretation of eye tracking data. Eye-tracking data quality can furthermore place fundamental limits on what studies can be conducted with an eye tracker, and one may be required to report eye-tracking data quality when publishing a study. However, how does one determine the quality of eye-tracking data? This article provides an overview of operationalizations of accuracy, precision, and data loss and practical advice for determining eye-tracking data quality. Furthermore, the programming code for calculating various quality metrics for a segment of eye-tracking data is provided in MATLAB, Python, and R. Also provided is ETDQualitizer, a tool designed to enable anyone to easily determine the data quality of their recordings. We provide a version that is browser-based ( https://dcnieho.github.io/ETDQualitizer ) and enables determining eye-tracking data quality without installation or programming, while ensuring data privacy by running entirely locally. ETDQualitizer is further provided as a MATLAB, Python, and R library ( https://github.com/dcnieho/ETDQualitizer ) that can be integrated in one's analysis scripts. We hope that this article enables any researcher to determine, critically evaluate, and report on eye-tracking data quality, and that it spurs researchers to adopt a data quality perspective in all their future eye-tracking studies.
Abstract reproduced from the open-access article under the recorded cc by licence; © the authors. PMID 42230437 · DOI · PMC13230380
Key reviews and meta-analyses
The most cited review-type papers among PubMed’s best-match results for this heading, listed with bibliographic details only. Follow the links for the full text and access terms.
- Research article Measuring agreement in method comparison studies [5]
- Research article Is the CVI an acceptable indicator of content validity? Appraisal and recommendations [6]
- Research article Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: individual participant data meta-analysis [7]
- Research article A psychometric toolbox for testing validity and reliability [8]
- Research article Constructing validity: New developments in creating objective measuring instruments [9]
- Research article Reproducibility in science: improving the standard for basic and preclinical research [10]
References
- Douglas BD, Ewell PJ, Brauer M. Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. . 2023. PMID 36917576 · DOI · PMC10013894
- Hauser DJ, Moss AJ, Rosenzweig C, Jaffe SN, Robinson J, Litman L. Evaluating CloudResearch's Approved Group as a solution for problematic data quality on MTurk. . 2023. PMID 36326997 · DOI · PMC10700412
- Lee KJ, Carlin JB, Simpson JA, Moreno-Betancur M. Assumptions and analysis planning in studies with missing data in multiple variables: moving beyond the MCAR/MAR/MNAR classification. . 2023. PMID 36779333 · DOI · PMC10396404
- Niehorster DC, Nyström M, Hessels RS, Benjamins JS, Andersson R, Hooge ITC. The fundamentals of eye tracking, Part 7: Determining data quality. . 2026. PMID 42230437 · DOI · PMC13230380
- Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999. PMID 10501650 · DOI
- Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007. PMID 17654487 · DOI
- Levis B, Benedetti A, Thombs BD, DEPRESsion Screening Data (DEPRESSD) Collaboration. Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: individual participant data meta-analysis. BMJ. 2019. PMID 30967483 · DOI · PMC6454318
- DeVon HA, Block ME, Moyle-Wright P, Ernst DM, Hayden SJ, Lazzara DJ, et al.. A psychometric toolbox for testing validity and reliability. J Nurs Scholarsh. 2007. PMID 17535316 · DOI
- Clark LA, Watson D. Constructing validity: New developments in creating objective measuring instruments. Psychol Assess. 2019. PMID 30896212 · DOI · PMC6754793
- Begley CG, Ioannidis JP. Reproducibility in science: improving the standard for basic and preclinical research. Circ Res. 2015. PMID 25552691 · DOI
About this article
- Sources
- NLM Medical Subject Headings (definition, synonyms); NCBI PubMed (counts, identifiers); Europe PMC (open-access abstracts and citation counts).
- Verification
- Every reference was resolved against PubMed on 2026-10-31; title, year and DOI matched.
- Review state
- automated source verified; no human expert review has been recorded yet.
- Version
- 1, last updated 2026-10-29.
See how EvidenceLib is built. Spotted an error? Email support@evidencehelix.com.
Evidence strength
EvidenceHelix's computed view of the study designs behind this topic: how common meta-analyses, randomized trials and reviews are, how recent the literature is, and a plain-language reading. It describes the design mix of the literature. It is not a quality appraisal of any study, and it is not medical advice.