A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results. / Sandau, Nicolai; Aagaard, Thomas Vedste; Hróbjartsson, Asbjørn; Harris, Ian A.; Brorson, Stig.
In: Journal of Clinical Epidemiology, Vol. 156, 2023, p. 95-104.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - A meta-epidemiological study found that meta-analyses of the same trials may obtain drastically conflicting results
AU - Sandau, Nicolai
AU - Aagaard, Thomas Vedste
AU - Hróbjartsson, Asbjørn
AU - Harris, Ian A.
AU - Brorson, Stig
N1 - Publisher Copyright: © 2023 Elsevier Inc.
PY - 2023
Y1 - 2023
N2 - Objectives: To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures. Study Design and Setting: A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates. Results: We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%–99%; quality of life 58%–76%; adverse events 87%–91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%–78%, quality of life; 10%–24%; adverse events 10%–13%). Conclusion: Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.
AB - Objectives: To assess how much protocols reduce methodological flexibility and variation of results in meta-analyses comparing operative with nonoperative treatments for proximal humerus fractures. Study Design and Setting: A meta-epidemiological study. We searched four bibliographic databases for eligible meta-analyses. We contacted the authors of the meta-analyses, requesting a copy of their protocol. We identified the included studies and extracted intervention effect data for functional outcome, quality of life, and adverse events. Using the same intervention effect data for each outcome domain, we conducted multiple meta-analyses using differing combinations of methodological choices (analytical scenarios) without restricting the available methodological choices (as if no protocol was used). For each protocol, we repeated this process but restricted the available choices to those listed in the protocol. We then compared the distributions of effect estimates with and without protocols. Methodological flexibility was estimated as the difference in number of possible meta-analyses and the variation of results as the difference between the most conflicting effect estimates. Results: We identified 23 meta-analyses, included 24 primary studies, and obtained three protocols. The protocols markedly reduced methodological flexibility (range for functional outcomes 94%–99%; quality of life 58%–76%; adverse events 87%–91%). The corresponding reduction in the variation of the results varied (range for functional outcomes; 33%–78%, quality of life; 10%–24%; adverse events 10%–13%). Conclusion: Although the protocols consistently reduced methodological flexibility, the reduction in the variation (bias) of results varied. Consequently, review authors may obtain conflicting results based on the same data. We advise caution when interpreting the conclusions of meta-analyses with absent or imprecise protocols.
KW - Meta-analysis
KW - Methodological flexibility
KW - Protocol
KW - Proximal humerus fractures
KW - Variation of results
KW - Vibration of effects
U2 - 10.1016/j.jclinepi.2023.02.002
DO - 10.1016/j.jclinepi.2023.02.002
M3 - Journal article
C2 - 36822442
AN - SCOPUS:85151933379
VL - 156
SP - 95
EP - 104
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
SN - 0895-4356
ER -
ID: 369127326