Overview of systematic reviews of intervention and association studies in HSDR

This chapter presents findings from WP 2 of the project, which was an overview of 200 randomly selected systematic reviews of intervention and association studies, as described in Chapter 2.

Characteristics of included intervention and association reviews

As planned, 200 systematic reviews (100 intervention reviews and 100 association reviews) were included in this meta-epidemiological study. The characteristics of the included systematic reviews are shown for both intervention and association reviews in Table 4. The majority of the 200 systematic reviews (79%) included at least 10 studies, with association reviews more likely than intervention reviews to include ≥ 10 studies (86% vs. 71%; p = 0.01). Only 22% of all the systematic reviews performed a meta-analysis, which was more common in the intervention reviews than in the association reviews (33% vs. 10%; p < 0.0001). Of the 157 reviews that did not include meta-analysis, 90 (57%) provided reasons for this (mainly heterogeneity between studies and a small number of comparable studies).

Intervention and association reviews also differ significantly in several other characteristics, such as inclusion of study design beyond controlled trials, undertaking quality assessment of included studies, mentioning the use of systematic review reporting guidelines and GRADE, their A MeaSurement Tool to Assess systematic Reviews (AMSTAR) rating and the impact factors for the journals in which they were published (see Table 4). Just over half (52%) of the systematic reviews searched for grey and unpublished literature, and the proportions were similar in both association reviews (52%) and intervention reviews (51%). Seventy per cent of the journals in which the reviews were published endorsed a reporting guideline for systematic reviews, with similar proportions for both intervention reviews and association reviews (71% vs. 69%; p = 0.758) (see Table 4).

Publication bias

We found that 85 (43%) of the systematic reviews considered publication bias, and this was more common in intervention reviews than in association reviews (54% vs. 31%; p = 0.001). However, only about 10% (19/200) formally assessed publication bias through statistical analysis, mostly using funnel plots and related methods (Figure 5). Again, intervention reviews assessed publication bias more frequently than association reviews (14% vs. 5%; p = 0.03) (see Table 4). Five59–63 of the 19 reviews (26%) that assessed publication bias reported some evidence of publication bias. The remaining reviews mostly reported no risk or low risk of publication bias, and one reported that the funnel plot was not very informative owing to small numbers (the review included only four studies). Authors of reviews in which publication bias was mentioned but not assessed often reported conditions of using funnel plots not being met, especially insufficient number of studies and/or heterogeneity between included studies.

FIGURE 5 Methods used to assess publication bias in sampled systematic reviews. a, Nineteen reviews assessed publication bias but some reviews used more than one method of assessment. (Figure not reproduced here; see the original article.)

Factors associated with mentioning (including assessing) publication bias

Publication bias was more likely to be mentioned in intervention reviews than association reviews in the univariable analysis [odds ratio (OR) 2.61, 95% confidence interval (CI) 1.47 to 4.66]. The strongest association was observed in reviews that included meta-analysis, compared with those with no meta-analysis (OR 5.71, 95% CI 2.67 to 12.21). Mentioning publication bias was also associated with quality assessment of individual studies, authors reporting the use of GRADE, journal impact factor and reviewer reporting the use of systematic review guidelines (Table 5). Journal endorsement of systematic review reporting guidelines, search of grey and unpublished literature, design of included studies and number of included studies were not significantly associated with mentioning publication bias. In the multivariable analysis, only the inclusion of meta-analysis was statistically associated with mentioning or assessing publication bias (see Table 5). Intervention reviews were still more likely than association reviews to mention publication bias, although the relationship was no longer statistically significant (OR 1.63, 95% CI 0.85 to 3.15).

Factors associated with assessing publication bias

As with mentioning of publication bias, assessment of publication bias was most strongly associated with the inclusion of meta-analysis (OR 112.32, 95% CI 14.35 to 879.03) in the univariable analysis. Intervention reviews were more likely than association reviews to include such an assessment (OR 3.09, 95% CI 1.07 to 8.95). Inclusion of only RCTs and controlled trials, journal impact factor and reviewers reporting the use of systematic review guidelines were also associated with the assessment of publication bias (Table 6). There was no significant association between the assessment of publication bias and the number of included studies, search of grey and unpublished literature, quality assessment of individual studies or journal endorsement of systematic review guidelines. In the multivariable analysis, only the inclusion of meta-analysis and reviewers reporting the use of systematic review guidelines were significantly associated with the assessment of publication bias (see Table 6). The relationship between types of review and assessment of publication bias diminished after they were adjusted for other factors (OR 0.94, 95% CI 0.20 to 4.55). As current methods for assessing publication bias primarily rely on constructing funnel plots and/or performing related statistical tests as part of meta-analyses in which data are available from ≥ 10 studies, we further explored and illustrate the influence of these factors in Figure 6.

FIGURE 6 Assessment of publication bias in sampled reviews stratified by type of review, number of included studies and whether or not a meta-analysis was performed. (a) Intervention reviews; and (b) association reviews. PB, publication bias. (Figure not reproduced here; see the original article.)

Outcome reporting bias

Outcome reporting bias was mentioned in 34 (17%) systematic reviews and, again, this was more frequent in intervention reviews than in association reviews (30% vs 4%; p < 0.0001). All of the 34 reviews stated that outcome reporting bias was assessed as part of quality assessment of included studies. The risk-of-bias tool used most often was the Cochrane’s risk-of-bias tool (28/34).64 Two reviews used the Agency for Healthcare Research and Quality’s Methods Guide for Effectiveness and Comparative Effectiveness Reviews,65 one used the Amsterdam–Maastricht Consensus List for Quality Assessment, and the remaining three reviews used unspecified or bespoke tools. Three of the 34 reviews did not report the findings of outcome reporting bias assessment, despite having stated such an assessment in their methods section. Of the remaining 31 reviews that reported the findings, 35% (11/31) reported having at least one study with high risk of selective outcome reporting, 32% (10/31) judged all included studies as low risk, and the remaining 10 reviews (32%) had at least one study in which the authors were unable to judge the risk of bias and were classed as ‘unclear’.

Factors associated with assessing outcome reporting bias

Intervention reviews were more likely than association reviews to include an assessment of outcome reporting bias (OR 10.29, 95% CI 3.47 to 30.53). Authors mentioning the use of GRADE (OR 9.66, 95% CI 3.77 to 24.77) and inclusion of RCTs or controlled trials only (OR 7.74, 95% CI 3.39 to 17.75) were also strongly associated with assessment of outcome reporting bias. Other features found to be associated with assessing outcome reporting bias in the univariable analysis included the number of included studies, inclusion of meta-analysis, journal impact factor, journal endorsement of systematic review reporting guidelines and reviewers mentioning the use of systematic review guidelines (Table 7). Search of grey and unpublished literature was not associated with assessing outcome reporting bias. All the studies which assessed outcome reporting bias performed quality assessment of individual studies; therefore, the variable was not included in the regression analysis. In the multivariable analysis, two variables were found to be significantly associated with assessing outcome reporting bias: (1) author mentioning the use of GRADE and (2) being an intervention review (see Table 7). Lack of pre-registered protocols was frequently reported as a major barrier to adequately assessing outcome reporting bias.