Follow-up of HSDR study cohorts for investigating publication bias

This chapter presents findings from WP 4, in which we followed up cohorts of HSDR studies to ascertain their publication status and its association with their findings. Four study cohorts were assembled from the studies funded by the NIHR HSDR programme and its predecessors (n = 100), studies registered with the HSRProj database (n = 100), abstracts presented in the HSRUK (n = 50) and ISQua (n = 50) conferences, as described in Chapter 2. Based on the nature of these cohorts, we also refer to the NIHR and HSRProj cohorts collectively as the ‘inception cohort’, and the HSRUK and ISQua cohorts collectively as the ‘conference cohort’.

Verification of publication status

After the initial online search for publications, we contacted investigators for 145 of the 300 studies in order to verify their publication status. Investigators for 34 of these studies were unreachable, as we were not able to find their contact details (n = 26) or the e-mails were undeliverable (n = 8). Thus, 111 investigators were contacted. The overall response rate was 60% (67/111) and this varied by cohorts: NIHR, 79% (33/42); HSRUK, 60% (6/10); HSRProj, 50% (23/46); and ISQua, 38% (5/13). Of the 26 investigators whose contact details we could not obtain, 23 were from the ISQua cohort. Organisers of the ISQua conference could not provide the contact details to us owing to data protection regulation, but agreed to forward our information request to the abstract presenters so that they could contact us directly. However, no response was received.

Characteristics of included cohorts of HSDR studies

The characteristics of included studies stratified by different cohorts are presented in Table 10. In summary, 140 (47%) of the selected studies were intervention studies (inception cohort, 44%; conference cohort, 53%). Of the intervention studies, just over one-third (36%) were RCTs (inception cohort, 44%; conference cohort, 23%), but nearly two-thirds (62%) had a concurrent control group (inception cohort, 77%; conference cohort, 38%). Two-thirds of studies (67%) used bespoke data sources rather than using data from databases. The majority of the studies (79%) reported statistically significant findings. Findings were judged to be positive in the same proportion (79%) of studies.

Publication status of included HSDR studies

Of all 300 studies, 62% were published in academic journals (inception cohort, 70%; conference cohort, 48%). Findings for 20% of the 300 studies were available as grey literature (inception cohort, 26%; conference cohort, 8%). Fifty-three studies (18%) were unpublished (inception cohort, 4.5%; conference cohort, 44%). The pattern of publication varied between individual cohorts (Figure 11). The highest academic journal publication rate was observed in the HSRProj cohort (75%) and the lowest was in the ISQua cohort (26%). All studies from the NIHR cohort were published either in academic journals or as grey literature (technical reports available through the NIHR’s website), whereas results for 68% of the ISQua were not published.

FIGURE 11 Publication status of selected studies at follow-up. (Figure not reproduced here; see the original article.)

Factors associated with publication in academic journals

We explored factors associated with publication in academic journals for all studies and separately for intervention studies, using logistic regression. The results are shown in Table 11. For the analysis including all studies, being a study from the ISQua cohort was the only statistically significant factor associated with a reduced odds of publishing in an academic journal in univariable analysis. Neither statistical significance nor positivity of study finding was associated with publication in academic journals.

In multivariable analysis for all studies, studies from both the NIHR cohort and the ISQua cohort were significantly less likely to be published in academic journals (compared with studies from HSRProj cohort). In addition, studies with bespoke data collection were more likely to be published in an academic journal than those utilising data from routine databases or surveys. Statistical significance and positivity of finding also appeared to be associated with increased odds of publication in academic journals, having adjusted for the other factors.

Findings from analyses focusing on intervention studies identified similar factors associated with publication in academic journals, with having a concurrent control group being an additional strong predictor in univariable analysis and approaching statistical significance in multivariable analysis.

Factors associated with non-publication

In this subsection we examine factors associated with non-publication (i.e. when study findings were neither published in academic journals nor available as grey literature). Given that non-publication did not occur in the NIHR cohort, these studies were excluded from the analyses. Findings from univariable and multivariable logistic regression for all studies and for intervention studies are presented in Table 12.

For the analyses across all (intervention and association) studies, the ISQua cohort was the only statistically significant predictor for non-publication in the univariable analysis. However, in the multivariable analysis, studies from both HSRUK and ISQua cohorts were significantly more likely than studies from the HSRProj cohort to remain unpublished, and statistical significance (or positivity of findings if the variable was entered in the model in place of statistical significance) also became a significant predictor of reduced risk of non-publication.

The influence of being from a conference cohort and bespoke data collection remains similar in the univariable and multivariable analyses focusing on intervention studies (although the analyses had less statistical power). However, having a concurrent control group or being a RCT appeared to be an additional independent factor associated with reduced odds of non-publication.