In-depth case studies on the applicability of methods for detecting and mitigating publication and related biases in HSDR

In the absence of comprehensive registration of HSDR and accessible study protocols for direct identification and verification of publication and related biases, statistical methods remain key tools for facilitating the detection and mitigation of publication and related bias when quantitative evidence from HSDR is synthesised. Methods, such as funnel plots, are widely used in systematic reviews, including those of HSDR topics, as shown in Chapters 3 and 4. As with any statistical techniques, these methods are based on certain assumptions, the violation of which could impact on the validity of their results. Although there is a large volume of empirical evidence and methodological research in the use of these methods for detecting and mitigating publication and related biases in clinical research, our systematic review in Chapter 3 reveals that there is paucity of literature concerning the practical application of these methods in HSDR. This chapter aims to fill in this gap in evidence by exploring issues concerning the applicability of these methods in HSDR through the use of in-depth case studies.

Four case studies are described in this chapter:

1. case study 1 – association between weekend and weekday admissions and hospital mortality66
2. case study 2 – relationships between organisational culture, organisational climate and nurse work outcomes52
3. case study 3 – the effect of electronic prescribing on medication errors and adverse drug events67
4. case study 4 – effects of standardised hand-off protocols on information relay and patient, provider and organisational outcomes.68

The cases were chosen from topics of general interest to HSDR stakeholders, with the aims of illustrating problems, issues and potential solutions pertinent to different aspects of health services research when applying statistical methods for assessing publication and related biases.

Case study 1: association between weekend/weekday admissions and hospital mortality (Chen et al.66)

Description of the case

This systematic review evaluated international literature on the ‘weekend effect’, defined as ‘differences in patient outcomes between weekend and weekday hospital admissions’.66 The authors meta-analysed quantitative estimates of the weekend effect on mortality following hospital admissions, and explored factors that influence the magnitude of estimated weekend effect using meta-regression and subgroup analyses. The review focused on unselected (hospital-wide) admissions; studies that focused on admissions associated with specific disease condition(s) were excluded. This case study represents an association review drawing evidence primarily from observational studies carried out using routine administrative databases of health service activities.

The stated review question was:

What is the magnitude of the weekend effect associated with hospital admission, and what are the likely mechanisms through which differences in structures and processes of care between weekdays and weekends contribute to this effect?

The review was carried out as part of a mixed-methods synthesis, which also incorporated a framework synthesis (a qualitative method that allows the development and refinement of a conceptual model using emerging findings from evidence being reviewed).69

The review included 68 studies examining the weekend effect quantitatively on mortality, adverse events, length of stay and patient satisfaction. However, quantitative meta-analyses carried out in the review focused only on comparing the mortality following weekend admissions with that following weekday admissions.

Methods adopted in the original publication

Sources searched

Seven databases, MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Health Management Information Consortium, Electronic Theses Online Service (EThOS), Conference Proceedings Citation Index (CPCI) and The Cochrane Library, were searched from year 2000 onwards in April 2015. The MEDLINE search was updated in November 2017.

Inclusion of grey and unpublished literature

Conference abstracts and grey literature were excluded because of the difficulty in assessing risk of bias.

Methods for assessing publication and reporting bias

A funnel plot was constructed to assess potential publication and reporting bias, and other small study effects. A data augmentation approach,70 in which studies are assumed to be missing with probabilities that are a function of their lack of statistical significance, was adopted in order to derive a pooled estimate ‘adjusted for’ publication bias caused by the assumed missing studies.

Main review findings and findings and statements related to publication and related bias

The pooled OR obtained from a Bayesian meta-analysis for weekend mortality effect was 1.16 [95% credible interval (CrI) 1.10 to 1.23; I² 16%, 95% CrI 0% to 62%]. The funnel plot for this analysis appeared asymmetrical and some statistical heterogeneity between studies of large sample sizes was noted. Given this apparent funnel plot asymmetry, a sensitivity analysis using a data augmentation approach (which ‘adjusted for’ the asymmetry assuming that it was caused by publication bias) was performed, and this lowered the pooled OR by a small amount, from 1.16 to 1.11 (95% CrI 1.08 to 1.13). The finding suggested that, if the observed funnel plot asymmetry was caused by publication bias and if those ‘missing’ unpublished studies were to be included in the analysis, the estimated increase in the odds of death associated with weekend admissions compared with weekday admissions would reduce from 16% to 11%.

The authors also carried out meta-regression and subgroup analyses to explore potential factors influencing the estimated weekend effect. Multivariate meta-regression indicated that the estimated weekend mortality effect was larger for studies including elective admissions and smaller for studies including maternity admissions, and that studies incorporating measures of acute physiology (which are likely to better reflect the urgency and severity of a patient’s condition at admission, but which are usually not available in health service databases for administrative purpose) tend to report smaller weekend mortality effects that are closer to null. Subgroup analyses using data from both between- and within-study comparisons corroborate these findings, but also highlight that substantial statistical heterogeneity exists within individual types of admissions. The authors of the review concluded that the estimates of the weekend effect were influenced by ‘many clinical (case mix), service (e.g. route of admission) and methodological (e.g. statistical adjustment and data completeness) factors’.66 Outcome reporting bias was not assessed in this review.

Comments and further exploration of different methods

As the Bayesian meta-analysis and data augmentation approach used by the review authors are technically demanding, we reapply other commonly used (frequentist) statistical methods to explore their applicability in HSDR scenarios.

Egger’s test, Begg and Mazumdar’s rank correlation test, and trim and fill method

Egger’s regression test40 (in which the standardised effect estimate is regressed against its standard error) suggested no funnel plot asymmetry (p = 0.686), whereas Begg and Mazumdar’s rank correlation test41 (which is a test of correlation between the ranks of effect sizes and the ranks of their variances) indicated a strong small study effect (p = 0.004). These results are surprising, as the statistical power is usually greater for Egger’s regression test than for Begg and Mazumdar’s rank correlation test. The above results are likely due to the fact that most of the studies have very small standard errors (very large sample sizes). The lack of studies with large standard errors (small sample sizes) and, hence, a spread of studies with varied sample sizes makes the regression test unreliable but Begg and Mazumdar’s rank correlation test is less affected owing to its non-parametric nature. Application of the trim and fill procedure resulted in little change in the overall estimate (meta-analysis with missing studies filled, OR 1.15, 95% CI 1.11 to 1.20); this is probably because any effects of ‘filling’ missing studies would be minor considering the weight given to studies with very large sample sizes based on data from population databases.

The above example highlights an important technical requirement for the appropriate use of funnel plot and related methods, that is, the studies cannot be mostly of similar sample sizes.26 Failure to meet this requirement may occur in HSDR, for which studies often rely on analyses of data from large administrative databases.

A further issue arises when studies draw data from population databases that essentially cover the entire population or health system, such as the English Hospital Episode Statistics. The concept of sampling error that underpins a funnel plot becomes irrelevant if a study has access to and utilises whole-population data. Publication bias associated with smaller studies (e.g. based on data from a region or a few hospitals within the population) may seem irrelevant, as these studies represent repeated analyses of a fraction of the same data (admissions) and could be regarded as redundant. However, an important issue to consider here is that, unlike RCTs of interventions, in which large sample sizes tend to confer a high level of methodological rigour and hence a low level of bias, observational studies based on whole-population data do not necessarily confer the same advantage over smaller studies that are based on bespoke data collected locally. Indeed, the contrary may be true. For example, the weekend mortality effect is hypothesised to reflect the effect of differential care quality between weekend and weekday, but this assertion is valid only if potential confounders, such as differential case mix in patients admitted between weekends and weekdays, have been accounted for. It has been speculated that the weekend mortality effect demonstrated in studies based on large administrative databases may be, at least in part, attributed to unmeasured (and thus unadjusted) differences in the frailty and urgency of patients admitted at weekends and on weekdays, and emerging evidence conducted by independent research groups has shown that the estimated weekend effect attenuates when data (not usually available in administrative databases) reflecting patients’ acute physiology or admission context are taken into account.71–73

In such a scenario, in which an association may exist between estimated effect sizes and sample sizes, but for a different reason and in a different form from what is expected of funnel plot asymmetry caused by publication bias, use of funnel plot and related methods and interpretation of their findings require further caution.

A further issue, which is commonplace in HSDR, and which is another important consideration in the application of funnel plot and related methods, is the existence of heterogeneity between studies.26 The authors of this weekend effect review focused on hospital-wide studies and excluded studies that examined only condition-specific hospital admissions. This might have helped to reduce statistical heterogeneity between studies. Nevertheless, heterogeneity associated with differences in clinical (e.g. emergency vs. elective vs. maternity admissions; different geographic locations) and methodological factors (e.g. variables included in statistical adjustment; variation in outcome measures, such as inhospital, 7- or 30-day mortality, definition of weekends, etc.) between the included studies is still evident, as shown in Figure 7a, in which many studies lie outside the 95% confidence limits, indicating that they were not measuring a common effect.

FIGURE 7 Funnel plots for the meta-analysis of weekend mortality effect. (a) Funnel plot showing 95% confidence limits (dashed lines), centred at the fixed-effect pooled estimate and a fitted regression line for Egger’s test; and (b) contour-enhanced funnel plot showing contours of different levels of statistical significance, centred at null effect (OR = 1, or log-OR = 0 as shown here). (Figure not reproduced here; see the original article.)

Heterogeneity poses an issue for the use of funnel plots and related methods, as it confounds the association between effect size and sample size. For example, studies that include only elective admissions are likely to have smaller sample sizes than studies that cover all types of admissions (both emergency and elective). If the weekend effect is more pronounced among elective admissions than other types of admissions, an association between sample size and effect size may be observed, even if there is no publication bias. Attributing the funnel plot asymmetry to publication bias and attempts to ‘correct’ for this purported bias could therefore be misleading.

There are a couple of ways to further investigate whether apparent funnel plot asymmetry may be attributed to publication and reporting bias, or is likely to be caused by other factors, such as heterogeneity. One is to use a contour-enhanced funnel plot (as shown in Figure 7b). For asymmetry caused by publication bias, the ‘missing’ (presumably unpublished) studies that could have made the funnel plot symmetric would have lain within the area corresponding to non-significant results (p > 0.05, the dark grey and white areas in Figure 7b), which seems likely here. In addition, p-hacking and reporting bias could also result in a higher density of (published) studies appearing in the area corresponding to borderline statistical significance (the area within and surrounding 1% < p < 5%), which is not apparent here. Another way is to produce funnel plots by study-level features that might create distinct subgroups contributing to the heterogeneity. Figures 8 and 9 show funnel plots in which studies are sorted by whether or not they included elective admissions and maternity admissions, respectively. Overall, the asymmetry appears to be more profound in studies including elective admissions and in studies excluding maternity admissions. These suggest that the degree of funnel plot asymmetry may differ within different types of admissions, but the asymmetry observed in the pooled analysis is unlikely to be purely caused by inclusion of different types of admissions.

FIGURE 8 Funnel plot with pseudo-95% confidence limits of the weekend effect, sorted by whether individual studies include or exclude elective admissions. (Figure not reproduced here; see the original article.)

FIGURE 9 Funnel plot with pseudo-95% confidence limits of the weekend effect, sorted by whether individual studies include or exclude maternity admissions. (Figure not reproduced here; see the original article.)

Meta-regression

A further approach to investigating publication and related bias is to use meta-regression to examine the presence or absence of an association of between effect sizes and the sample sizes of individual studies (this is similar to Egger’s regression test), while simultaneously controlling for other study-level variables. The authors of the systematic review carried out a meta-regression, although sample size was not included as one of the covariates in the analysis. We replicated the meta-regression, including variables initially included by the review authors, and then added the total study sample size as a binary variable (categorised into two groups of greater or less than 3 million hospital admissions). The findings are shown in Table 8.

The analysis suggested that estimates obtained from larger studies with a total sample size of > 3 million hospital admissions showed a weekend effect approximately 8% larger than those obtained from studies of smaller sample sizes. This is in contrast with evidence from randomised trials, which tend to show that larger studies tend to report smaller effects.12 A possible explanation is that smaller studies were able to make better statistical adjustment, with additional data not available from administrative databases. However, the completeness of statistical adjustment (based on review authors’ judgement) was taken into account in the meta-regression reported in the review. Although the level of statistical adjustment did not appear to significantly influence effect size in the analysis, it remains a possibility that the review authors’ judgement did not properly capture adequacy of statistical adjustment and, hence, the existence of residual confounding between bias related to adjustment, sample size and effect size. Finally, the finding from this meta-regression is exploratory. It could simply be a chance finding and, therefore, needs to be treated with great caution.

Application and applicability of p-curve

The risk of p-hacking may be particularly high in studies involving analysis of large data sets with many different variables, such as the ones included in the weekend effect systematic review. We therefore used the p-curve technique to explore the potential occurrence of p-hacking.2 First, we calculated corresponding p-values for the 119 estimates (adjusted ORs and their CIs) of the weekend effect included in the above meta-regression, using the approximation method described by Altman and Bland,74 and then created a p-curve using the online application provided by Simonsohn et al.43 The p-curve (Figure 10) suggests that, based on the estimates from included studies, the weekend effect does exist (the curve peaks on the left), but there is no evidence of p-hacking (there is no peak on the right close to p = 0.05).

FIGURE 10 p-curve for estimates of the weekend effect included in meta-regression. The observed p-curve includes 82 statistically significant (p < 0.05) results, of which 78 are p < 0.025. There were 37 additional results entered but excluded from p-curve because they were p > 0.05. (Figure not reproduced here; see the original article.)

The p-curve is a relatively new tool and, thus, its validity and utility are still subject to debate and require further empirical testing.75,76 Our exploratory analysis provides some assurance that p-hacking is not evident among the literature included in the weekend effect systematic review. It is plausible that with very large sample sizes available from administrative databases and myriad variables/groups that can be analysed, it would not be necessary to resort to p-hacking to obtain a statistically significant finding to report. On the other hand, the strong right-skewness does not necessarily confirm the existence of a true ‘weekend effect’; as Bruns and Ioannidis76 demonstrated in their simulation study, the p-curve cannot distinguish true effects from null effects in the presence of even a minimal level of unaccounted confounding, which is a likely scenario in the case of studies of the weekend effect.

Key summary points

• This case study explores issues that might need to be considered when using statistical techniques to assess publication and reporting biases in systematic reviews in which most of the included studies are based on analyses of administrative databases of health service activities.
• Funnel plots highlighted the lack of studies of smaller sample sizes and showed a high level of statistical heterogeneity between studies of very large sample sizes. Both features may compromise the validity of funnel plots as a tool for detecting publication bias.
• Two commonly used tests (Egger’s regression test and Begg and Mazumdar’s rank correlation test) produced discrepant results. The use of Egger’s regression test might be particularly problematic given the lack of small studies.
• In contrast to evidence from RCTs, in which larger studies tend to report smaller effects, studies of larger sample sizes tended to report larger effects in this case study. The possibility that large database studies might suffer from a higher level of bias (due to lack of required information to adequately adjust for confounders) than smaller studies (which may be able to access detailed clinical data) could not be ruled out.
• Adjustment of pooled estimates based on imputation of ‘missing’ small studies had little impact, given the negligible weight assigned to them compared with large database studies. Such adjustment might also be inappropriate if the underlying assumption that funnel plot asymmetry was caused by publication bias was invalid.
• Investigation of publication bias might be secondary to identification of the sources of statistical heterogeneity, which was of primary importance in the scenario described in the case study.

Case study 2: relationships between organisational culture, organisational climate and nurse work outcomes (Fang52)

Description of the case

This case study is based on a Doctor of Philosophy (PhD) dissertation, in which a systematic review was undertaken to assess the relationships between organisational culture, organisational climate and nurses’ job satisfaction and nurses’ turnover in hospitals the USA, and to examine moderators that influence such relationships.

Organisational culture was defined in the systematic review as ‘a pattern of shared assumptions among organizational members that guides the formation of managerial practices and members’ behaviours to achieve external adaptation and internal integration of an organization in order to maintain the organization’s survival and growth’.52 To facilitate analyses, the author mapped different measures of organisational culture to the three cultural patterns identified by the Organisational Culture Inventory:77 (1) constructive culture, (2) passive and defensive culture, and (3) aggressive and defensive culture.

Organisational climate was defined as ‘members’ perceptions of managerial practices in the work unit that direct members’ work-related behaviours’ in an organisation.52 Measures of organisational climate were mapped to the five elements described in the ‘model of climate, culture, and productivity’ proposed by Kopelman et al.:78 (1) goal emphasis, (2) means emphasis, (3) reward orientation, (4) task support and (5) socioemotional support. An additional measure of ‘global climate’ was also adopted.

Overall, the review covered 23 relationships between different measures of organisational culture and climate and job-related outcomes for nurses:

• association between each of the three patterns of organisational culture described above and job satisfaction
• association between each of the six elements/measures of organisational climate described above and nurse job satisfaction
• association between each of the three patterns of organisational culture and nurse turnover, measured at two different levels (rate of turnover at hospital level and turnover intent at individual level), thus creating six pairs of association)
• association between five of the six elements/measures of organisational climate (excluding goal emphasis climate as no study was found) and nurse turnover (all measured at individual level)
• association between the three patterns of organisational culture and global climate (as no study was found for individual elements of organisational climate).

When data were available, the contribution of the various study characteristics and contextual factors towards variation in the magnitudes of these associations was explored using subgroup analyses. Factors that were examined included measurement instrument (Organisational Culture Inventory or other), publication year, whether or not hospital was under redesign, number of hospitals (one or more than one), professional level (registered nurse, manager or mixed), type of nursing unit (medical, surgical, intensive care unit or obstetrics, gynaecology or multiple), whether or not psychiatry units were included, US region, whether or not the survey was given by manager, whether or not the survey was collected at work, adoption of random sampling and quality rating of the study. Estimates of association from individual studies were transformed into a standardised effect size and pooled using both fixed- and random-effects modelling for each of the associations examined. Effect sizes were interpreted as < 0.30, small effect; between 0.30 and 0.49, medium effect; and ≥ 0.50, large effect.

Methods adopted in the original publication

Sources searched

Thirty-five computer databases were searched. Reference lists of primary studies included in the review were also checked. For publications in which the desired effect sizes were not provided, authors were searched in computer databases to identify other relevant publications and authors were also contacted by letter to request relevant data, when necessary.

Inclusion of grey and unpublished literature

The review included both published and unpublished studies, although unpublished studies were limited to doctoral dissertations. The dissertations were identified from computer databases. Overall, the review included 32 primary studies, covering 15 unpublished doctoral dissertations and 17 published journal articles.

Methods for assessing publication and reporting bias; subgroup and sensitivity analyses

When data were available, a funnel plot was constructed and a ‘fail-safe N’ was calculated for each of the associations examined. Fail-safe N is the number of studies with a null result (or a specified effect size) that would be required in a meta-analysis to render a statistically significant finding insignificant.79 A large fail-safe N suggests that a large number of (unpublished) studies with null result (or a specified effect size) would be required to overturn the observed effect and association and, therefore the finding is robust and is unlikely to be influenced by publication bias (and vice versa). In addition, the author undertook subgroup meta-analyses based on the sources of included studies (journal articles vs. doctoral dissertations), and thus provided an opportunity for comparing published and unpublished studies.

Main review findings and findings and statements related to publication bias

Of the 23 associations examined, 14 were found to be statistically significant. Of these, the effect size was considered large for only one association (global climate and job satisfaction, effect size 0.51, 95% CI 0.41 to 0.60, I² = 88%), medium for seven and small for six associations (see Appendix 7 for further details).

The total number of studies available for each association ranged from one to 12, and the original author was unable to formally assess publication bias for 13 of the associations owing to the small number of studies. For the remaining 10 associations, the author constructed a funnel plot for each of them and calculated a fail-safe N for which the pooled estimate of the association was found to be statistically significant. In addition, the author compared the pooled effect size from journal articles with that from (unpublished) doctoral dissertations in subgroup analyses and calculated a fail-safe N for each of these two subgroups. The findings are summarised in Table 9.

Of the 10 associations for which publication bias was assessed, the original author suggested that there was no publication bias for six of these, according to funnel plots, comparison of pooled estimates between journal articles and doctoral dissertations, and the values of fail-safe N calculated (see Table 9). Publication bias was suspected for three of the associations, and for one of the associations the author indicated that more studies were required. The pool estimates between published journal article(s) and doctoral dissertation(s) differ significantly (based on test of homogeneity between subgroups) for two of the 10 associations. However, in both cases there was only one study in the journal article subgroup and, therefore, the findings need to be interpreted with caution. In addition, in both cases the estimated effects were in the same direction (showing a positive association), and the estimates based on unpublished doctoral dissertations showed larger effects than those based on the published journal article. Consequently, publication bias was not suspected in either case. The author also cautioned that the meta-analyses might be susceptible to reporting bias.

Comments and further exploration of different methods

This case study illustrates a common scenario in HSDR, in which associations between abstract concepts, such as organisational culture, organisation climate and job satisfaction, are explored. As exemplified in the case, quantitative measurement of these concepts and synthesis of evidence from these association studies face many challenges, including diverse measurement tools that may cover different domains or components of a concept, measurements made at different levels (e.g. individual or hospital level), lack of standardised terminology, diverse contexts in which the associations are measured and the large number of factors that could potentially modify these associations. Consequently, evaluation of associations between an apparently small number of concepts could rapidly expand into examination of a large number of associations between variables that represent different domains and measurement approaches for these concepts. There are some key implications in relation to publication and related bias:

• The large number of associations that may be examined in individual studies creates a theoretical risk of selective outcome reporting of only statistically significant findings. However, a possible counterargument would be that the examination of a large number of association increases the chance that at least one statistically significant finding would be observed and, thus, publication bias associated with completely null findings would be reduced. In addition, exploration of such associations is often guided by theories, and observations both conforming to and disagreeing with existing theories may be considered of interest, irrespective of statistical significance.
• The large number of decisions that researchers have to make during the data analysis process, such as the choice of measurement instruments and unit, inclusion, exclusion and grouping of individuals in the collected sample, selection of variables for statistical adjustment and so forth, creates a ‘garden of forking paths’, as described by Gelman and Loken,80 in which researchers have a large, and potentially unknown, ‘degree of freedom’ in making decisions contingent on data in the process of data analyses, resulting in a problem similar to multiple comparison even without p-hacking. p-values (statistical significance) of analyses from such exploratory analyses should therefore be interpreted more cautiously.
• Attempts to divide evidence and studies into more refined and homogeneous groups based on conceptual domains or contextual factors during evidence synthesis tend to result in a smaller number of studies being included in each meta-analysis. This reduces precision and increases uncertainty of individual effect estimate, and hampers the applicability and usefulness of statistical methods such as funnel plots, for detecting publication and reporting bias.

Given the above, caution is required in the interpretation of findings in the assessment of publication bias presented in this case study. First, for the associations for which a funnel plot was constructed by the original author, the number of studies included for each association was no more than 12, with several associations including less than the minimum number of 10 studies recommended for using funnel plot.26 The determination of funnel plot symmetry was likely to be very subjective and potentially unreliable.

Second, the method of fail-safe N focuses primarily on statistical significance, and the calculation is heavily dependent on the assumed effect size of unpublished studies (zero in the current case study). Its use is therefore not recommended by the Cochrane Collaboration, and judgement of the presence and absence of publication bias based on fail-safe N was better avoided.

Despite these caveats, the original author undertook a very comprehensive search of the literature, and the comparison of data from published journal articles with unpublished doctoral dissertations provided a unique opportunity and stronger evidence for assessing potential publication bias. Although there appeared to be no indication of a high level of publication bias mediated by observed strength of association, the volume of evidence was inadequate for many of the associations examined and, therefore, the lack of evidence of publication bias should not be equated to evidence of no publication bias until further evidence has been accrued.

Case study 3: the effect of electronic prescribing on medication errors and adverse drug events (Ammenwerth et al.67)

Description of the case

This case study was based on a systematic review that aimed to determine the effect of electronic prescribing on the risk of medication errors and adverse drug events, and to analyse the factors contributing to those effects. The review covered all settings (e.g. inpatient and outpatient) and all patient groups, but focused on systems of which physicians were the primary users (i.e. excluding studies with pharmacists or nurses as the primary users). Studies which provided quantitative empirical evidence on the effectiveness of electronic prescribing systems on the outcomes of interest were included. RCTs, non-RCTs, before-and-after and time series were all eligible, whereas studies based on laboratory experiments and simulation, as well as those that did not focus primarily on medication errors or adverse drug events, were excluded. This case study provides an example from the prominent area of using technologies to improve service delivery. Components of such interventions may vary substantially between different locations and may evolve over time. The interventions may have a generic effect across different services and patients, but their effects may be difficult to measure owing to the infrequent occurrence of the events (which could have serious consequences) that they aim to prevent or reduce, such as medication errors and adverse events.

Methods adopted in the original publication

Sources searched

Electronic databases MEDLINE and EMBASE were searched, and the authors examined reference lists of relevant reviews identified from the Cochrane Database of Systematic Reviews. Three journals were also hand-searched: Journal of the American Medical Informatics Association, International Journal of Medical Informatics and Methods of Information in Medicine (1990–2006). In addition, reference lists of all retrieved articles were examined. Searches were conducted in 2006 without any language restriction.

Inclusion of grey and unpublished literature

The review included only published studies.

Methods for assessing publication and reporting bias; subgroup and sensitivity analyses

Methods of assessing publication bias were not described in the published article of the systematic review. However, the authors mentioned publication bias in their discussion and highlighted this as a potential issue in more detail in a separate publication.81 Outcome reporting bias was not assessed.

Several subgroup analyses were performed for the outcome of medication errors using subgroups defined a priori, including the clinical setting (inpatient, outpatient or intensive care), patient group, type of drugs, type of system (home grown or commercial), functionality (no, limited or advanced decision support), study design and method to detect errors in order to explore factors that may influence intervention effectiveness.

Main review findings and findings and statements related to publication bias

The literature search identified 172 evaluation studies, but only 27 met the review’s inclusion criteria. Fifteen of these evaluated medication errors only, two assessed adverse drug events only and 10 reported on both medication errors and adverse drug events. The majority of the studies were before-and-after and time series studies. Only two included studies were RCTs.

Meta-analyses were not carried out due to substantial heterogeneity between included studies. Twenty-three studies (out of 25 studies) on medication errors showed statistically significant reduction with the intervention, but the risk ratios compared with control groups ranged from 0.01 (99% relative risk reduction) to 0.87 (13% relative risk reduction). Six out of nine studies on potential adverse drug events (defined as ‘medication errors with significant potential to harm a patient that may or may not actually reach a patient’) reported a statistically significant risk reduction, with risk ratios ranging from 0.02 to 0.65. Four of the six studies with data on adverse drug events reported a statistically significant risk reduction, with risk ratios ranging from 0.16 to 0.70. Although no meta-analysis was conducted, the authors plotted effect estimates for different subgroups separately in forest plots. These suggested that intervention effects may differ between home-grown and commercial systems, between electronic prescribing and hand-written ordering, between systems with different levels of decision support functions and manual chart review and automated database analysis as the methods for detecting medication errors. Nevertheless, substantial heterogeneity persisted within most of the subgroups.

No findings in relation to publication bias were reported in the review article, but the authors stated that ‘in our CPOE [computerised physician order entry] meta-analysis, the funnel plot showed a slight asymmetry, which may indicate a potential publication bias’ in the related publication.81

Comments and further exploration of different methods

Electronic prescribing systems are one of the archetypal generic service interventions that have been evaluated in HSDR. Quantitative assessments of the effectiveness of such interventions and synthesis of this body of evidence are fraught with challenges, including lack of standardised terminology and outcome measures, inadequate quality of study design and reporting, and a high level of heterogeneity between studies, as the authors highlighted in their reflection of undertaking the systematic review.81 In addition, they pointed out several pertinent issues which, in combination with or as a contributor towards publication bias, may lead literature included in such a review to largely represent the maximum effectiveness of the intervention in favourable conditions, rather than a representative range of possible (both positive and negative) effects. These include preferential publication and reporting of positive rather than negative results by developers or sponsors of the technologies due to conflict of interest; researchers reporting and publishing findings only after a (potentially long) optimisation process has been completed; and the tendency of published findings to be generated from atypical, so-called ‘alpha sites’,82 where staff are highly motivated and strong technical and financial support is available. Findings from meta-analyses of the evidence, if undertaken, are therefore likely to overestimate intervention effectiveness in a representative setting.

The high level of heterogeneity among studies included in this systematic review means that application of funnel plots and other statistical methods is unlikely to be informative. The number of available studies is too small for the application of techniques such as meta-regression. Consequently, being aware of the caveats highlighted above when interpreting evidence may be the best approach for evidence reviewers and users before a comprehensive study registry is established, which the authors of the systematic review advocated.81

Case study 4: effects of standardised hand-off protocols on information relay, patient, provider and organisational outcomes (Keebler et al.68)

Description of the case

This systematic review was conducted to determine whether or not implementing a standardised hand-off (handover) protocol improves the process of hand-off (with regard to information being passed on between care providers), and to evaluate its effects on patient, care provider and organisational outcomes.

The authors did not place any restrictions on clinical areas, type of hand-off protocols (e.g. this could be mnemonics, checklists, protocols or computerised sign-out programs), study dates or study design, other than a requirement that a study needed to have described an empirical investigation of a standardised protocol and have reported sufficient data for calculating effect sizes.

Measures reported in studies included in the review were classified into four broad categories as defined below.

Hand-off outcomes

Measure of the amount of information passed between care providers (e.g. adequacy of patient information provided to on-call residents).

Patient outcomes

Outcomes related to patients’ health or their opinions about their care (e.g. cardiopulmonary bypass time, monitor alarms checked and adapted to patients’ status, mortality, length of stay and satisfaction).

Provider outcomes

Measures of providers’ performance or opinions, including their response to hand-off implementation (e.g. satisfaction with the sign-out process, time spent on tasks and effects on interruptions).

Organisational outcomes

Outcomes related to higher levels of the organisation, including measures of culture and organisation-wide improvements (e.g. leadership and culture to learn from errors).

Hedges’ g,83 which is one form of standardised mean difference calculated with a correction for bias associated with small sample sizes, was computed for all reported outcomes, including both continuous and binary measures. Meta-analyses were carried out using a random-effects model.

The review included 36 studies, all of which were of uncontrolled before-and-after design. The authors reported the total sample size in terms of ‘measurement data points’, with 106,724 pre-intervention measures and 97,642 post-intervention measures.

Methods adopted in the original publication

Sources searched

Four databases, MEDLINE, Sage, EMBASE and PsycINFO, were searched from inception up to and including March 2015.

Inclusion of grey and unpublished literature

The authors initially did not search for grey literature, but after finding only a small number of studies with null findings from the initial literature search they conducted another search on greylit.org, but found no relevant articles. The authors also reported searching the World Health Organization database of registered clinical trials, but found no study with relevant data.

Methods for assessing publication and reporting bias

Funnel plots were constructed to assess publication bias. The funnel plots were assessed visually and by Egger’s test.

Main review findings and findings and statements related to publication bias

Among the 36 included studies, 27 studies assessed hand-off information (with 273 different measures and variables reported among them), 16 studies assessed patient outcomes (with 66 different measures reported), 13 studies assessed provider outcomes (with 85 different measures reported) and three assessed organisational outcomes (with 26 different measures reported). Pooled estimates suggested that standardised protocols are effective in improving hand-off outcomes (Hedges’ g 0.71, 95% CI 0.63 to 0.79, I² = 83%), patient outcomes (Hedges’ g 0.53, 95% CI 0.41 to 0.65, I² = 97%), provider outcomes (Hedges’ g 0.51, 95% CI 0.41 to 0.60, I² = 66%) and organisational outcomes (Hedges’ g 0.29, 95% CI 0.23 to 0.35, I² = 14%).

The authors reported that funnel plots indicated publication bias for all four outcome categories. Results of Egger’s test were statistically significant for all four corresponding funnel plots (p < 0.001).

To further assess the potential impact of publication bias, the authors examined the articles included in the meta-analyses to assess negative outcomes reported in the studies. They found that, although many articles reported mainly positive effects, there are some instances in which the intervention resulted in negative outcomes. Negative outcomes are reported in < 20% of the total variables analysed for each outcome measure, 12% for hand-off information, 19% for provider outcomes, 18% for patient outcomes and 3% for organisational outcomes. It was not clear if low occurrence of negative outcomes were due to under-reporting of negative outcomes.

Comments and further exploration of different methods

This review represents a laudable effort to synthesise available evidence concerning the effectiveness of hand-off protocols on a comprehensive range of outcomes. The findings of predominantly positive effects on the handover of information from uncontrolled before-and-after studies agree with the findings of other systematic reviews on similar topics,84,85 but the overall effectiveness of these interventions, in particular their effects on patient outcomes, has been questioned, particularly when the quality of evidence is taken into account. It is beyond the scope of this case study to critically appraise these systematic reviews and underlying evidence. However, two issues related to the methodological approach adopted in the example selected for the case study merit further discussion. The first issue concerns the approach to meta-analyses and corresponding funnel plots and the second issue relates to the use of proportion of negative outcomes as an indicator for publication bias. We explicate these issues below.

It appears that, within each outcome category, the review authors had included multiple effect sizes for multiple outcome variables in the same meta-analysis and funnel plot. For example, although only three studies reported data for outcome variables within the organisational outcome category, the meta-analysis and funnel plot for this outcome category included 26 effect sizes associated with 26 different organisational outcome measures reported in these studies. Most (23/26) of the effect sizes and outcome measures were contributed by one of the three studies; the other two studies contributed only one and two effect sizes. This approach departs from the usual practice for meta-analysis, where each study contributes one effect size from one outcome measure towards the pooled estimate to ensure independence of individual observations and to avoid double counting a study due to reporting of multiple outcome measures. The pooled effect sizes reported in this review are likely to have been driven, at least in part, by studies that measured and reported a large number of items and outcome variables within a given outcome category and, therefore, were potentially inaccurate. Similarly, as data points included in the funnel plots were not independent, the shapes of their distributions and the results of Egger’s test based on the data could be unreliable.

The authors noted that only a very low proportion of results across all outcome categories were ‘negative’, and suspected publication bias as a potential reason behind this. As we highlighted in Chapter 3, the proportion of studies or outcomes with negative findings is not a reliable measure of publication and related bias, as this proportion depends on the nature of the intervention and outcome measure; one would expect an approximately equal split of positive and negative findings from a random sample of studies only if an intervention truly had no effect on the specific outcome. Nevertheless, the authors’ further examination of negative findings included in their meta-analyses unearthed the important issue of including heterogeneous outcome measures (within an outcome category) in the same meta-analysis, as we pointed out above. In this case, it appears that the negative findings were mainly associated with specific items or measures, such as delay and increased duration of handover activities, and omission of relevant information items not explicitly listed on the hand-off checklist. Rather than publication bias or selective outcome reporting being the reason behind the low proportion of negative findings, the low proportion of negative findings might have arisen from the standardised hand-off protocols having positive impacts on most aspects of measured processes or outcomes, or alternatively because few studies had measured the outcomes on which standardised hand-off protocols had negative impacts. Nevertheless, the findings showed that potentially unintended effects could have gone unnoticed had people focused on the pooled summary estimates derived from combining different outcome measures together.