In this chapter, we describe the methods and process applied for the systematic review and NMA. The protocol for the study was registered with PROSPERO (CRD42016048184). Changes from the protocol and clarifications relating to inclusion criteria are listed in Chapter 1 and are described in detail in Table 1. They are also briefly noted throughout this chapter.
Patient and public involvement
As part of the systematic review process, we consulted with the Advice Leading to Public Health Advancement (ALPHA) research advisory group of young people aged 14–21 years, facilitated by the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer) Centre at Cardiff University.55 The aim was to identify health and social outcomes of importance to young people that could be considered in the systematic review. On the basis of this focus group, we included a post hoc outcome looking at the impact of prevention interventions on the stigma associated with mental disorders. We also met with members of The Caerphilly County Borough Parent Network to explore their views on young people’s mental health and the role of schools in preventing and identifying problems.56 There were no outcomes from that focus group that fed directly into this report.
Methods for the systematic review of effectiveness studies
Criteria for considering studies for this review
In the absence of direct head-to-head evidence comparing interventions, indirect comparisons can be made across a connected network of interventions. For example, the effect of intervention C against intervention B (BC) can be obtained indirectly from the effect of C against intervention A (AC), minus the effect of B against A (AB). The combination of indirect and direct evidence across a network of intervention comparisons is known as a NMA.
The following eligibility criteria were specified to address the key consistency assumption required for a valid NMA. In a three-intervention network, the consistency assumption requires the true BC intervention effect estimated in the B versus C trials to be the same as the BC intervention effect estimated by the A versus C and A versus B trials (had they also included the B and C arms).57 For this to hold, one should check that the populations included across all trials in the analysis are comparable to each other, with respect to any potential effect-modifying characteristics.58 This requirement has been conceptualised as ‘joint randomisability’ of the interventions for the target population.59 ‘Joint randomisability’ implies that a hypothetical, multiarm trial of every included intervention would be reasonable, in principle, and that all participants would be randomisable to any of the interventions included.60 This requires clearly and specifically defined inclusion criteria, to ensure the included studies, populations and interventions are sufficiently comparable. Further details on NMA are provided in Methods for the evidence synthesis of effectiveness studies.
Study design
Parallel-group RCTs and quasi-randomised controlled trials were eligible for inclusion. We defined quasi-randomised trials as those for which allocation was based on a pseudo-random sequence, such as the order in which participants were recruited or their date of birth. Both individually randomised and cluster randomised trials were eligible for inclusion. We did not plan to exclude crossover trials, but only the first period was considered eligible for inclusion.
Population
We followed the NAM’s definition of primary prevention, which refers to universal, selective and indicated populations (see Figure 1).27 Briefly, universal prevention addresses whole populations not defined on the basis of risk; selective prevention is targeted at subgroups with a higher than average risk of developing a mental disorder; and indicated prevention is targeted at high-risk subgroups and/or individuals with detectable, but subclinical, symptoms of a mental disorder. In the first instance, we used author-reported classifications of the intended prevention level. However, when interventions were delivered to a whole class or school with the same at-risk characteristic (such as schools in low-income areas), they were combined with universal prevention. Studies were excluded if the intervention was described by the author as indicated prevention, but baseline symptoms scores were suggestive of clinically meaningful symptom levels (see Definition of disorder).
Age
As noted in Chapter 1, the eligible age range was modified during the screening stage of the review. Further details of this change to protocol are provided in Table 1.
Studies including participants between the ages of 4 and 18 years (age at study recruitment), in full- or part-time education, were eligible for inclusion. The lower age limit was set in accordance with the de facto school starting age in England and Wales. However, owing to global differences in school starting age, we determined that studies implemented in preschool settings would be eligible for inclusion if (1) the mean age of participants was 5 years or (2) the majority of enrolled children were aged 5 years at the time of the baseline assessment. The upper age limit reflects the minimum age of entry to higher (tertiary) education in England and Wales. However, studies were eligible for inclusion if the mean age of participants at baseline was ≤ 19 years. Studies targeted at young people not in education or training were excluded.
Definition of disorder
The original proposal listed CMDs among CYP as obsessive–compulsive disorder, phobia, post-traumatic stress, panic disorder, anxiety, depression and conduct disorder. However, we anticipated focusing on anxiety, depression and conduct disorder, as they are the most common, and we expected the greatest number of studies for these conditions. The structure and connectivity of a network are important considerations in a NMA, as estimates can be obtained only for connected networks, and sparsely populated networks with few participants can lead to imprecise estimates.57 Further details of this clarification to the original protocol are provided in Table 1.
Studies were included if they were explicitly aimed at the primary prevention of anxiety, depression and/or conduct disorder as operationalised according to categorical or clinically referenced definitions of disorder [e.g. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5)].61 This was to differentiate studies addressing related mental health constructs, such as emotional health or well-being, which were excluded (see Interventions and comparators). Studies were eligible if they focused on either prevention of disorder onset or prevention of symptoms.
However, studies that addressed individual symptoms, or combinations of symptoms, associated with anxiety, depression and conduct disorder, but without explicitly linking these to a clinically identifiable disorder, were excluded. For example, interventions to prevent insomnia, rumination or low self-esteem were excluded, even though these symptoms are associated with depressive and/or anxiety disorders, and interventions to prevent truancy, bullying or aggressive behaviour were excluded, even though these behaviours are associated symptoms of conduct disorder. Studies were included only if they addressed the whole condition, not individual symptoms or combinations of associated symptoms. We consulted trial registrations and protocols, when available, for further information.
Studies in indicated populations were eligible if participants had subclinical mental disorder symptoms as identified by a screening instrument, an interview or a teacher referral. Subclinical symptoms could be defined in reference to diagnostic criteria such as the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10)- or DSM-5-categorised disorders, or ‘in research’ via use of a disorder-specific screening instrument, for example the Children’s Depression Inventory or Revised Children’s Manifest Anxiety Scale. The boundary between indicated prevention and early intervention (treatment) is debated,26,27 with no definitive diagnostic threshold. Studies were excluded if baseline measures were suggestive of clinically meaningful symptoms in > 40% of participants, even if the study had been defined as indicated prevention by the author. Young people at risk of comorbid mental health disorders were eligible for inclusion. However, we excluded studies for which > 40% of participants had an identifiable or pre-existing mental disorder. To ensure a clinically homogeneous population for analysis, studies in which the whole population had a diagnosis of attention deficit hyperactivity disorder (ADHD) or an autism spectrum disorder were excluded, as these form distinct diagnostic categories.
Setting
As noted in Chapter 1, the operationalisation of setting was clarified from the original proposal. Full details are provided in Table 1.
Interventions implemented in an educational setting were eligible for inclusion. For the purposes of analysis, this was operationalised as being primary, secondary or tertiary educational settings. However, to accommodate global differences in educational systems, we did not restrict to interventions implemented in these settings if the age eligibility criteria were met. For example, an intervention delivered in a kindergarten setting would be eligible for inclusion if the mean age of participants was 5 years, or the majority of enrolled children were aged 5 years at the time of the baseline assessment. Interventions implemented in school-affiliated settings (e.g. after-school and holiday clubs) were eligible for inclusion if they were implemented on school grounds. This clarification from the original protocol is explained in Table 1. Studies that used schools as the source of recruitment but for which the intervention was not school based were excluded.
Health service settings, such as primary care and outpatient and inpatient settings, were excluded. Interventions implemented in young offender institutions and for looked-after children in residential care were also excluded. Interventions implemented in low-income countries (LICs), middle-income countries (MICs) or high-income countries (HICs) were eligible, as defined by 2017 World Bank classifications.62
Interventions and comparators
Inclusion criteria
Interventions were eligible for inclusion if they addressed a universal, selective or indicated population, and the primary study aim was to prevent anxiety, depression or conduct disorder.
Eligible intervention types included psychological and psychosocial, educational or physical interventions that were implemented in educational settings, either individually or in groups. Inclusion was not restricted by mode of delivery. Interventions were included if delivered by peer educators, teachers, youth workers, clinicians, health visitors, school nurses or counsellors. However, digital and online interventions were eligible for inclusion only if they were primarily delivered in the education setting or were a clear adjunct to a wider programme delivered in the school/educational setting (e.g. as homework).
All relevant non-pharmacological control interventions were considered eligible for inclusion, for example standard provision/usual curriculum, waiting list, no intervention, attention control or ‘placebo’ interventions, and other active psychological and psychosocial, educational or physical interventions. Further details on active and control interventions are provided in Chapter 3.
Exclusion criteria
The following intervention exclusion criteria were informed by the need to ensure the validity of the NMA. The key assumption underpinning a NMA is described in the Cochrane handbook59 as ‘transitivity’, but is also known as the consistency assumption. Regarding the interventions included in the network, transitivity requires ‘all competing interventions of a systematic review to be jointly randomizable’59 and that intervention A is ‘similar’ when it appears in the A versus B and A versus C studies.
Assessment of transitivity for public health interventions is not straightforward. To ensure the validity of the NMA, we included only interventions for which the primary aim in a given study was to prevent anxiety, depression or conduct disorder. Unless the study was explicitly focused on disorder-specific prevention, then mental health promotion, awareness, literacy or information interventions were not eligible for inclusion. Social and emotional well-being and positive psychology interventions to improve mental well-being were also excluded, as research suggests that well-being is a separate construct to mental ill health.63,64 When possible, we consulted trial protocols or registrations if this was ambiguous in the publication. Interventions designed to target prevention of behaviours or social problems that might be on the causal pathway to a mental disorder (e.g. prevention of stress, anti-bullying interventions, substance abuse prevention) were also excluded. Similarly, classroom management and school readiness interventions were not eligible. ‘Parenting’ interventions such as parent management training or parenting skills interventions were not eligible for inclusion. However, interventions that took place in schools, with a parenting component, were eligible if the parenting component was not > 50% of the whole intervention.
Outcomes
According to the NAM classification of primary prevention, the overall, longer-term aim of preventative interventions ‘is the reduction of the occurrence of new cases’26 of mental disorders. However, it also recognises the importance of shorter-term prevention in terms of reducing symptoms, which, in turn, may delay or reduce the risk of the onset of the disorder. All are considered beneficial at a population level and are ‘worthwhile goals of prevention’.26 In this report, we focus on the effect of prevention interventions on symptoms of anxiety, depression and conduct disorder. The main outcome was prevention or reduction of disorder-specific symptoms for self-reported anxiety, depression and conduct disorder.
All validated disease-specific measurement scales for CYP were eligible for inclusion. When studies reported multiple outcome measures, we applied a prespecified hierarchy to select the most appropriate outcome for analysis from each study (see Appendix 1). We did not exclude studies reporting a composite mental health scale from the systematic review [e.g. the Strengths and Difficulties Questionnaire (SDQ)]; however, they were not combined with disorder-specific scales in the main NMA. In a change from protocol (see Table 1), a post hoc analysis for composite ‘internalising’ symptom scales was conducted. For example, measurement scales reporting a total or combined score across depression and anxiety symptoms, such as a total Revised Children’s Anxiety and Depression Scale (RCADS) score, or a composite outcome such as the SDQ ‘emotional symptoms’ subscale, were included in this post hoc outcome.
The following additional primary and secondary outcomes were also specified a priori. However, in the absence of a core outcome set65 or guidelines for the selection of measurement scales for school-based mental health interventions,66 we determined that an inclusive approach to additional primary and secondary outcomes was appropriate. Therefore, we did not specify how these outcomes should be measured in advance, or which scales should be used. Instead, we extracted outcomes as reported by the study authors.
Additional primary outcomes were as follows:
- self-reported well-being, as defined by study author(s)
- self-reported suicidal ideation, behaviour and self-harm, as reported by study author(s)
- intervention impact on inequalities in health, as defined by study author(s).
The primary time point of interest was immediately post intervention. However, as sustainability of intervention effect is an important question for public mental health,67 we also report results for mid-term (6–12 months) and longer-term (13–24 months) follow-ups. If studies had a follow-up of ≥ 25 months, these results were also extracted.
Secondary outcomes of interest were as follows:
- Mental health-related stigma, as defined by study author(s). During our initial patient and public involvement (PPI) focus groups, reducing the stigma associated with mental health problems was identified as an important outcome for young people.
- Acceptability of an intervention to young people, as reported by the study author(s).
- Parent-reported prevention or reduction of disorder-specific symptoms, as reported by the study author(s).
- Self-reported problem behaviour, such as substance use or involvement in violence.
- Academic attainment, as defined by the study author(s).
Study identification, inclusion and data extraction
Identification of studies
As noted in Chapter 1, the approach to searching was modified from that of the original proposal. Full details and explanation are reported in Table 1.
The revised search strategy involved three stages, which might be considered to combine the ‘known items’ and ‘law of diminishing returns’ approaches described by Booth,52 to optimise searching. First, working with an information specialist (SDa) and following the Cochrane MECIR guidance on conducting searches,50,51 we searched MEDLINE, EMBASE, the Cochrane Central Register of Controlled Trials (CENTRAL) and PsycInfo electronic databases. The search strategies for each electronic database are described in Appendix 1. The final searches were carried out on 4 April 2018. Searches were not restricted by language, country or date of publication.
Second, in addition to the database searches, we conducted searches of Epistemonikos (www.epistemonikos.org; Epistemonikos Foundation, Santiago, Chile) to identify published systematic reviews of interventions for the prevention of CMDs among CYP. Epistemonikos was searched on 16 November 2016. The reference lists from these reviews were added to the results of the database searches, ready for screening.
Finally, after screening, we conducted an informal scoping search of the Education Resources Information Center (ERIC) database. This was to check whether further relevant studies could be located, cross-referencing with those already identified from the previously mentioned approaches. If scoping revealed further relevant studies, a formal search was planned. In response to reviewer comments on the draft version of this report, the ERIC scoping searches were formalised and extended to the British Education Index (BEI). Further details are reported in Appendix 1.
Screening for study inclusion/exclusion was independently assessed by two reviewers (SRD, JCP, DMC, PC); disagreement was resolved by a third reviewer if necessary (SRD, JCP, DMC, PC, SEH). Owing to the volume of potentially relevant studies retrieved, reasons for study exclusion were recorded at full-text screening only.
We used a standardised data extraction form in Microsoft Excel® (Microsoft Corporation, Redmond, WA, USA) to extract information from included studies. Data were extracted by one reviewer and checked by a second (SRD, CF, PC, JCP, DMC). Discrepancies were discussed and a consensus reached. Disagreements were resolved by a third reviewer if necessary. The following information was extracted from the papers:
- Study design and the target CMD of intervention (i.e. anxiety, depression or conduct disorder); whether the intervention was universal or targeted (indicated or selected); and number of participants recruited, randomised and assessed (or clusters, if a cluster RCT).
- Details of participants (country, intervention setting, age, sex, ethnicity).
- Details of intervention as reported by trial author. Narrative description of components and delivery process for experimental and control interventions. This included number of sessions; intervention dose (calculated as intensity of intervention: total session time × duration in minutes); whether the intervention was group or individual, face to face or digital; who facilitated the intervention; and intervention fidelity measures.
- Outcome(s) assessed and all follow-up time points.
- Risk-of-bias assessment, including additional assessment for cluster trials.
- Mean total symptom score and standard deviation (SD) at baseline and follow-up time points for primary measurement scale, change from baseline or mean difference between arms; details on whether results were for completers only or use of methods for handling missing data such as last observation carried forward; and intracluster correlation coefficient (ICC), the statistical model used to account for clustering (if any).
Study authors were contacted for additional data, if necessary.
Classification of interventions and components
Eligible interventions were psychological and psychosocial, educational, or physical interventions. Following previous studies,68–72 and based on author-reported descriptions, the content of psychological and psychosocial interventions was classified into four broad intervention types: cognitive–behavioural, behavioural, third wave and interpersonal. Physical interventions were further classified as exercise or biofeedback. Further categories identified were a combined mindfulness/relaxation intervention, psychoeducational and psychosupportive interventions, and occupational therapy. Control interventions were classified into four categories: no intervention, waiting list, usual curriculum and attention controls.72–74 A full description of all interventions is given in Chapter 3.
We also conducted an intervention component analysis (ICA)75 to identify features of intervention content and process, influenced by the why, what, who, how, when and how much domains of the Template for Intervention Description and Replication (TIDieR).76 We applied the constant comparative method,77 whereby the intervention descriptions reported by authors were used to develop a coding scheme to classify the components of each intervention and control. If necessary, descriptions were supplemented by intervention manuals and/or correspondence with the study author. ICA is an iterative and inductive process. One reviewer (SRD) developed and piloted a list of provisional component codes based on published studies.46,68,72,78 The components were discussed with a second reviewer and a preliminary coding exercise was then undertaken on a sample of interventions by a wider group (SRD, DMC, JK and SEH), and the coding scheme was further refined based on discussion. The refined coding scheme was then applied inductively by one researcher (SDa) and audited by another (DMC). Iterative coding refinements were made until application failed to reveal any new information and the component categories could be described as ‘saturated’.
The working definitions of each intervention and control classification and the classifications of intervention components are provided in Chapter 3.
Assessment of risk of bias in included studies
Two reviewers independently used the Cochrane Risk of Bias tool79 to assess whether there was a high, low or unclear risk of bias in the following domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessor, incomplete outcome data, selective outcome reporting and other sources of bias (including cluster-specific issues such as contamination, recruitment bias and unit-of-analysis errors). All eligible studies were included in the NMA regardless of their risk-of-bias classification, and sensitivity analyses examined the impact of excluding studies deemed to be at high and unclear risks of bias for random sequence generation and allocation concealment.
Methods for the evidence synthesis of effectiveness studies
Data preparation
For continuous outcomes, data were extracted for number randomised to each intervention arm at baseline, and baseline mean and SD, and number assessed at follow-up, and follow-up mean and SD (for each time point listed previously). If the mean change from baseline was reported, then this was extracted, together with the standard error (SE) for the mean change from baseline (if reported). Data were extracted for complete cases. However, if authors reported means and SEs from an appropriate model accounting for participant dropout or non-response, this was preferred.
For analysis, we used the standardised mean change from baseline, as a variety of outcome measurement scales were used across the studies. An adjustment for small sample size was applied, following the formula for Hedges’ g.80 For studies that did not report mean change from baseline, we derived this from reported baseline and follow-up means and SDs.81 Here we assumed a correlation coefficient of 0.7, based on previous analyses.82 This value was explored in sensitivity analyses. Results are summarised using SMDs and 95% credible intervals (CrIs).
For dichotomous outcomes, data were extracted for available cases unless authors clearly reported events and number of participants following the intention-to-treat principle. Dichotomous outcomes were summarised using odds ratios (ORs) and 95% CrIs.
If key statistics (e.g. SDs) were not available in the published report, we contacted trial authors for further information. In cases of non-response, or if missing data were not available, we did not impute the data and these studies were excluded from the NMA (but not the systematic review).
For cluster randomised trials that did not account for the effect of clustering, we followed the advice in the Cochrane handbook (section 16.3.4)81 for calculating an approximate sample size. We reviewed reported ICCs from all included papers and used an ICC estimate of 0.03, which is the mean of the values reported and is similar to ICCs used in previous public health systematic reviews.68,83–85 This value was also explored in sensitivity analyses.
Pairwise and network meta-analyses
Both standard pairwise meta-analyses and NMAs were conducted. A NMA was planned for each primary outcome and for the primary time point only.
A NMA allows data on multiple interventions to be pooled in a single analysis.86 It is considered a ‘core method’ by Cochrane59 and is routinely used in National Institute for Health and Care Excellence (NICE) technology appraisals and guidelines.87,88 For the novice, the Cochrane handbook81 provides an accessible introduction; for further depth, we recommend consulting tutorials and introductory papers on NMA.44,45,60,89–92 However, we describe the fundamentals in the following paragraphs.
A NMA requires that the intervention comparisons made in RCTs can be displayed pictorially as a network of comparisons that are ‘connected’ (i.e. there is a path from any one intervention to another formed by RCT evidence).59,93 In the absence of direct head-to-head evidence comparing interventions, indirect comparisons can be made via common comparator(s) across the network. For example, the effect of intervention C against intervention B can be obtained from the effect of C against A, minus the effect of B against A. The combination of indirect and direct evidence across a network of intervention comparisons is known as a NMA.
The validity of a NMA assumes that there are no differences between studies in factors that might interact with the intervention effect (effect modification). This is the same assumption made in a pairwise meta-analysis,57 but in a NMA applies across intervention comparisons. It is therefore important to consider separate analyses according to factors that may be potential effect modifiers. In the first instance, separate analyses were conducted by population (universal or targeted) and setting (primary, secondary or tertiary education). Separate analyses were run for the main outcomes of self-reported anxiety, self-reported depression and self-reported conduct disorder symptoms. Following research suggesting common mechanisms and pathways within internalising and externalising disorders (transdiagnostic factors),94–97 we ran analyses across (1) all studies aiming to prevent depression and/or anxiety and (2) studies aiming to prevent conduct disorder. That is, studies contributing to either the depression or anxiety outcome analyses could be studies that aimed to prevent (1) anxiety (2) depression or (3) anxiety and depression. Studies contributing to the analysis of the conduct disorder outcome were only those aiming to prevent conduct disorder. We explore this decision further in a subgroup analysis (see Subgroup, metaregression and sensitivity analyses). A visual check of the inclusion/exclusion characteristics of trials in each network was conducted, to ensure that potential effect modifiers were evenly distributed across studies.
Network plots were drawn in Stata® version 15 (StataCorp LP, College Station, TX, USA) to allow visual inspection of network connectedness.98
For the primary time point of post intervention only, we considered three NMA models, each allowing for increasing specificity of intervention detail:46,82
- Intervention-level model – interventions were analysed as ‘clinically meaningful units’.48 For example, cognitive–behavioural therapy (CBT) was analysed as a distinct intervention to psychoeducation or third wave-based interventions.
- Additive model – components nested within intervention. A main intervention effect was estimated (as per the intervention-level model), plus additional effects for specific components within each intervention. For example, we estimated an overall CBT effect, which represents the effect for CBT interventions with components that were common across all the included CBT interventions, and also estimated the additional effect of CBT interventions containing a psychoeducation component, a mindfulness component and so on.
- Full interaction model – components nested within intervention: under this model, each unique combination of intervention and components was considered as a separate intervention. For example, CBT with cognitive + behavioural + psychoeducation components was considered a distinct intervention to CBT with cognitive + behavioural + psychoeducation + relaxation components.
For follow-up time points, where we anticipated finding fewer studies, we ran the intervention-level model only as prespecified in the protocol. Intervention-level analyses were implemented in a Bayesian framework using OpenBUGS software (version 3.2.3). Component analyses were implemented in WinBUGS99 (version 1.4.3; MRC Biostatistics Unit, Cambridge, UK). Statistical details are reported in Appendix 1. Data and WinBUGS code can be obtained by contacting the corresponding author. Vague prior distributions were specified for intervention effect and heterogeneity parameters (see Appendices 1 and 3). We assessed convergence for the intervention-level NMA based on three chains using the Brooks–Gelman–Rubin diagnostic tool and history plots in OpenBUGS. Specific convergence details for each model and population/setting analysis are reported in the model fit tables in Appendix 3, Table 29–46.
Random-effects models were run for the main outcomes, assuming a common between-study SD (known as ‘homogeneous variance’).100 However, we assessed both fixed- and random-effects models on the basis of model fit. Heterogeneity was evaluated by examining the posterior median between-study SD (τ) and 95% CrIs from the random-effects model, and by comparing model fit of the fixed- and random-effects models. Model fit was measured by the posterior mean of residual deviance. In addition, we examined the deviance information criterion (DIC), which penalises model fit with model complexity. Differences of ≥ 5 points in posterior mean residual deviance and the DIC were considered meaningful, with lower values preferred.101 Model fit statistics are reported in Appendix 3.
As described previously, a key assumption for a valid NMA is that of consistency between the direct and indirect evidence. If the effect estimates from the direct and indirect evidence in a network do not agree, this is known as inconsistency. The strict inclusion/exclusion criteria described previously were specified to avoid inconsistency, but they do not guarantee consistency. For this reason, the statistical agreement of the evidence was formally checked. Consistency was assessed by comparing the goodness of fit of a model assuming consistency with that of one allowing for inconsistency (i.e. a model that provides effect estimates based on direct evidence only). A common between-study SD was also assumed for these inconsistency models.57
Pairwise meta-analyses were also conducted when head-to-head evidence was available. The method of estimation is similar to the NMA, except that the consistency assumption is removed such that intervention effects for separate comparisons are unrelated and separate intervention effects can be estimated.57 Estimates are reported for the post-intervention time point only and are from a random-effects model that assumes that the heterogeneity parameter is common across intervention comparisons. This better reflects the assumption made in the NMA and, therefore, allows a fair comparison of the intervention effect estimates obtained from both approaches.
Subgroup, metaregression and sensitivity analyses
Metaregression and subgroup analyses were performed in OpenBUGS following the Evidence Synthesis Technical Support Unit code available from the NICE Decision Support Unit’s website and described in Dias et al.102,103 Subgroup analyses were conducted to assess whether or not intervention effects differed by intended focus of the intervention, for example if interventions addressing anxiety had a larger effect on anxiety outcomes than interventions intended to focus on depression but which also recorded anxiety outcomes.
Metaregression was planned for the intervention-level NMA and main outcomes only, to examine if intervention effects differed by mode of intervention delivery and who facilitated intervention delivery:
- Mode of intervention delivery – interventions were categorised as being delivered face to face or via a computer/internet. To explore whether or not intervention effects were modified by mode of delivery, we fitted a metaregression model for face-to-face (covariate value = 0) and computer/internet (covariate = 1) interventions. A random-effects NMA model was fitted. However, the regression coefficient for the covariate was assumed to be a fixed effect across studies. The between-study SD was assumed to be common for face-to-face and computer/internet interventions. Vague priors were specified.
- Who facilitated intervention delivery – interventions were categorised as being facilitated by a teacher or a mental health professional (MHP). There was considerable variation within the category of ‘MHP’ and it should be regarded as a simplification. Here, MHP included school counsellors, qualified psychotherapists and graduate and post-doctoral psychology students. Graduate and post-doctoral students included those studying general psychology, educational psychology or counselling psychology, when specified. We fitted a metaregression model that enabled us to estimate the intervention effect at each value of the covariate (0 or 1), for each intervention, comparing the effect of each facilitator (e.g. CBT-teacher vs. CBT-MHP vs. usual care). To allow for networks containing multiarm studies with more than two interventions facilitated by a teacher or MHP, a hierarchical model was fitted. In this hierarchical model, a regression coefficient for each intervention was assumed to come from a normal distribution with a common mean and between-intervention SD. The between-intervention SD was assumed to be common for each value of the covariate. Vague priors were specified.
We explored the potential for small-study effects using comparison-adjusted funnel plots.98
Sensitivity analyses were conducted for the intervention-level NMA, main outcomes and primary time point only. Analyses explored the robustness of results to the following:
- Excluding studies deemed to have a high/unclear risk of bias on the domains of random sequence generation and allocation concealment.
- The ICC value of 0.03 for cluster randomised trials. Sensitivity analyses were conducted assuming an ICC of 0.01 and 0.06.
- The correlation value of 0.7 assumed for calculating change from baseline SD. Sensitivity analyses were conducted assuming a correlation of 0.6 and 0.8.
Interpreting the results and evaluating evidence of effect
As described previously, summary effect estimates and their 95% CrIs from the NMA are reported. In interpreting these statistical findings, we followed the guidance from the Cochrane handbook (section 15.3) that interpretation of results from a meta-analysis should not rely on statements of ‘statistical significance’ or thresholds implying ‘significance’.104 Instead, we interpret the strength of statistical evidence on a graduated scale from weaker to stronger evidence of an intervention effect.105,106
In Chapter 9, we also provide a summary of these statistical findings for the primary outcomes of anxiety, depression and conduct disorder symptoms at the primary time point of post intervention. This interpretation forms the basis of the conclusions for the report. The criteria used are based on the considerations outlined in Chapters 14107 and 15104 of the Cochrane handbook. These considerations are informed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) domains of imprecision, inconsistency (heterogeneity), risk of bias and publication bias. It is not a formal application of the GRADE rating system, which necessitates up- or downgrading of the evidence on the basis of the assessments to form an overall assessment of ‘quality’.107
Specifically, our interpretation of the evidence is not solely based on the magnitude and direction of the summary point estimate, but also incorporates an evaluation of all of the following:
- the precision of the effect estimate, as described by the 95% CrI
- the extent of the between-study heterogeneity observed in each analysis, as described by τ and its 95% CrI
- the risk-of-bias assessment
- the possibility of non-reporting biases or evidence of small-study effects.