Methods and analyses

Overall study design

This was a cluster RCT with an embedded economic and mixed-methods process evaluation. The unit of recruitment and randomisation was the school, as this was an intervention that operated across the whole school.

The trial was registered with the International Standard Randomised Controlled Trial Register (reference ISRCTRN95909211). We published the protocol on 18 October 2016.63 We made some small changes to the protocol since it was originally submitted, which are summarised in Table 1. The full statistical analysis plan is available at URL: www.bristol.ac.uk/population-health-sciences/projects/wise/publications/ (accessed 5 November 2020).

Study population

The population from which the schools in our sample were recruited was secondary schools in two geographical areas: (1) Bristol and surrounds up to a 30-mile radius, comprising 64 mainstream secondary schools and (2) South East and South Central Wales, comprising 88 mainstream secondary schools. Exclusion criteria were as follows:

• fee-paying schools
• special schools (e.g. for those with learning disabilities)
• pupil referral units
• schools that had been WISE pilot schools
• schools participating in other, similarly intensive, research studies
• schools already delivering MHFA or other mental health training
• schools without available FSM data for stratification purposes
• schools within the same academy chain and local authority as one that had already been selected for participation.

In total, we aimed to recruit 24 schools (i.e. 12 in each area).

Our study population within each school was all teaching staff and all students in year 8 (aged 12–13 years) at baseline. At 24-month follow-up (T2), the same students were year 10 (aged 14–15 years). We assumed that any effect on this year group would be the same for all students, as the intervention was intended to be at the whole-school level.

Recruitment and randomisation

In both England and Wales, the sampling frames were stratified by the proportion of children in each school who were eligible to receive FSM to ensure schools from a wide range of socioeconomic catchment areas were included in the study. In Wales, the sample was divided into two administrative regions (educational consortia), South East and South Central Wales, and schools were stratified into three FSM levels (high, medium and low vs. the national average). Two schools were randomly selected from each stratum in each region, and selected schools were approached via a relevant senior leader (e.g. a deputy head in charge of pastoral care) and invited to participate in the study. Schools that declined were replaced by a randomly selected school from the same stratum and region. In England, the study was advertised to head teachers at all eligible schools and invitations were followed up with relevant senior leaders, identified with the help of local public health teams that worked with the schools. Those who expressed interest in participation were stratified into three levels according to FSM (high, medium and low vs. the national average) and local authority (Bristol/non-Bristol). When more than two schools fitted into one stratum, two were randomly selected.

Within each study area and stratum, selected schools were randomly allocated to a study arm. Sequence allocation was generated by the study statistician, who was blinded to the schools’ identities, using the Stata ralloc command (version 14, StataCorp LP, College Station, TX, USA). Random allocation to study arm took place after baseline data collection.

Main outcomes

Primary outcome

The primary outcome was teacher well-being, measured using the Warwick–Edinburgh Mental Wellbeing Scale (WEMWBS).62 The WEMWBS is comprehensive (incorporating elements of both subjective and psychological well-being), short enough to be used in population-level questionnaires, responsive to change and validated among community samples of adults in the UK. [Licenses to use the WEMWBS are available at https://warwick.ac.uk/fac/sci/med/research/platform/wemwbs/using/ (accessed 11 February 2021).] We selected well-being rather than a measure of mental health difficulty as the primary outcome because it is a broad enough measure to capture changes among staff with a range of mental health needs, including those who have good well-being, but may still benefit from cultural changes at the whole-school level.

The WEMWBS scores can range from 14 to 70, where a higher score indicates better well-being. This outcome was measured at baseline, T1 and T2.

Secondary outcomes

Teacher depression was measured using the Patient Health Questionnaire-8 items (PHQ-8).64 The PHQ-8 is suitable for measuring levels of depressive symptoms in population-based studies and is short enough to be used in self-complete questionnaires. It was analysed as a continuous variable, but also as an ordinal variable (0–4 indicating no depressive symptoms, 5–9 indicating mild symptoms, 10–14 indicating moderate symptoms, 15–19 indicating moderately severe symptoms and 20–24 indicating severe symptoms) and a binary variable, with a cut-off point of ≥ 10 indicating depression.64 These cut-off points were established for the Patient Health Questionnaire-9 items, but are the same for the PHQ-8, with the only difference between the two scales being an additional question about self-harm behaviour, which does not contribute to the overall depression score.

Teacher absence was measured by self-report (i.e. the ‘number of days missed from school because of health problems in the past 4 working weeks’) and routine data collected by the schools (i.e. the total number of days of absence over the previous year). Both measures were treated as continuous, but the self-report measure was also treated as binary (any vs. no absence) and ordinal. Based on the distribution of responses in the pilot study, in which the majority of respondents had taken very little absence, categories were 0 days, 1 day, 2–7 days or ≥ 7 days.

Teacher presenteeism was measured by self-report, using the relevant item from the Work Productivity and Activity Impairment questionnaire65 and adapted to fit teachers’ work schedule. Teachers were asked ‘during the last 4 working weeks, how much did health problems affect your productivity while you were working?’ and selected a number between 0 (no effect) and 10 (completely prevented me from working). This measure was treated as continuous, but also as binary (no effect vs. 1–10 effect on work over the previous 4 working weeks) and ordinal. Based on the pilot responses to presenteeism, in which roughly one-third of participants scored 0 and a majority scored < 6, the categories were 0 days, 1–5 days or 6–10 days.

Teacher retirements and leaving for other reasons were collected as total number reported by the schools for the previous year.

Year 8 student well-being was measured using the WEMWBS, which has been validated for use among teenagers from age 13 years.66 This measure was treated as continuous.

Year 8 student psychological distress was measured using the Strengths and Difficulties Questionnaire (SDQ), a widely used measure among this age group, with well-established norms for a UK population.67 A total difficulties score was calculated (this can range from 0 to 40, with a higher score indicating greater difficulties) and treated as continuous.

Student attendance was collected as a total proportion for each school from routine data sources for England68 and Wales.69

Student attainment at the end of year 11 was collected from the same routine data sources as student attendance.68,69 Exams at the end of statutory school age (key stage 4) are graded and reported differently for the two countries. For English schools, we reported per cent of students achieving grade 5 or above in both English and Maths General Certificates of Secondary Education, and for Welsh schools we reported per cent of students achieving grade C or above in English/Welsh, Maths and Science GSCEs.

The baseline questionnaires are included in Appendices 1 and 2.

Outcome data collection process

Teacher self-report outcome questionnaires were completed during staff meetings at baseline (May–July 2016), T1 (May–July 2017) and T2 (May–July 2018). Those who were absent from the data collection session were offered the option of completion via an online survey. Data on teacher absence, retirement and leaving for other reasons were collected via routine data recorded by the schools.

Student mental health outcomes were gathered via self-report questionnaires at baseline and T2, completed by all students in year 8 (aged 12–13 years at baseline), during tutor groups or lesson time. Where fewer than five students were absent, questionnaires were left with school for those students. Where five or more students were absent, a second data collection was arranged with schools. School-level student absence and attainment were collected using government statistics published online.

Teacher routine outcomes (school-reported absence and leavers data) and all student outcomes were measured at baseline and T2 only. This was for practical reasons, and also because the intervention’s logic model implied that teacher mental health would need to improve before changes to student mental health were maximised.

The study team planned to attend all data collections, introduce the questionnaires, answer any questions and take them away at the end. However, three schools at baseline did not manage to secure meeting time to conduct the data collections. In these schools, the team briefly introduced the questionnaires in a meeting, but left them with staff along with envelopes in which questionnaires could be sealed and then either posted or returned via a box left at the school. Two of these schools followed the same process at both T1 and T2, despite attempts by the study team to organise meeting time.

Compliance and loss to follow-up

We encouraged the schools to set the dates for data collection sessions far in advance to try and ensure maximum attendance. We built in time to follow up those who were absent, either in person or by leaving questionnaires in named envelopes, which the research team collected once completed. We also made an online version of the questionnaire available to teachers who missed the data collection session. We incentivised schools to remain part of the study by providing them with their own school-level data from each time point at the end of the study. We also offered control schools a payment at the end of the study, which they could choose to spend on the intervention if it was found to be effective. This payment was contingent on a response rate of at least 80% at follow-up among teachers and students.

Sample size

The study was powered to detect a mean change of 3 points on the WEMWBS (i.e. the primary outcome measure). This difference was chosen as the minimum meaningful change discussed in a WEMWBS user guide.70

A change of 3 points was also close to the difference in mean WEMWBS scores between staff in the highest- and lowest-ranked schools in the pilot study.

It was estimated that each school would have approximately 60 teachers and 150 year 8 students, depending on size. The sample size took account of clustering. In the pilot data, the teacher WEMWBS intracluster correlation coefficient (ICC) at baseline was 0.01 (95% CI 0.00 to 0.03) and at follow-up it was 0.00 (95% CI 0.0 to 0.02). We assumed a mean of 50 (83%) teachers followed up per school (with a coefficient of variation of sample size of 0.5) and a standard deviation (SD) for WEMWBS of 8.4 (based on the pilot data). A sample size of 24 schools (i.e. 12 intervention schools and 12 control schools) would achieve 83% power for an ICC of 0.05. This would rise to 98% if the ICC is 0.02, and fall to 65% for an ICC of 0.08. Sample sizes were calculated using the clustersampsi command in Stata. For the tests for interactions (see Main outcomes analysis), we would have 80% power to detect an interaction that is twice as big as the main effect, and 30% power to detect an interaction that is the same size as the main effect.71

Blinding and breaking of blind

Allocation to study arm took place after baseline data had been collected to ensure blinding among all parties during this first data collection. It was not possible for the schools, teachers or students to be blind to intervention status (although students were likely to be unaware of the intervention). The research assistants/associates leading outcome data collection also collected the process data, which prevented blinding among the study team.

Clustering

All statistical analyses took account of clustering by school using robust standard errors and, where appropriate, by using random-effects models with schools as a random effect. Analyses were conducted using Stata.

Main outcomes analysis

Primary analysis

The primary analysis was carried out under the intention-to-treat principle, analysing participants as randomised without the imputation of missing data. Repeated-measures (random-effects) models were used to examine pattern of change in primary outcome over baseline, T1 and T2, adjusted for stratification variables, sex and years of experience. This included a random effect for individual participants and another for school. This model included every teacher who had at least one measure of the outcome (i.e. at baseline, T1 or T2). Using a maximum likelihood estimator, this analysis was robust to data that are missing at random (MAR).72,73 Results are presented in Chapter 3 as mean difference in the primary outcome between the trial arms over the follow-up period, with associated 95% CIs and p-values.

The primary analysis was repeated with a treatment by time interaction term added to the model. This allowed estimation of treatment effect at T1 and T2 separately.

Secondary analyses

Analysis of secondary outcomes included linear, ordinal and logistic regression models, dependent on the nature of the outcome variable being analysed (continuous, ordinal or binary, respectively).

For secondary individual-level outcomes that are measured at baseline, T1 and T2 (e.g. PHQ-8), repeated-measures models were used and these models included random effects for clustering by individual (because of repeated measure) and by school. All individuals with at least one observation of the outcome measure were included in the model for that outcome using maximum likelihood under a MAR assumption. For each secondary individual-level outcome that was measured at baseline, T1 and T2, three models are presented:

1. Unadjusted model (model 1): repeated measures of the outcome regressed on treatment arm, accounting for clustering because of repeated measures and by school (using random effects).
2. Partially adjusted model (model 2): model 1 plus adjustment for stratification variables.
3. Fully adjusted model (model 3): model 2 plus additional adjustment for the covariates sex and years of teaching experience.

For each secondary individual-level outcome measured at baseline and T2 only, three models are presented:

1. Unadjusted model (model 1): outcome at T2 regressed on treatment arm and baseline value of the outcome, accounting for clustering by school (using a random effect).
2. Partially adjusted model (model 2): model 1 plus adjustment for stratification variables.
3. Fully adjusted model (model 3): model 2 plus additional adjustment for sex and ethnicity as covariates.

For each secondary school-level outcome measured at baseline and T2 only, two models are presented:

1. Unadjusted model (model 1) – outcome at T2 regressed on treatment arm and baseline value of the outcome (no need for any adjustment for clustering).
2. Adjusted model (model 2) – model 1 plus adjustment for stratification variables.

Missing data

We assessed the impact of missing data and non-response on teacher WEMWBS and PHQ-8 outcomes and student WEMWBS and SDQ outcomes using multiple imputation. The multiple imputation model73 included all outcomes at baseline, T1 and T2, variables included in the primary analysis model and baseline characteristics associated with missingness. Analyses were repeated on the imputed data sets and results were combined using Rubin’s rules. We also used sensitivity analyses to examine the impact of potential missing not at random (MNAR) data by incorporating a scaling parameter that was allowed to differ between arms (i.e. allowing the missing data mechanism to differ between arms).74 This was carried out by imputing under the MAR assumption and then multiplying imputed values of the outcome by a scaling parameter. For example, a scaling parameter of 0.95 in the control arm and 0.90 in the intervention arm implies that missing WEMWBS scores in the control arm are assumed to be 5% lower than estimated under a MAR assumption, whereas scores in the intervention arm are 10% lower.

Sensitivity analyses

We re-ran the primary analysis without identified outliers (defined as a WEMWBS score > 3 SDs from the mean) to assess the effect of outliers on the main outcome.

Subgroup analysis

We used a complier-average causal effect (CACE) approach (using instrumental variable analysis)75 to examine the impact of MHFA training on follow-up WEMWBS and PHQ-8 scores, comparing those who completed the 1 or 2 days, training in the intervention schools with those in the control schools who would have completed the training, had they been offered it. For each participant, we calculated a summary score of the two outcomes and used the summary score to account for the repeated measurements (therefore reducing the two measures to a single measure). If only a single measure was available then we used just the single measure. We included robust standard errors to account for clustering at the school level.

Tests for interactions

The effect of the intervention on teacher WEMWBS and PHQ-8 score was tested for interaction with baseline well-being/depression score (grouped as above or below the bottom quartile of the WEMWBS, or with a score of ≥ 10 on the PHQ-8), geographical area (Wales/England), school-level FSM and sex. The effect of the intervention on student WEMWBS and SDQ score was tested for interaction with baseline well-being/SDQ score (grouped as above or below the bottom quartile of the WEMWBS and a score of ≥ 20 on the SDQ), geographical area, school- and individual-level FSM, sex and ethnicity. p-values were interpreted with caution because of the low power and number of interactions being tested (we used Bonferroni corrected or permutation p-values).

Exploratory analysis: mechanisms of change

To explore the study’s logic model and hypothesised mechanisms of change, we collected data in the teachers’ questionnaires regarding stress and satisfaction at work, support given/received at school, school’s perceived attitude to staff and student well-being, and perceived quality of relationships in school. These questions were asked with a series of Likert responses (see Appendix 2 for full wording and response options). Logistic regression models were used to compare binary measures of these variables between arms at follow-up adjusted for baseline scores, school-level FSM and geographical area. We examined whether or not baseline measures of these variables, which provide indicators of school psychosocial context, moderate the effect of the intervention by including interaction terms between these baseline variables and intervention arm in the analysis model.

Appropriate mixed models were applied as for the primary analysis, but, additionally, these measures were included as covariates. We assessed the degree to which the estimated treatment effect attenuated compared with our chief analysis model (substantial attenuation would indicate the proposed mechanisms of change are indeed acting as such).

Exploratory analysis: level of intervention

Data from the process evaluation [i.e. training participant evaluation forms, peer supporter feedback meetings, peer supporter logs of support and follow-up questionnaires (see Process evaluation for more detail)] were used to divide intervention schools into ‘low’ or ‘high’ implementation groups. Factors, measured as binary variables, that were used for this analysis were as follows.

Dosage

• At least 8% of teachers completed the 1-day MHFA for schools and colleges training (vs. < 8% of teachers).
• At least 8% of staff in the school completed the 2-day standard MHFA training and went on to become a peer supporter (vs. < 8% of staff).
• At least 8% of staff in the school still acting as peer supporters by T2 (vs. < 8% of staff).

Reach

• At least 75% of teachers attended the 1-hour awareness training (vs. < 75% of teachers).
• Higher than study mean for those who select ‘staff peer supporter’ in response to the question ‘if a work-related problem was making you stressed or down who would you talk to about it at school?’ in the follow-up questionnaires.

Fidelity

• 100% of course attendees indicated that all topics were covered in the 1-day MHFA for schools and colleges training and 2-day standard MHFA training (vs. < 100%).
• The peer supporters set up a confidentiality policy for the service (vs. no policy).
• The peer support service was advertised in three or more ways initially (vs. advertised in two or fewer ways).
• The peer support service was re-advertised at the beginning of the 2017–18 academic year (vs. not re-advertised).
• The peer support service had been championed by a member of the senior leadership team (vs. not championed).

Correlation between all the implementation measures was poor and so we were unable to determine one composite binary measure ‘low/high implementer’. Instead, we scored each school on each factor and split the schools based on their total scores, with the six highest being high implementers and the six lowest being low implementers. We then compared primary outcomes among intervention schools that were high implementers with those that were low implementers. We repeated this analysis with implementation as a continuous variable. Adjustment was made for both school- and individual-level covariates, as in the primary analysis.

Economic evaluation

Measurements

The economic analysis took a public sector perspective, calculating the financial and opportunity costs to schools of participating in the WISE intervention. We collected self-reported information on the impact of teachers’ health on absenteeism and presenteeism during the previous 4 weeks at T1 and T2. However, in common with other school-based interventions,76 we elected not to track teacher health-care use at T1 or T2.

Data were collected on the resources used for the following four activities: (1) the HSC MHFA instructor training (which was relevant for Welsh schools only); (2) the 2-day standard MHFA training; (3) the in-service day MHFA training for schools and colleges; and (4) the awareness raising session. In each case, the project team completed a pro forma after the training session, documenting the resources used and expenses claimed.

As noted in Chapter 1, in Wales, Public Health Wales paid for a course to train seven HSCs as MHFA instructors who could then deliver the MHFA training to teachers in Welsh schools. The course included 6 days of training with a MHFA trainer and 2 additional days of independent preparation for the HSC trainees. The costs comprised HSCs’ time (estimated based on salary costs), actual travel expenses incurred, MHFA course fees, MHFA trainer expenses and course refreshments. The venue was provided at no cost by Cardiff University (Cardiff, UK).

The resources used during the 2-day standard MHFA training comprised MHFA trainer fees, trainer travel expenses (in English schools this was sometimes included in the trainer fee), MHFA manuals, refreshments (in most schools) and the time of the teachers being trained as peer supporters (a minimum of five teachers and a maximum of 16 teachers). There were no venue costs for training, as this happened either on the school site or at a university. Several schools provided estimates of the cost of hiring supply teachers to cover the teaching commitments of staff attending the MHFA training. We used these estimates to impute the opportunity cost of staff time at all schools.

The resources used for the 1-day MHFA schools and colleges training comprised MHFA trainer fees, trainer travel expenses (in English schools this was sometimes included in the trainer fee), MHFA manuals and the time of the teachers attending training (a minimum of seven teachers and a maximum of 18 teachers). As training took place during in-service days, no supply teachers were necessary and staff costs were estimated based on anonymised salary information provided by the schools. When salary information was not provided by the school (n = 1), these costs were imputed based on the number of attendees and the mean salaries of attendees from other schools.

The resources used for the 1-hour awareness raising session for all teachers comprised the trainer fees, and travel costs and the staff costs for the teacher and support staff attending. For the English schools, staff costs were estimated from the number of staff attending (a minimum of 31 staff and a maximum of 85 staff) and estimated salaries were based on salary band information provided by schools. The mean school staff costs in England were used to impute the staff costs in Welsh schools, where data on staff attendance was not available.

All resource use was valued in monetary terms using published UK unit costs (Table 2) or participant valuations estimated at the time of analysis (2016/17). Support staff salaries were obtained by asking schools/individuals for this information. Teachers’ salaries were estimated using national pay scales after confirming participants’ pay bands.

Cost–consequence analysis

The economic evaluation is a cost–consequence study that provides evidence on whether or not the incremental costs of the intervention are justified because of improved teacher or student outcomes. The potential benefits of the intervention are multifaceted and affect multiple agents (e.g. staff, students and schools), even those not directly observed in the study (e.g. students in other years). Therefore, a cost–consequence framework that tabulates costs against several outcomes for staff and students is appropriate, rather than a more reductive cost-effectiveness or cost–utility analysis that attempts to summarise efficiency in a single ratio.77 It provides a more complete analysis for policy-makers than a ‘cost per unit improvement in WEMWBS score’, which has no easy interpretation. Previous research has demonstrated limited correlation and overlap between the WEMWBS well-being scores and health-related utility scores (such as the EuroQol-5 Dimensions, three-level version) used to estimate quality-adjusted life-years.78 As quality-adjusted life-years are unlikely to be sensitive to small changes in well-being, we did not include the EuroQol-5 Dimensions or other preference-based health-related quality-of-life measures.

Subgroup analysis

Subgroup analysis is reported by country to investigate any differences in findings between England and Wales. This is important, given that the costs and, potentially, the outcomes will differ because of the different use of MHFA trainers in England and Wales.

Process evaluation

Aims

A mixed-method process evaluation was integrated into the outcome and economic evaluations,79 following a ‘nested’ design.80 The primary aim of the process evaluation was to support the interpretation of the outcome data and further refine the programme theory.81 The process evaluation had five central objectives, which, for the purpose of the study, were categorised as domains of inquiry. These were as follows.

Implementation

The aim was to assess the implementation of the intervention, using a multicomponent framework.

Implementation is assessed separately for the MHFA for schools and colleges and standard MHFA training courses provided for schools, and the delivery of the peer support services. For the training, the aim was to assess (1) reach (i.e. the number who attended), (2) dose (i.e. the completion of the training), (3) fidelity to the course and adaptations undertaken, (4) variations in fidelity between schools in England and Wales, and (5) quality of training delivery. For delivery of the peer support service, the aim was to assess (1) reach (i.e. the proportion of teachers who make use of the peer support service), (2) fidelity to the peer support service model and adaptations undertaken, and (3) contextual characteristics that determined barriers to and facilitators of implementation.

Mechanisms of change

The aim was to explore if the intervention’s hypothesised mechanisms of change were activated as intended, and the extent to which these mechanisms were modified through their interaction with contextual characteristics. Postulated mechanisms, refined through the feasibility and pilot phase of intervention evaluation, are depicted in the logic model (see Figure 1). We also considered unintended consequences and, in particular, unanticipated harms and the pathways by which these may have occurred.82

Programme differentiation and contamination

The aim was to understand usual practice to assess programme differentiation and ascertain if contamination had occurred in control schools.

Sustainability of the intervention

The aim was to assess the extent to which the intervention is sustainable and its scope to become routinised as part of usual practice outside a trial setting.

Acceptability

The aim was to explore participants’ perceptions of and interactions with the intervention components, and how this differed across school contexts and across the course of delivery. The purpose of exploring intervention acceptability was to understand if and in what ways its acceptability to participants influenced the operationalisation of the mechanisms of change. Therefore, it is explored as an embedded theme throughout the process evaluation results in Chapter 4, rather than as a discreet theme.

Research questions

Table 3 lists the research questions that the process evaluation addressed and outlines the domain of inquiry to which each one was aligned.

Sample

Four intervention schools and four control schools were purposively sampled to serve as ‘case study schools’, in which more extensive process evaluation was undertaken (e.g. focus groups and observations). To sample the case study schools, all study schools were stratified by trial arm allocation (intervention vs. control), site (England vs. Wales), administrative region (educational consortia vs. local authority) and proportion of students eligible for FSM (high/low vs. the national average). One intervention school within each stratum was then purposively sampled to achieve variation across the cases in school size and assessment ranking by the educational inspectorate. One control school was selected within each stratum to match the intervention cases as closely as possible. Table 4 presents the final sample of case study schools.

Data sources

The process evaluation adopted a mixed-methods approach and used both quantitative and qualitative data sources. The different types of data collected from the study schools, and how they relate to the research questions, are summarised in Report Supplementary Material 3.

Teacher and student questionnaires

The teacher and student questionnaires that were designed to collect the main outcome data also contained questions relating to the mechanisms of change domain of inquiry. As already noted in Main outcomes, the teacher questionnaires asked about stress and satisfaction at work, quality of relationships between teachers and students and between staff, whether or not the school cares about teacher well-being and how much individuals had given support to colleagues over the previous year. The questionnaire also asked about perceptions of how much their school cared about student well-being and the extent to which they had supported students in the previous year. Follow-up questionnaires in the intervention group only also examined intervention reach (i.e. the numbers who had completed MHFA training and who had received support from the peer supporters) as part of the implementation domain of inquiry. The student questionnaires asked about perceptions of the extent to which the school and teachers cared about students, how often individuals had asked a teacher for help over the previous year and whether or not this had been useful.

Audit of school policies and interventions

Participating schools were audited at baseline and T2 (see Report Supplementary Material 4 for a copy of the audit questions). The main contact teacher for each school reported on existing policies and interventions within the school in relation to the mental health and well-being of teachers and students. The baseline audit allowed for comparison of pre-existing activities in intervention and control schools to explore possible baseline imbalances and to understand the general context in which the intervention was being delivered. The T2 audit captured relevant policies or interventions that had been introduced during the study period. It therefore enabled exploration of programme differentiation from usual practice in intervention schools and assessment of contamination in the control schools by identifying any new activities that were similar to components of the WISE intervention. An additional question was included in the follow-up audits that asked about any key events that had occurred during the study that were likely to have an impact on mental health and well-being. This, again, enabled consideration of the impact of the context on any intervention effects.

Attendance records for the Wellbeing in Secondary Education intervention training

The MHFA course instructors completed an attendance record for each training course, which enabled an assessment of reach of the training. Instructors also conducted a head count of staff attending the 1-hour awareness raising session; however, some schools opened this session to all teaching and non-teaching staff, and, therefore, it was not possible to calculate the proportion of teaching staff that attended these sessions.

Observations of the Wellbeing in Secondary Education intervention training

In the four intervention case study schools, two members of the research team observed the 1-hour awareness raising session, the 2-day MHFA training course for peer supporters and the 1-day MHFA for schools and colleges course. Standardised observation schedules (see Report Supplementary Material 2 for an example of a template) were completed to quantitatively assess coverage of materials (yes/no for each topic), quality of delivery and participant engagement (5-point Likert scales). Observers also qualitatively documented any information relevant to understanding the quantitative assessment and other issues of importance not covered by the quantitative scale, including course adaptations and general contextual observations. This generated data relating to the implementation and acceptability domains of inquiry.

Post-mental health first aid training course fidelity checklist

In all 12 intervention schools, participants completed a checklist recording whether or not key topics were covered (yes/no), and rating the instructors knowledge and skills (5-point Likert scale). An example checklist is included in Report Supplementary Material 5. This enabled further consideration of the training implementation.

Training evaluation forms

Attendees of all MHFA training in the 12 intervention schools completed standardised MHFA evaluation forms. These forms recorded self-assessed knowledge and confidence to support others before and after the course, and views on course quality (5-point Likert scale).

Attendees of the 1-hour awareness raising session completed a study-specific evaluation form (see Report Supplementary Material 6) which asked them to rate how much their knowledge had improved (no impact, a small increase, a large increase) in relation to six topics and how highly they rated the instructor (5-point Likert scale). These data illuminated the mechanisms of change domain of inquiry.

Peer supporter logs and feedback sessions

The peer supporters completed a termly electronic log that documented delivery of support to colleagues during the previous 2 working weeks (see Report Supplementary Material 7). To mitigate the risk of seasonal bias (e.g. stress associated with end of term examinations), peer support logs were issued at different times during the academic term. The log assessed the reach (i.e. how many colleagues received support), the role of the staff members supported, the type of problem addressed (e.g. work or personal life), time of day and length of support episode, and the outcome of the interaction. The logs also invited peer supporters to provide any information about difficulties they had experienced as a peer supporter to allow monitoring of any harms. The log was used to record when peer supporters had left the school.

In addition, one or two peer supporters in all 12 intervention schools were invited to a feedback session approximately 6 months and 18 months after their training had been completed. The purpose of the feedback sessions was to assess the extent to which the peer support service guidelines had been adhered to, again shedding light on the implementation domain of inquiry. The sessions took the form of structured interviews (see Appendix 3 for the questions asked), with responses audio-recorded.

Interviews with mental health first aid instructors

As noted above, the 2-day, 1-day and 1-hour training sessions were delivered by independent MHFA instructors to English schools, and by HSCs employed by Public Health Wales or local authorities to Welsh schools. Semistructured interviews were conducted with the three MHFA instructors in England and a subgroup of three of the seven HSCs in Wales. The HSCs were purposively sampled to ensure that a representative from each of the six intervention schools was interviewed (HSCs delivered the intervention in pairs). Topic guides explored experiences of training delivery, including barriers to and facilitators of delivering a successful session, the extent to which instructors adhered to the course and motivations for any adaptations undertaken. In addition, the perceived acceptability of the training to course participants was also explored.

Interviews with head teachers

Semistructured interviews were conducted with the head teachers from all 25 schools. Topic guides explored the perceived role of schools in addressing the mental health and well-being of staff and students, the acceptability of usual practice, motivations to take part in the study, and the barriers to and facilitators of supporting teacher and student mental health. These data contributed to understanding of the acceptability and sustainability of the intervention.

Interviews with peer support service users

In the four intervention case study schools, we had planned to conduct semistructured interviews with teachers who had utilised the peer support service. However, despite trying a number of methods to recruit volunteers (e.g. the peer supporters asking colleagues they had helped, putting reply slips in the teacher questionnaires) we had only one volunteer and so we did not complete this part of the data collection.

Focus groups with peer supporters and recipients of mental health first aid for schools and colleges training

In the four intervention case study schools, focus groups were conducted with peer supporters and with teachers who had attended the MHFA for schools and colleges training. A focus group was conducted with each group within 6 months of training delivery, and a second focus group was conducted with each group approximately 18 months after training delivery. For the selection of participants for the focus groups, convenience sampling was used (i.e. all trainees in the case study schools were invited to attend and the actual make up of each group was determined by availability). Focus groups comprised four to eight participants.

Focus groups with peer supporters explored the acceptability of MHFA training and attendees’ preparedness to become peer supporters, the acceptability of delivering support, barriers to and facilitators of implementation of the peer support service, fidelity to the guidelines and reasons for adaptations undertaken, and sustainability of the intervention. Focus groups with 1-day MHFA for schools and colleges trainees covered the acceptability of and potential improvements to the training, impact of the training on how participants interacted with and supported students, and sustainability of the intervention. At the second time point, 18 months after training delivery, one-to-one interviews were conducted in place of a focus group in one school because of logistical difficulties in organising a focus group, but the same questions were explored.

Focus groups with teachers not in receipt of the mental health first aid training

It was planned to conduct one focus group in each of the four intervention and four control case study schools with teachers who did not receive any MHFA training, approximately 18 months after training delivery. For selection of the focus groups, random sampling was used (i.e. all teachers in the school were stratified according to sex and department, and participants were randomly selected from each stratum using random number generation software). If someone declined to participate, a replacement was randomly selected from the same strata until eight participants had been confirmed. Each focus group comprised six to eight teachers. In one intervention school, a one-to-one interview was held, as a focus group was too difficult to organise. In the intervention schools, focus groups explored participants’ views of the peer support service (including comparison with ‘usual care’), perceived barriers to and facilitators of uptake, and potential service sustainability. In the control schools, focus groups explored evidence of contamination, perceptions of ‘usual practice’ and views on a hypothetical peer support service.

Focus groups with year 10 students

In all eight case study schools, one focus group was held with a group of year 10 students (aged 14–15 years), approximately 18 months after training delivery. For selection of students, lists of all students in year 10 split by sex were used. Six names were selected using random number generation (three from each sex). Those students were invited to participate and were asked to also invite a friend to participate. As the target number for each focus group was six to eight, we did not need all students to consent to participate or to bring a friend. This methodology was used to help ensure that participants felt comfortable to talk during the focus groups. Each focus group comprised 3–12 students (although we had aimed for six to eight attendees, actual numbers were determined by those who actually attended on the day for whom we had been able to obtain parental consent). In all case study schools, the student focus groups followed a topic guide that explored participants’ views of teacher–student relationships and provision of mental health and well-being support in their schools. Given that the intervention was directed towards staff, students in the intervention schools were unlikely to have known about the intervention, and, therefore, were not asked directly about it. The data generated contributed to understanding of whether or not the mechanisms of change had been activated in the intervention schools and whether or not there was any indication of contamination in control schools.

Interviews with funding organisation representatives

Semistructured interviews were conducted with a representative from each funding organisation that contributed to intervention costs (n = 3). Interviews assessed fit with existing organisational priorities and the feasibility of sustained resource allocation to the WISE intervention, if it was found to be effective.

All topic guides are included in Appendix 4.

Analysis

Process evaluation data were analysed independently of the trial outcome data. Findings were then integrated following the completion of outcome data analysis so that the process evaluation could inform the interpretation of these data, specifically by explaining how the mechanisms of change worked (or did not work) in context and how the intervention had been implemented.

Qualitative analysis of the process data

All interviews and focus groups were audio-recorded, and recordings were transcribed and uploaded to NVivo version 11 (QSR International, Warrington, UK) for data storage and management. We followed Braun and Clarke’s83 thematic analysis approach in which emergent themes (codes) are identified and organised into an analytic framework (a coding frame). Separate frameworks for each data set were initially constructed (e.g. one for peer supporter focus groups, one for service user interviews, one for head teacher interviews, etc.). For each data set, two members of the research team independently coded two transcripts and then discussed and agreed the coding frame. A priori codes that mapped onto the process evaluation domains were included in the initial coding frameworks, along with novel codes that emerged from the data. One of the team then analysed the remaining transcripts within that data set according to the coding framework, with refinements being made to the frameworks as new themes emerged. A second team member coded 10% of the remaining transcripts to ensure intercoder reliability. All discrepancies and refinements to the coding framework were discussed between the two coders until an agreement framework was reached.

Once the frameworks were complete, common and contrasting themes across the different data sets were examined. The themes, pertinent to questions of reach, contamination/differentiation, fidelity, acceptability, mechanisms of change and sustainability, were drawn together and used to construct interpretation of the study’s impact on the main outcomes. Integration and triangulation of data adopted a complementary approach, whereby all participant narratives were equally privileged in the generation of new theoretical and empirical insights.83

Quantitative analysis of the process data

Descriptive statistics (totals, means and SDs) were produced for all checklists, evaluation forms, records of materials used, peer supporter logbooks and feedback reports, audits and observational data. Teacher questionnaire data regarding use and views of the peer support service and other support services were also summarised descriptively. When appropriate, logistic or linear regression models were conducted to examine the difference between groups.

Patient and public involvement

During the development of the intervention, senior leaders at schools and public health professionals working with schools were consulted about the intervention design and plans for evaluation. Findings from the pilot trial were discussed at a workshop with school teachers and leaders and, as a result of these discussions, changes were made to the peer supporter guidelines, training delivery was built into in-service time and the target population was changed from all staff to teachers. In the present trial, two senior leaders from local schools were part of the advisory group, and an employee of the Education Support (London, UK) was part of the Trial Steering Committee. We consulted the Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement’s (DECIPHer’s) young people’s advisory group twice,84 once to obtain comments on the student questionnaire and topic guide and a second time to discuss implications of the findings from the perspective of young people. We also conducted two workshops at the end of the study, one in England and one in Wales, with representatives from participating schools, local public health teams, Public Health Wales and third-sector organisations to share the findings and discuss implications for practice and future research.

Ethics considerations

Ethics approval for the study was granted by the University of Bristol’s Faculty of Health Sciences Research Ethics Committee (reference 28522).

Informed consent

The head teachers at all participating schools were given an information leaflet and had up to 1 month to confirm participation. At this point, they signed an agreement consenting to the school’s participation (or another senior leader signed with the head teacher’s consent), confirming that they would comply with all intervention and data collection requirements. All teachers and student participants were given information sheets at least 2 weeks before each questionnaire data collection, outlining the right to opt out of participation, the purpose of the study, any potential benefits or harms in taking part, details about data storage and details about what the data would be used for. The information sheets also contained contact information for the study team. Letters containing the same information were posted or e-mailed to parents by schools (in accordance with their usual procedures for communicating with parents) at least 1 week before student data collection, accompanied by opt out forms. Any student whose parent returned the opt out form did not take part in the questionnaire data collection. Consent was assumed from participants who completed a questionnaire. Written consent to participate was obtained from all staff peer supporters and from all participants who agreed to take part in an interview or focus group. We attempted to obtain parental written consent for students invited to take part in a focus group, but, where this was not returned, the school contact obtained verbal parental consent over the telephone.

Safety reporting procedures

School contacts and those delivering the intervention (i.e. MHFA trainers, HSCs and those trained as peer supporters) were asked to contact the study team within 2 working days if any untoward incident or adverse event (AE) occurred to a student or member of staff (1) as a direct result of taking part in the WISE study or (2) because of changes that had occurred in the school environment because of participation in the WISE trial (e.g. heightened awareness among staff of mental health problems, leading to inappropriate referrals to specialist help sources for ‘normal’ student behaviour). In these cases, study-specific AE/incident forms were completed, recording information on the event. Members of the research teams in Bristol and Cardiff were also required to complete a form about any incidents or AEs that they encountered during data collections. All AE/incident report forms were discussed with the principal investigator to assess seriousness and to explore causality. All AEs deemed to be ‘serious’ [i.e. a serious adverse event (SAE)] were to be reported to the sponsor within 24 hours. All AEs and SAEs not deemed to be related to the study were reported to the Trial Steering Committee at the next scheduled meeting. There were no SAEs reported that were suspected to be related to the intervention.

All student questionnaires were checked on return to the university for any indication of serious harm. If a student had written something on their questionnaire indicating that they were at risk of serious physical or emotional harm, the principal investigator (Bristol) or study manager (Cardiff) was to be informed and they would contact the child protection officer at the school to report the concern. No such issues arose, possibly because the questionnaire did not directly ask about things indicating risk of serious harm.

If a teacher wrote something on their questionnaire indicating that their life may be in danger (e.g. ‘I would like to bring an end to it all’), the plan was for the study team to break anonymity and write to the individual at their school, in an envelope marked confidential, expressing concern and recommending seeking help. The teacher participant information sheets made clear the circumstances in which we would break anonymity and the action that would be taken. No such action was needed during the course of the study.