Process evaluation

The information presented in this chapter has been published elsewhere.99–102

Introduction

Parallel to the main ORBIT RCT, a process evaluation was undertaken, which followed the guidelines as recommended by the MRC’s framework on process evaluations of complex interventions.103 ORBIT is regarded as a complex intervention because it is composed of multiple components with the potential for interactions between them and with several possible outcomes.104 The MRC guidelines stipulate that in order to carry out a process evaluation of a complex intervention the following three key functions must be examined: (1) implementation (identifying what was delivered and how this was done or achieved), (2) mechanisms of impact (factors that contributed to the delivered intervention producing or not producing change) and (3) context (contextual factors external to the intervention that affected implementation, intervention mechanisms and outcomes).

Implementation fidelity

Implementation can refer to how an intervention can be implemented within routine clinical practice. However, this can only be achieved once an intervention has shown efficacy in an outcome evaluation. Implementation can also refer to how the delivery of an intervention was achieved within the context of a RCT and the structures and processes through which an intervention was delivered as intended.103 This is often termed ‘implementation fidelity’. In short, implementation fidelity refers to the degree to which a study was implemented according to design or protocol. If an intervention is designed according to well-established theoretical and empirical underpinnings, including identifying ‘essential ingredients’ and their subsequent relationships with the intended outcomes, implementation fidelity is seen as crucial.105

There are multiple benefits to a trial that rigorously assesses implementation fidelity. These include improving the validity of intervention outcomes,106,107 enabling replicability108 and aiding in the understanding of why an intervention succeeded or failed in its intended outcome (e.g. symptom reduction).109 For example, a study may erroneously determine that the lack of impact of an intervention was caused by elements of the programme itself if no process measures were evaluated (i.e. a type III error).110 Therefore, it is essential that a RCT that includes a process evaluation should contain a rigorous analysis of implementation fidelity.

Mechanisms of impact

The second key component involves exploring the mechanisms through which the intervention produces change. This is crucial to understanding how the effects of the intervention occurred and how these effects might be replicated in future iterations of similar interventions.111 By exploring the mechanisms of potential impact of an intervention, a process evaluation can provide a better understanding of the causal pathways and identify any unexpected consequences.112 Mechanisms of impact include mediators, moderators and contextual factors associated with change.

Mediators

Part of the MRC guidance for conducting an analysis of impact involves assessing the extent to which the causal assumptions underpinning the intervention can be tested through mediation. Mediator analysis refers to the examination of the mechanisms underlying an intervention’s theory of change. This means extending the fundamental assumptions from ‘if intervention A is implemented then B will occur’ to ‘if intervention A is implemented, this will lead to a change in the mediating variable or variables, which will lead to a change in outcome B’.113 Mediator analysis is useful as it can uncover how an intervention produces change, and mediators generally have an impact during treatment.

Moderators

Moderator analysis typically involves assessing who benefits from an intervention, and a moderator generally precedes treatment. This involves an assessment of any pre-existing characteristics of the sample that may predict who will gain the most from an intervention. In the ORBIT trial, child and parent participants’ as well as therapists’ backgrounds, beliefs and circumstances may have dictated the way in which they interacted with the intervention.102 Thus, it is important to the evaluation of an intervention to pursue contextual factors and moderator configurations in order to understand any potential variability in outcomes.

Overall, mediator and moderator analyses are key to understanding how and for whom an intervention does or does not work, which allows for an identification of ‘essential ingredients’. This will ensure that an intervention is implemented on a wider scale with only the essential components being delivered and targeted at the people who will benefit most.

Context

The final component is context, which refers to any factors external to the intervention that may have acted as barriers or facilitators to the way it is implemented or to the outcomes. The uptake and use of DHIs are largely dependent on context; thus, understanding the context is crucial to interpreting the findings and making generalisations from the results. In conclusion, a process evaluation of a complex intervention such as ORBIT is crucial to explaining trial outcomes and will aid in understanding its overall implementation.

Aims and objectives

The aims of the ORBIT process evaluation were to understand the causes of the observed behaviour change data obtained from the RCT, to explore the fidelity of intervention delivery, acceptability of the intervention and reasons for observed variations in uptake and use and to consider the resources and implementation processes required.102

Specific objectives were:

1. to assess the fidelity, reach and dose of intervention delivery
2. to explore whether any of the intervention features were tailored for individual needs, enabling potential recommendations for adaptations
3. to explore the intervention from the perspectives of children, parents, therapists and clinicians in order to gain a deeper understanding of potential mechanisms underlying participant behaviour change whilst probing for any unexpected consequences
4. to evaluate any factors external to ORBIT that may have affected delivery (i.e. the environment and its characteristics) or whether its mechanisms of impact worked as intended
5. to consider the resources and implementation processes required for effective implementation, uptake and use of the intervention.

In this chapter, we present a planned, mixed-methods, two-phase process evaluation of the ORBIT trial following the MRC’s 2015 guidelines103 to explore the fidelity of delivery and the contextual factors influencing engagement with the intervention (phase 1) and the impact of the intervention and mediators, contextual factors and moderators of this impact (phase 2).101

Phase 1: fidelity of delivery and contextual factors influencing children’s level of engagement with ORBIT

The phase 1 results were published in Khan et al.99

Here we report phase 1 of the process evaluation of ORBIT. The aim of this study was to conduct the first phase of a two-stage process evaluation of ORBIT as outlined in the protocol.101 Phase 1 focuses on intervention implementation by exploring the fidelity of delivery experienced by participants through usage statistics, reach and the acceptability of the intervention. It also investigates contextual factors associated with the observed variation in uptake and usage by examining the components specified in MRC guidelines.103

Introduction

The MRC have developed guidelines for conducting process evaluations of complex interventions such as ORBIT.103 In order to evaluate intervention implementation, MRC guidelines for process evaluations suggest researchers assess (1) reach (the extent to which a target audience encounters the intervention), (2) dose (how much intervention is delivered and received), (3) fidelity (the quality of what was delivered) and (4) adaptations (any modifications made to an intervention in order to achieve better contextual fit).102 The intended target audience for ORBIT was CYP with tic disorders; however, there were pertinent questions that could be asked, such as whether there were socioeconomic biases in who was reached. In terms of dosage, the ORBIT protocol1 states that the intervention should consist of 10 individual intervention chapters following a suggested frequency and total duration of 10–12 weeks. There were four core chapters (chapters 1–4), and this was deemed the minimum requirement for treatment completion. There were six additional chapters offering information regarding reinforcement, further practise and relapse prevention. For DHIs, the fidelity of delivery of the intervention is assured by the online delivery platform. However, the intervention that is experienced by the user is highly dependent on the extent to which they engage with the intervention and use it as intended. Hence, in this process evaluation, the focus is on usage and the proportion of participants receiving the predefined ‘minimum effective dose’ of four or more chapters.102 Finally, understanding adaptations to the intended intervention involves exploring whether these improve its contextual fit or compromise its functioning,114 or whether they represent innovation or intervention drift.115 Participants were able to make modifications to various components of the intervention, such as the ‘tic stopwatch’, which was used to self-time the length of tic control.

Methods

Study design

This study followed MRC guidelines103 for the process evaluation of complex interventions and used a mixed-methods, longitudinal design to explore the implementation fidelity of an online intervention for CYP with tics1 and the contextual factors that influenced their level of engagement. Here, an overview of the methodology of the process evaluation, including study-specific outcome measures, is given as context to the main trial.

Participants

The sample included in the quantitative component of the process evaluation consisted of key information from all participants (N = 112) from the intervention arm of the RCT. The sample included in the qualitative component of the process evaluation consisted of interviews with child and parent participants (target n ≥ 20), interviews with all therapists delivering the intervention or supervising the therapists and interviews with referring clinicians (target n > 5).

Participants were only contacted if they gave explicit written consent to participate in an interview for the ORBIT trial and, for a child under 16, parental assent was obtained.

There were two therapists at the Nottingham study site and two at the London study site, with one supervisor at each site. The therapists were educated to graduate level and were not required to have previous experience in treating tic disorders but were trained on the platform and its contents and received regular expert supervision. Therapists received training in BT for tics during the trial. ‘Clinician’ refers to any health-care professional (usually a doctor) who was responsible for referring participants to the ORBIT trial and was purposively selected from the PIC sites. Whilst these clinicians were not explicitly involved in the ORBIT trial, the main purpose of interviewing them was to gain their views on potential implementation in routine care.

Measures

Demographic and clinical data

Demographic and clinical information was recorded from a baseline demographics questionnaire. These data included the child’s age, residence (full postcode), gender, ethnicity, parental education level and occupation, all current suspected or confirmed diagnoses and interventions and medication use.

Index of multiple deprivation

The index of multiple deprivation (IMD) is a relative measure of deprivation across seven different domains: income deprivation, employment deprivation, education, skills and training deprivation, health deprivation and disability, crime, barriers to housing and services and living environment deprivation (https://www.gov.uk/government/statistics/english-indices-of-deprivation-2019). Based on the postcode, a rank of deprivation associated with participants’ area of residence was calculated from 32,844 small areas or neighbourhoods in England, with higher ranks indicating greater deprivation. Ranks were recoded into quintiles, with quintile 1 being the most deprived and quintile 5 being the least deprived.

Usage metrics

Online usage data were collected and recorded from participants throughout the trial. These included the following measures: number of chapters completed per child and per parent; total therapists’ time per child and per parent; individual therapists’ telephone time with participants; volume of written communication (total number of characters) submitted by child and parent via the online system; total number of logins for child and for parent; average time between each login (in days) for child and for parent; and average pages visited per login for child and for parent.102

Satisfaction and treatment credibility

At the 3-week post-randomisation point of treatment, all participants were asked to rate treatment credibility. Two questions were asked: one relating to how well suited the participant felt the intervention was for helping the CYP to manage their tics and the other about how much better they expected to feel as a result of the intervention. The responses were on a Likert scale of 0–4 for each question, with higher scores indicating higher treatment credibility. At the primary end point, all participants were asked to rate their satisfaction with the intervention. Eight satisfaction questions were asked, with responses rated on 0–4 scales, meaning the overall satisfaction score was out of 32.102

Semistructured interviews

In line with previous literature,116,117 four semistructured interview schedules were developed. The child and parent interview schedules were drafted and underwent revision from the main process evaluation researcher and three academics. Participant interviews aimed to elicit information regarding why they took part, their views of the content and chapters, impact of the therapy, what they found helpful or not, barriers to participation and their views of their assigned therapist. The revised drafts were sent to two dyads of the PPI group – including two children with tics – for feedback and were revised accordingly. The PPI input was invaluable, as it ensured the questions were understood by children as young as 9 years of age.102

The therapist interview schedules were drafted and underwent revision from the main researcher and the same three academics, as well as with input from a therapist and a clinical researcher with specific expertise in the field. Therapist interviews aimed to elicit information about their role on the trial, what specific skills they felt they needed, any training needs identified, experiences of supervision and their perceptions of their interactions with participants.102

The clinician interview schedules were drafted and underwent revision from the same team as above and were guided by normalisation process theory (NPT).118,119 NPT attempts to identify factors that promote and inhibit the routine incorporation of complex interventions into routine practice. It also attempts to explain how such interventions work, looking not only at early implementation, but also beyond it, whereby an intervention becomes so embedded into routine practice that it is normalised.119 As the purpose of the clinician interviews was to explore their views about the feasibility of integrating the intervention into everyday practice, including any potential barriers to or facilitators of this, the NPT framework approach seemed the most appropriate. The clinician interview schedule questions aimed at eliciting information on how they got involved in the ORBIT trial and why, their experience of recruiting for the trial, including factors that affected recruitment, and how the NHS could incorporate the intervention into everyday practice.102

Data collection

Quantitative process data were collected simultaneously along with enrolment, intervention delivery and outcome data collection in the main RCT. Qualitative process data were collected at various time points. Interviews with therapists involved in the ORBIT trial were conducted early in the study and near the end of recruitment to gain an understanding of their experience at different time points. Interviews with referring clinicians were conducted at the end of recruitment. Interviews with CYP and parents were conducted following completion of the intervention at the 3-month follow-up assessment (primary end point) in the main RCT to minimise the risk of bias in outcomes. Recruitment for the interviews began in August 2018 and ended in October 2019.

All interviews were conducted either face to face, by telephone or via videoconferencing (Webex or Skype). Younger children were interviewed together with their parents, while older children were interviewed separately. Participants were purposively sampled so that a diverse range of views on the intervention were voiced. This included ensuring perspectives were heard from participants of a range of ages, genders, ethnicities and levels of interaction with the intervention. The overall sample enabled a diversity of views of the intervention to be expressed and ensured that data reached a level of saturation. In addition to the interviews, at the end of treatment all participants were asked to give their overall feedback on the intervention, in which they could provide open-ended responses.

Data analysis

The quantitative data set was presented with total numbers and percentages and means with SDs or medians (ranges), if not normally distributed. Data were tested for normality using the Kolmogorov–Smirnov test. A principal component analysis was run to determine a composite measure of level of engagement. Correlations between variables were examined using bivariate Spearman correlations, and t-tests were conducted to explore any significant differences between groups with chi-squared tests to explore for differences between categorical variables. A multiple linear regression was calculated to identify predictors of engagement with independent variables. All statistical analyses used a significance level of p < 0.05 and were conducted using IBM SPSS Statistics version 27 (IBM Corporation, Armonk, NY, USA).

All interviews were recorded either by videoconferencing software or by Dictaphone and were then transcribed verbatim. Transcripts were checked for accuracy against the recordings, with any corrections made as appropriate and anonymised for confidentiality purposes. As the process evaluation was a combination of exploration and description, the framework method120 of analysis was used to identify, analyse and report patterns within the transcribed interviews. Moreover, the steps outlined by Gale et al.121 were systematically followed to create an overall framework matrix using categories of engagement and contextual factors. Consistency of analysis was ensured throughout using a codebook and through frequent meetings between researchers. Researcher bias was minimised through regular cross-checking of data and outcomes by members of the research team.

The software package QSR NVivo 12 (QSR International, Warrington, UK) was used to analyse the interview data. In addition, the end-of-treatment feedback questionnaire was exported into an Excel spreadsheet and quantitative content analysis122 was performed. Overall, the findings from the qualitative analysis were linked to relevant quantitative adherence outcomes and contextual factors to assess which potential variables might have influenced implementation fidelity and in what way, in an approach termed ‘triangulation’.123

Ethical considerations

Ethical approval for the process evaluation was obtained from North West–Greater Manchester Central Research Ethics Committee as part of the ORBIT trial (REC: 18/NW/0079). All child and parent participants provided written informed consent and all interview participants provided oral consent for audio-recording.

Results

Overview of qualitative sample

From the intervention group, 38 children and their parents were contacted to participate in semistructured interviews. Eighteen of these did not respond or declined to participate. Overall, semistructured interviews were conducted with children (n = 20) and parents (n = 20), therapists (n = 4) and clinicians (n = 6). The average age of child interviewees was 12 years (range 9–16 years) with 16 (80%) of the sample being male and four (20%) being female. Most of the sample was white (n = 18, 90%). The mean YGTSS-TTSS was 28.8 (SD 7.2), with a range of 13–45 for child interviewees. All 20 of the interviews with the parents were with the CYP’s mother, with all 20 having completed at least further education. One of the therapist interviewees was a therapist’s supervisor and half of the clinicians were consultant psychiatrists (n = 3, 50%).

Reach

A total of 445 families expressed an interest in taking part in the study either through self-referral via Tourettes Action’s website (n = 251) or via clinic referral (n = 194); however, 47 were subsequently uncontactable and 90 were ineligible to take part for reasons such as having had BT in the last 12 months or being due to start BT, living outside of England or being an immediate risk to self/others.102 Of the 308 potentially eligible CYP, 84 families (27.3%) declined to take part due to the child not wanting to participate, the family not wanting to attend the baseline appointment, having insufficient time or no specific reason given. Thus, 112/224 CYP (90 male, 22 female) with an average age of 12.2 years (range 9–17; see Table 4 for baseline characteristics of sample) were randomised to the intervention arm of the ORBIT trial and were included in the process evaluation. The sample was predominantly white (n = 96, 85.7%) and well educated, with just over half (n = 60, 53.5%) of the participants’ mothers having completed university/higher education.102

The median IMD rank was 19,318, with a range of 147–32,668 (out of 32,844). Of the 112 participants, 8 (7.1%) were in the most deprived quintile (1), 31 (27.7%) in quintile 2, 18 (16%) in quintile 3, 26 (23.3%) in quintile 4 and 29 (25.6%) in the least deprived quintile (5). Although the reach of the intervention was not limited geographically, for the purposes of the research participants did have to attend a baseline screening assessment at either the Nottingham study site (n = 57, 50.9%) or the London study site (n = 55, 49.1%) depending on personal preference and/or location of residence. All participants were based in England, with 63 (56.3%) participants living in towns, 30 (26.7%) in cities and 19 (17%) in villages.102

In terms of clinical characteristics, the intervention reached a moderately severe symptomatic sample with a mean YGTSS-TTSS of 28.4 (SD 7.7) out of a maximum of 50, with a range of 12–50. Most participants (n = 98, 87.5%) were not on any medication for their tics, and just under half of the overall intervention sample had no diagnosed or suspected comorbidities (n = 51, 45.5%). Of those who did have a comorbid diagnosis, the most common was anxiety disorder (n = 34, 30.4%) followed by ADHD (n = 26, 23.2%). An assessment of depressive symptoms by the MFQ showed a mean score of 16.3 (SD 11.3) out of 66, with 14 (12.5%) participants scoring above the cut-off (≥ 29), suggesting clinically significant depression.124

It was not possible to interview people who had not taken part in the study, so the qualitative data threw little light on reach; however, one clinician identified that some families were worried about the level of commitment involved and associated travel to one of the study sites under the category ‘Initial response to ORBIT’ and theme clinician perceptions of and contribution to recruitment:102

So children quite often with autism … other kind of family reasons where I think they were just worried about the level of that kind of commitment to … an intervention to be able to kind of travel to Nottingham or London for the initial assessment.

Dose

Child participants completed an average of 7.5 (SD 2.7) out of 10 chapters of the intervention and their parents completed an average of 7.6 (SD 2.8), indicating high engagement. Only 13 (11.6%) child participants and 17 (15.2%) parents failed to meet the criteria for treatment completion (i.e. minimum of first four chapters completed as per protocol), with a total of 99 (88.4%) child participants and 95 (84.8%) parents completing their treatment, meaning that adherence to the intervention was high. Indeed, 46 (41%) CYP and 52 (46.4%) parents completed all 10 chapters of the intervention and only one child participant failed to complete any chapters. Child participants logged on to the online treatment platform an average of 19.8 (SD 10.9) times throughout the 10–12 weeks, with an average of 4.2 (SD 2.6) days between logins. In terms of total interactions with their assigned therapist, child participants required their therapist’s online assistance for an average of 59 minutes and 14 seconds (SD 29 minutes and 8 seconds) over the course of treatment, which results in approximately 6 minutes per child per week. Parents interacted online with their assigned therapist for an average of 1 hour, 23 minutes and 55 seconds (SD 42 minutes and 45 seconds), which results in approximately 8 minutes per parent per week. Of 112 CYP, only 2 (1.8%) were contacted by telephone by their assigned therapist. Of 112 parents, 49 (43.7%) were contacted by telephone by their assigned therapist.

Interview data relating to participants’ perceptions of ORBIT organisation (category: ‘ORBIT programme content’) covered the implementation component of dose. Although most participants felt that the intervention was just the right length, some CYP wished to have a longer period in which to access their therapist:

I just liked doing the whole bit of ORBIT and chatting to my therapist, but I think it was too short. Cause I could only chat to my therapist for 10 weeks, but then we had a full year logging on to ORBIT but we could not chat to our therapist which I found a bit annoying.

Fidelity

At the 3-week point of post-randomisation, participants were asked to rate treatment credibility. Treatment credibility was rated highly by child participants, with a mean score of 6.4 (SD 1.5) out of 8. Furthermore, at the primary end point, participants were asked to rate their overall satisfaction with the intervention. Child participants were highly satisfied with the intervention, with a mean score of 24.8 (SD 5.2) out of a total of 32. At the end of treatment, participants were asked to give their feedback on the intervention within the online platform and they were able to give open-ended responses. Only 67 (59.8%) child participants provided this feedback. From the quantitative content analysis conducted, four categories were generated relating to implementation fidelity, namely: ‘limitations of ORBIT’ (n = 51), which captured how participants felt that overall ORBIT was helpful but was limited by certain factors; ‘ORBIT as a suitable treatment’ (n = 49), which suggested that participants felt that the online delivery of treatment for tic disorders was suitable; ‘problems with using ORBIT’ (n = 20), which captured those participants who stated that they felt ORBIT was not helpful to them or was associated with negative factors; and ‘feeling supported’ (n = 19), where participants mentioned that they felt supported in a way that they had never been before (e.g. by their therapist).102

Although satisfaction was rated as high, some participants felt that the role of the therapist was somewhat misleading. This was captured by the theme expectations of role of the therapist (category: ‘Initial response to ORBIT’). Some felt that a therapist was not needed for the delivery of the intervention:

Like that just I don’t like emailing so I think I felt a bit awkward cause I didn’t really know how to write back but I felt most of the comments were quite generic … I don’t know just I’d say something and [therapist] be like ‘oh well done’ … but I don’t think [therapist] necessarily has to be there. I think you could have done it on your own.

The therapists themselves concurred with this, suggesting that perhaps they should not have been called ‘therapists’ within the ORBIT study:

I think part of it would come down to whether we would want to use the word ‘therapist’ within ORBIT because there’s a lot of semantics and meaning about that word and I’m not sure off the top of my head if therapist or … what’s the lay meaning of therapist basically? Does that mean psychotherapist, does that mean someone who’s got a doctorate, who knows? So, everyone could … participants come into that with their own meaning, and it also assumes that I … they’ve got expectations about what a therapist is, it assumes that I’m the expert and I really felt like I wasn’t in this. My supervisors were experts.

Adaptations

Regarding adaptations, the intervention did not appear to evolve in any way from the original plans. There appeared to be consistency in the way the intervention was delivered and received. Interviews with therapists confirmed how consistency was maintained in delivery. For example, they created a list of standardised responses to common queries (theme: ‘Strategies to support therapists’):

We had standardised documents, of like a collection of standardised responses so any time we’d come across something unique or difficult or not immediately obvious to answer, after sort of emailing around and reviewing potential answers we’d obviously say how to come up with an answer to send to the participant and once I’d done so, I’d add a section into the collection of responses and add it in. So basically, we had something we could look at and call upon when we see someone and go ‘look, we’re not sure how to answer that, let me check this document’ and then you can see if there was anything similar, or it’s been answered before, umm, that was very useful …

Parts of the intervention were designed to be adapted by the user and tailored to their needs and preferences, such as the ‘tic stopwatch’ and ‘tic ladder’ (hierarchy of exposure exercises). For instance, on the ‘tic stopwatch’ participants could modify the difficulty levels of the given exercises, such as the ‘focusing on tic signals’ task, which could be altered depending on how difficult the participant found it. The ‘tic ladder’ could also be modified so that participants could add their own places to the hierarchy depending on where they tic the most frequently. This was captured by the theme ‘Adaptations’:

I had to answer questions in the chapters and when I finished it, I could go back and change it and I could change my ladder when I do my tics and where I do my tics most often and my tic list of what I have. I liked the idea that I could change it. And it helped me.

Contextual factors influencing intervention implementation

Engagement with a complex intervention is seen as crucial in determining the effective implementation of the intervention. Thus, in order to establish a measure of intervention implementation that captured both the breadth and depth of participants’ usage, a principal component analysis with Varimax (orthogonal) rotation was conducted on the seven items relating to the dose of intervention received. The analysis suggested a two-factor model. The strongest factor accounted for 47% of the variance (eigenvalue 3.3) and seemed to capture strength of engagement with the intervention. Factor scores ranged from −2.65 to 2.26, with a mean of 0.001 (SD 0.99), and these scores were used as the engagement measure.102

Data met assumptions of independence and linearity and did not deviate substantially from normality; therefore, parametric tests were conducted. A two-tailed t-test found that participants who were enrolled at the London site (mean 0.25, SD 0.90) scored significantly higher on engagement compared to those enrolled at the Nottingham site (mean −0.22, SD 1.03), t(109) = −2.58, p = 0.011. Moreover, those who were self-referred (mean 0.16, SD 0.94) scored higher on engagement than those who were referred through clinics (mean −0.24, SD 1.04), t(109) = −2.06, p = 0.041. Spearman’s rho correlations were run to determine the associations between engagement and various contextual factors. CYP’s engagement factor score was strongly correlated with parents’ chapter completion (ρ = 0.73, n = 111, p < 0.001) and moderately correlated with therapist time with parents (ρ = 0.46, n = 111, p < 0.001). There were no significant relationships between CYP’s engagement factor score and age, parental education, IMD, YGTSS-TTSS at baseline or MFQ baseline score. There were also no statistically significant relationships between a child’s gender, comorbidities or use of tic medication and CYP’s engagement.102

A multiple linear regression was conducted with CYP’s engagement factor score as the dependent variable and site, child’s age, child’s gender, IMD, YGTSS-TTSS at baseline, method of referral, parental education, therapist time with parents and parents’ chapter completion as the independent variables. The results of the simultaneous regression indicated that collectively the independent variables accounted for a significant amount of variance in the CYPs engagement factor score, F(10,100) = 20.84, p < 0.001, R² = 0.64. There was no evidence of multicollinearity, with all tolerances above 50% and all variance inflation factors below 2. Only parents’ chapter completion [β = 0.69, t(110) = 10.18, p < 0.001] and therapist time with parents [β = 0.19, t(110) = 2.96, p = 0.004] were significant independent predictors in the model.102

Under the framework category ‘participant contextual factors’, the theme of parental persuasiveness was generated. Many of the parents interviewed outlined that they were often the main motivating force behind their child’s level of engagement by reminding their child to practise the learnt techniques:

If he’s got a really bad tic and I’ll say to him you know, [child’s name] use your tic timer in your head, try and see how long you can do he will then do it … but he doesn’t really use the techniques himself without being reminded to … So I suppose that was a little bit of a disappointment.

Discussion

This first phase of a two-stage process evaluation used a mixed-methods approach to investigate the extent to which the ORBIT intervention was implemented as planned within the context of an RCT and to explore participants’ experiences with the intervention and the contextual factors influencing children’s engagement.102 In doing so, this made it possible to identify reasons for variation in uptake, usage and engagement, to reflect on how implementation may ultimately give greater confidence in the outcomes and to outline lessons for potential future implementation within routine care. Uptake of the intervention was high, with nearly 90% of participants receiving the predefined minimum effective dose of the first four chapters completed. The median uptake was eight chapters, and only one child failed to access any chapters. Fidelity of delivery was also excellent, with participants reporting high levels of satisfaction and acceptability.102

The intended sample of CYP with a diagnosed tic disorder was reached, with 7.1% of families residing in the most deprived areas (IMD quintile 1) and over a quarter (25.6%) of the families residing in the least deprived areas (IMD quintile 5). As over half (53.5%) of the CYP’s mothers had completed graduate-level education against a UK average of 42%,125 it seems that more advantaged families may have been over-represented. This is a concern, as one of the aims of ORBIT was to increase access to evidence-based therapeutic interventions for CYP with tic disorders,100 particularly as access to services is generally limited for those from lower socioeconomic status backgrounds.126 Although the initial baseline visit with associated travel may have been a disincentive to more disadvantaged families, this would not be relevant if ORBIT was delivered entirely remotely in routine care rather than as part of a RCT. Moreover, there was no evidence that socioeconomic factors influenced CYP’s engagement with ORBIT. Furthermore, a child’s age, severity of tics, well-being and comorbidities did not appear to influence their level of engagement with the intervention, providing further evidence that the intervention would have a wide reach within routine clinical care. However, due to the various factors relating to this RCT specifically as opposed to routine care, caution should be taken when interpreting the results from this study concerning reach.102

London study site, self-referral and higher parental engagement were all associated with higher levels of child engagement. The London site is a world-renowned centre of excellence for paediatric care, which may have increased parents’ motivation for treatment. However, the only independent predictor of child engagement in the multivariate analysis was level of parental engagement with the intervention, as measured by their chapter completion and by parent time with the therapist. This is consistent with previous literature,127–129 which found that parental involvement was particularly key for younger CYP to assist with their engagement with therapeutic interventions, which in turn leads to better outcomes.130–132 It has been shown in the literature that parental engagement may impact a provider’s ability to implement parent- and family-focused evidence-based treatment with fidelity.130 Therefore, it is crucial to understand the role of parental support for the implementation of DHIs for children, as without attention to the key processes of child and family engagement efforts to improve the effectiveness and efficiency of the treatment are less likely to succeed.100

An interesting finding relates to the usage and interactions with the therapist within this study. Therapists interacted online with their assigned child participants for an average of approximately 6 minutes per child per week, which is lower than the 24‐minute average time per week participants interacted with their therapists in the Swedish pilot trial of the BIP platform.43 In the UK, study therapists were encouraged to use pre-prepared scripts to respond to participants. Their responsibilities involved reinforcing the ORBIT treatment material, with the aim of spending approximately 6 minutes per week responding to each child, which was in the therapist guidance given by supervisors. It was also apparent from qualitative interviews that many participants felt that the term ‘therapist’ was somewhat misleading. Some participants felt that ‘therapist’ had connotations of a clinically trained individual delivering an intervention. This may have limited their reliance on the therapist. Therefore, in any implementation of this intervention within routine health care, it would be sensible to alter this title to ‘coach’, ‘guide’ or ‘mentor’ to better reflect the role of the therapist.102

Conclusions

In conclusion, the intervention had high fidelity of delivery and was evaluated positively by CYP, although some participants suggested some minor improvements, and reach may have been constrained by the nature of the RCT. Parental engagement was a strong, independent predictor of intervention implementation, which has important implications for the design and implementation of digital therapeutic interventions into CAMHS.102

Phase 2: factors influencing the effectiveness of ORBIT

The phase 2 findings have been published in Khan et al.100 and also Khan.102

Here we report phase 2 of the process evaluation of ORBIT. The aim of this study was to conduct the second phase of a two-stage process evaluation of ORBIT as outlined in the protocol.101 Phase 2 focuses on the impact and contextual factors influencing effectiveness as well as factors moderating and mediating the relationship between implementation of the intervention (child engagement) and the impact of the intervention on tic severity and clinical improvement.102

Introduction

Although BT is effective and avoids the unpleasant side effects associated with medication, access to therapy is limited due to insufficient numbers of specialists and the uneven geographical distribution of services relative to demand.21 DHIs have been shown to be effective for a range of neurodevelopmental disorders in CYP, including tic disorders, and thus offer potential to widen access to evidence-based behavioural treatments.133 However, despite an expanding body of evidence to support the acceptability and effectiveness of online therapy, uptake of DHIs into clinical practice has been disappointing.134 Previous research has focused on intervention outcomes, with little attention given to the mechanisms of impact: the way in which the intervention components and participants’ responses to the intervention produce change. Knowledge of how and why a DHI works increases the potential for replication across contexts.103 Furthermore, understanding mechanisms of impact is crucial for assessing core components of an intervention (e.g. ‘essential ingredients’), which helps with defining the minimum therapeutic dose.102

There are no studies in the tic disorder literature assessing mechanisms of impact of digital interventions; however, studies of face-to-face BT for tics have suggested that clinical factors can moderate the effectiveness of behavioural treatment. Sukhodolsky et al.135 found that the presence of tic medication significantly moderated impact. For participants receiving 10 weeks of BT, medication status did not impact on effectiveness. In contrast, participants in the psychoeducation and supportive therapy group who were receiving medication showed significantly greater tic reductions than participants not on medication. Tic phenomenology, age, gender, family functioning, treatment expectancy and comorbidities did not moderate response to treatment. A more recent study that examined moderators of treatment outcome after adolescents with CTD received either individual or group therapy found that a higher level of anxiety and a higher premonitory urge to tic favoured treatment in groups, whereas increased sensitivity and higher depression symptomology favoured individual treatment.55

This second phase of the process evaluation study uses qualitative and quantitative data to explore the contextual factors influencing effectiveness and the factors moderating and mediating the relationship between implementation of the intervention (level of child engagement) and the impact of the intervention on tic severity and clinical improvement.102 This provides insight into how and why outcomes occurred under given circumstances and what mechanisms underlie these impacts. To the best of our knowledge, this is the first study to examine potential mediators and moderators of an online intervention delivered to CYP with tic disorders.102

Methods

Study design

This study used a mixed-method, longitudinal design to explore the mechanisms of impact of an online, therapist-supported, parent-assisted ERP intervention for CYP with tics delivered via the BIP TIC platform.43 The study used quantitative analyses to explore contextual factors, mediators and moderators of impact and qualitative analyses to illuminate those relationships in more depth.

Participants

Participants were CYP randomised to receive the active intervention (N = 112) and their parents. Participants were purposively sampled to represent views from a range of ages, genders, ethnicities and levels of interaction with the intervention. Four therapists and six referring clinicians agreed to participate in semistructured interviews.

Measures

Outcomes

The primary outcome for assessing impact was the tic severity change score as measured by the YGTSS-TTSS4 at the primary end point (3-month post-randomisation) minus the YGTSS-TTSS at baseline. In this trial, the range of YGTSS-TTSS scores at baseline was 12–50. Hence, possible scores on the tic severity change measure could range from 38 (maximum possible deterioration) to −50 (maximum possible improvement in tic severity). The secondary outcome measure used was the CGI-I.65

Child engagement factor score

In order to establish a comprehensive measure of child’s level of engagement with the intervention, a principal component analysis with Varimax (orthogonal) rotation found that seven variables loaded on the engagement factor representing higher dose of intervention received more logins, more chapters completed, more therapist time for child (SMS-based support), greater total number of characters submitted to therapist within the online platform, fewer days between logins, fewer number of pages visited per login and less telephone contact by the therapist (engagement prompts).99

Contextual, mediator and moderator variables

Based on previous research on BT for tic disorders and theoretical assumptions,135,136 as well as recommendations about the domains that should be included when conducting moderator and mediator analyses in paediatric RCTs,137 the following four potential mediator variables were selected: depression change (as measured on the MFQ),7 anxiety change (as measured on the SCAS),138 treatment satisfaction (Likert scale: overall satisfaction 0–32) and treatment credibility (Likert scales: how well suited and how much better do you expect to be from treatment 0–8). Treatment satisfaction and credibility scales were constructed by the research team. The following seven potential moderator variables were selected that would potentially influence the strength of the relationship between intervention (level of engagement) and efficacy: medication use, comorbidity, parental support (number of chapters completed), baseline tic severity (as measured on the YGTSS-TTSS), age, deprivation (as measured on the IMD) and mother’s level of education. Contextual variables are those that directly influence level of effectiveness, and they were selected based on findings from phase 1 of the process evaluation and outcomes from the exploratory correlational analyses.102

Data collection

The data collection is described in detail in the process evaluation protocol101 and elsewhere within this monograph.

Data analysis

Data were tested for normality using the Kolmogorov–Smirnov test. A repeated-measures analysis of variance (ANOVA) was initially run to determine an effect on tic severity over time. The effect size of the intervention was calculated using Cohen’s d.139 Spearman correlations were used to analyse relationships between the outcome variables (YGTSS-TTSS change and CGI-I) and all contextual, mediator and moderator variables. This was done to establish whether a relationship between the variables existed prior to the next stage of analysis. Mediator analyses were conducted using the bootstrapping indirect effects method using the PROCESS macro in SPSS.140 Contextual variables were examined in a multiple linear regression model. Moderators were examined using regression models. Each potential moderator variable was considered in turn. To formally assess the statistical significance of any observed effect moderation, a multiple linear regression model was fitted with the inclusion of an interaction term between the child engagement factor score and the moderator variable.102

Each variable was centred prior to its inclusion in the mediator and moderator models; continuous variables were centred at their respective means, while binary variables were recoded as −0.5 and 0.5 (rather than 0 or 1). Centring refers to the subtraction of the overall mean from each observation. Therefore, each variable is ‘zeroed’ at its own mean. Centring the data aids interpretation of mediator and moderator analyses and diminishes the effects of multicollinearity.141 Given the exploratory nature of the mediator and moderator analyses, we did not correct for multiple comparisons.142 All data used a significance level of p < 0.05. All statistical analyses were conducted using IBM SPSS Statistics version 27.

Results

Impact

Primary outcome

A repeated-measures ANOVA with a Greenhouse–Geisser correction determined that mean YGTSS-TTSS statistically significantly differed between baseline and the primary end point [F(1,100) = 39.71, p < 0.001]. There was a reduction in YGTSS-TTSS from baseline (mean 27.92, SD 7.17) to the primary end point (mean 23.87, SD 8.18) in the group that had received the intervention, and this was statistically significant (p < 0.001). Cohen’s d for the pre–post change in YGTSS-TTSS was 0.5, indicating a moderate effect.102

Secondary outcome

Of 101 participants in the active intervention group, 36 (36%) were classified as ‘responders’, with their condition rated as being ‘very much improved’ or ‘much improved’ on the CGI-I. Thirty-seven (37%) were rated as ‘minimally improved’, 18 (18%) were rated as having no change in their condition and 10 (9%) were rated as being minimally worse. No participants were rated as being much or very much worse in their condition since the initiation of treatment.102

Correlations

Spearman’s rank correlations were used to analyse associations between all contextual, mediator and moderator variables and the primary (YGTSS-TTSS change) and secondary (CGI-I) outcomes. Only YGTSS-TTSS at baseline (ρ = −0.26, p < 0.001) was statistically significantly correlated with YGTSS-TTSS change so that higher scores at baseline were associated with a greater decrease in tic severity at the primary end point. Child engagement factor score (ρ = −0.23, p = 0.02) and parental chapter completion (ρ = −0.25, p = 0.01) were statistically significantly negatively correlated with CGI-I, suggesting that CYP with high levels of engagement with the intervention and CYP with parents who completed more chapters showed better overall clinical improvement at the primary end point.

Table 29 shows the intercorrelations between YGTSS-TTSS change score, CGI-I and contextual, mediator and moderator variables.

Contextual factors influencing impact

Following on from the correlational analysis, we explored whether parental engagement was an independent contextual predictor of YGTSS-TTSS change and overall clinical improvement (CGI-I). Multiple linear regressions were conducted with parental chapter completion, IMD and maternal level of education as the independent variables. There was no evidence of multicollinearity, with all tolerances above 50% and all variance inflation factors below 2. The results of the simultaneous regression indicated that collectively the independent variables did not predict YGTSS-TTSS change. However, the independent variables did explain a statistically significant amount of variance on CGI-I, F(3,97) = 3.14, p = 0.029, R² = 0.09. Parental chapter completion [β = −0.10, t(100) = −2.41, p = 0.018] was the only significant independent predictor in the model.102

Mediators

Simple mediation analyses found that the relationship between child’s level of engagement (child engagement factor score) and either tic severity change (YGTSS-TTSS change) or CGI-I was not mediated by depression change (MFQ change), anxiety change (SCAS change), treatment satisfaction or treatment credibility.102

Moderators

A moderator analysis was conducted to assess whether medication use, comorbidity, parental chapter completion, YGTSS-TTSS at baseline, age, deprivation (IMD) or mother’s level of education moderated the relationship between child’s level of engagement (child engagement factor score) and either of the outcome variables YGTSS-TTSS change or CGI-I. No variables were found that moderated the relationship between child engagement factor score and outcome in terms of either YGTSS-TTSS change or CGI-I.102

Framework categories

From the analysis of the qualitative data, three categories were generated relating to participants’ perceptions of the impact of the ORBIT intervention: ‘Mechanisms of impact’, ‘Intervention outcomes’ and ‘ORBIT programme content’ (see Appendix 2, Table 32 for full analytic framework categories and themes).

Mechanisms of impact

This framework category tried to determine how the intervention had worked. Within the theme features of online therapy to support tic reduction, CYP highlighted the need to practise and how the interactive aspects of the intervention had helped with their tics (quote 1, see Appendix 3, Table 33). Others mentioned that the extent to which they were able to visualise how the treatment and ‘tic cycle’ worked had helped them (quote 2). This visualisation also enabled child participants to see how and which tics were increasing or decreasing in severity or frequency.

Other participants felt that the use of the reward system motivated them to complete the practices within ERP (quote 3). Parents explained how during the middle of treatment their children began to disengage with the intervention and thus introduced the reward system, which reignited their motivation levels (quote 4).

Conversely, some participants felt that the nature of online therapy posed certain barriers that might have impeded its impact, with some CYP saying that they would have preferred face-to-face therapy. This was captured by the theme limitations of online therapy (quote 5). This theme also captured the frustration some felt at not receiving an immediate response from their assigned therapist (quote 6). One parent spoke about how their internet connection was substandard and found it quite frustrating whenever the treatment programme could not be accessed (quote 7).102

Parental chapter completion was a contextual factor that was significantly associated with clinical improvement. Qualitative analysis also highlighted the importance of the extent to which CYP received support from their parents in completing the treatment. CYP appreciated being able to complete the therapeutic activities and chapters with their family members. This was captured by the strong theme working together (quote 8). Parents also seemed to appreciate this time spent working together as a team, which they felt may have led to more positive outcomes (quote 9). One parent explained how the added complication of comorbid ADHD meant that she had to arrange a suitable time and place to enable her child to feel sufficiently relaxed to complete the chapters (quote 10). Although the working together theme was largely related to the parent-and-child relationship, some parents did appreciate having therapist support. They particularly seemed to appreciate having an expert on hand if they required their assistance on anything that they were unsure about (quote 11). Finally, one of the clinicians suggested that parental support seemed to be the key factor of this online intervention (quote 12).102

Intervention outcomes

From this category, various themes were gathered that outlined the impact that the intervention had on CYP and parents. The theme improvement in tics showed how participants felt that the intervention had allowed the severity and frequency of their tics to dissipate (quote 13). For one child participant in particular, the intervention did not have an impact on the overall severity and frequency of their tics; however, it did allow them to better understand their premonitory urges (referred to as ‘tic signals’; quote 14).102

From the theme expectations versus reality, some parents thought that the intervention would be more effective than it had been and were somewhat disappointed by the outcome (quote 15). Finally, from the theme improved self-esteem and confidence, some parents also noticed how there was an improvement in their child’s related psychological symptoms because of the intervention (quote 16).102

Online treatment content

In terms of what are known as ‘essential ingredients’, these were captured by the theme useful and enjoyable program resources. This theme elucidated how participants found certain components of the intervention to be the most impactful. Most CYP found the ‘tic stopwatch’ (timing how long they can suppress their tics) and ‘tic ladder’ (hierarchy of tics) to be the most useful to them (quote 17). Many participants were also attracted to the videos and animations, which some found to be engaging and an effective alternative to large quantities of text presenting key information (quote 18). Parents also stated that they liked the videos, as they clarified the ways in which they should deliver the therapeutic techniques to their children (quote 19).102

Finally, under the theme lack of fit between content and child, some older participants found the content and presentation of some of the materials in the intervention to be ‘child-like’ and aimed more towards younger children (quote 20). One of the therapists agreed with this sentiment but felt that due to the large age range involved in ORBIT they had to cater to all participants (quote 21).102

Discussion

In this second phase of the process evaluation, we examined the impact of the intervention in terms of reduction in tic severity and global clinical improvement. Within the active intervention group, tic severity was reduced by four points from baseline to 3-month follow-up with a Cohen’s d effect size of 0.5, and 36% were rated as ‘very much improved’ or ‘much improved’ on the CGI-I. A further 37% showed some improvement. This process evaluation, which aimed to understand mechanisms of impact, found that only tic severity at recruitment was associated with reduction of tic severity post-intervention. Level of usage as captured by the child engagement factor score99 was not associated with improvement in tic severity; however, higher levels of child engagement and higher parental chapter completion were associated with higher levels of overall clinical improvement. Only parental chapter completion was independently associated with CGI-I scores. No mediators or moderators were identified for either reduction in tic severity or clinical improvement. There may have been a lack of power to detect mediators and moderators for the relationship between usage and impact of the intervention due to the high level of overall uptake and the relatively modest impact compared to face-to-face therapy.3

Engagement and parental support

Although child engagement with the intervention led to better overall improvement at the primary end point only on one of our outcome measures, the CGI-I, the findings of this study are broadly consistent with the literature.143 Additionally, the association between parental chapter completion and positive outcomes is also consistent with previous research.130,132,144 The findings from this study suggest that parental involvement worked through encouraging CYP’s engagement, which led to positive outcomes, as CYP who were interviewed in the qualitative component of the current study emphasised that parental support was key to their levels of engagement. As parents had their own chapters to work through, they appeared to gain more knowledge of their child’s tics and were able to understand them better. Furthermore, both CYP’s and parents’ accounts of their experiences frequently cited being able to work together as a team as one of the main drivers of how they conducted their time on ORBIT. It was clear from the interviews that ORBIT managed to facilitate a symbiotic relationship between child and parent, which may have led to its impact.102

In terms of what are known as ‘essential ingredients’, this study appeared to shed some light on what these may constitute. Specific features such as the video demonstrations of therapy, animations, the ability to visualise which tics were increasing or decreasing in severity and frequency on the ‘tic ladder’ and the ‘tic stopwatch’ were all highlighted in interviews as important mechanisms of impact. Indeed, these interactive components were identified as key features of the intervention that seemed to be used most. This is consistent with evidence that interactive elements, including attractive audio-visual material, are amongst the most highly used features of DHIs as they tend to maintain users’ interest.145,146 This would be especially important to younger children whose concentration levels would not be maintained with material that was simply presented in writing, for example. Whilst many of the younger children appreciated the graphics and animations, older children felt they were somewhat ‘child-like’. However, there was no evidence from the quantitative data that child age influenced engagement99 or efficacy.102

Mediators and moderators

The absence of significant findings in the mediator and moderator analyses in this study suggests that treatment efficacy was not significantly affected by the sociodemographic or clinical characteristics of participants. Furthermore, interviews with parents, children and therapists also failed to reveal any perceptions of clinical factors influencing outcomes. Therefore, the findings from this study suggest that ORBIT is appropriate for a wide demographic of CYP with tic disorders, regardless of parental education or deprivation, age, baseline tic severity, tic medication use, anxiety or depression levels or comorbidities. The lack of statistically significant findings is encouraging for clinical practice, suggesting that health-care specialists can confidently recommend this digital intervention to most families.102

There may be some potential explanations as to why no significant mediators or moderators were found. As a process evaluation, this study only included participants in the active intervention arm. Thus, we explored factors moderating the relationship between level of engagement and outcomes rather than the relationship between group membership and outcome. This meant that the sample size was diminished, thus creating low statistical power. It is well established that large sample sizes and substantial power are necessary to be able to detect mediator and moderator effects.147

Conclusions

Overall, the intervention had a positive impact on participants as it reduced the severity of their tics and improved overall clinical condition. This study found that parental engagement was a significant contextual predictor of overall improvement of tics; however, there was no evidence of mediators or moderators of outcomes of the online ERP intervention delivered to CYP with tic disorders. The results of these analyses suggest that engaging parents is a key factor in successful outcomes and that whilst online therapy seems to be an effective alternative to face-to-face therapy for CYP with tic disorders there is no subgroup that is more or less likely to find this treatment beneficial. This is a positive finding from a clinical perspective, as it suggests that ORBIT can be implemented within routine health care for a broad range of CYP with tic disorders. However, more research needs to be carried out in this area with larger sample sizes and with a primary focus on the potential mediators and moderators of impact in order to fully understand the mechanisms through which online therapy has its desired effect.102

Overall discussion

The aims were to conduct a mixed-methods process evaluation of an ERP intervention for CYP with tics delivered online with therapist and parental support. Specific objectives were to assess the extent to which the intervention was implemented as part of a trial and the contextual factors that influenced child participants’ level of engagement, as well as to evaluate the overall impact of the intervention, the mechanisms through which it achieved its impact and the factors influencing its overall efficacy.102 In doing so, this process evaluation aimed to contribute to the wider understanding of how online BTs for CYP can improve outcomes and, more broadly, the factors influencing CYP’s engagement with DHIs and the potential resources needed in order to implement online ERP for CYP within routine clinical practice.

Overall, these findings contribute to our understanding of the way in which DHIs work and what is required to make their impact more successful. As some studies have shown that digital technology can be used to reduce symptoms in CYP with tic disorders, this process evaluation contributes to the wider knowledge of the underpinnings of their mechanisms of action.102 Table 30 provides a summary of the key findings from the process evaluation and their potential application to practice.

Strengths and limitations

This study marks a comprehensive evaluation of a digital intervention delivered to CYP with tic disorders. This understanding is important in order to fully understand the circumstances under which such interventions are likely to be effective, for whom and maximise their efficacy.148 In addition to explaining intervention-specific processes, the findings also contribute to the currently limited knowledge regarding how CYP and their parents engage with digital interventions to manage complex symptoms. Furthermore, it provides evidence of the feasibility and utility of digital interventions amongst a youth population.102

There are several strengths within the design of this process evaluation, which was carried out concurrently with the ORBIT trial. A particular strength was that the methodology was based on a peer-reviewed published protocol,101 which is considered ‘best practice’ within process evaluation research. The sample involved in the semistructured interviews and within the intervention group was comparable in terms of age, gender and baseline YGTSS-TTSS. As stipulated in MRC guidelines,103 a mixed-methods approach was undertaken in the form of an integrative mixed-methods design. Moreover, the qualitative data offered a more in-depth evaluation to participants, therapists and referring clinicians’ perceptions and experiences with ORBIT and were able to capture outcomes that the quantitative data set could not.102 Furthermore, the analysis of intervention usage data provided a detailed and objective insight into the important features and underlying determinants central to the implementation and effectiveness of this intervention. As recommended,103 process evaluation data were analysed prior to the trial findings being known, thus avoiding a biased interpretation of the data.

Another major strength of this study is the use of an objective, comprehensive measure of a child’s level of engagement with the intervention. By using a principal component analysis of dose of intervention received, we were able to capture an innovative measure of engagement, which could be replicated in future designs of such studies. Several strategies were utilised to strengthen the validity of the qualitative analyses. A multidimensional perspective was achieved by considering both CYP’s and parents’ as well as therapists’ and referring clinicians’ perceptions.149 A subset of the transcripts was also double coded and disagreements in coding were discussed, strengthening the validity and reliability of the qualitative data. A further strength of the qualitative data was the use of PPI in guiding the semistructured interview questions.102

Despite these strengths, this study and the subsequent findings should be interpreted with caution due to several limitations. Firstly, it was difficult to recruit participants for interview who had either dropped out of the ORBIT trial early or were not deemed treatment completers, and thus the data may have been skewed towards more positive experiences of the intervention. Limitations relating to the quantitative data set must also be considered. This study was embedded within a trial, which limited the sampling frame. Although the ORBIT trial was one of the largest in online tic disorder research, for the purposes of the mediator and moderator analyses in particular the sample size was somewhat underpowered. Therefore, the lack of statistically significant findings within these analyses may have been due to the lack of power, and so such results should be considered as exploratory. Finally, whilst process evaluations tend not to assess the control arm, doing so might have strengthened the generalisability of the overall findings while giving a more holistic view of ORBIT.102

Implications for research and practice

The findings of this process evaluation have implications for potential research in this area. Primarily, it would be important for any future work to overcome the limitations highlighted above.102 Indeed, a more intensive effort to recruit participants who did not engage satisfactorily with digital interventions or dropped out early would be welcome. This would enable researchers to gain a more holistic understanding of the implementation and mechanisms of action and prevent the findings from being skewed towards more positive experiences.102

Future RCTs of complex interventions should consider the integration of quantitative process data from the outset, which could then provide further insight. More broadly, it would be valuable for future studies of complex interventions for CYP to include process evaluations. These should include objective usage metrics and longitudinal qualitative data. Finally, and most crucially of all, future research should consider how this evidence-based online ERP intervention can be made deliverable in routine NHS care, thus giving more people access to much-desired non-pharmacological treatments.102

As discussed, tic disorders and associated conditions are highly debilitating, having a profound impact on both CYP and their parents. A range of tic-related difficulties with academic work and social and emotional well-being in CYP have been reported. Considering these tic‐related impairments and implications for future life, knowledge of the best treatment options for tic disorders in CYP is clinically important.102 Children and their parents generally prefer BT over medication due to the fewer associated adverse effects;21 however, the most widespread mode of treatment is pharmacotherapy.12

Therefore, the findings from this process evaluation have important implications for practice in a myriad of ways. They show how CYP engage with complex interventions and the importance of parental support and motivation, identify those who benefit the most from such interventions and, more broadly, show how DHIs can be designed and implemented in order to maximise their efficacy.102 Most importantly of all, the findings demonstrate that an online intervention is effective, can be delivered with high fidelity and is highly acceptable to CYP with tic disorders, and that there is no subgroup who benefits the most. This has important implications for how this intervention could be delivered to patients within the UK. An online intervention that could be deployed to large numbers of patients at a relatively low cost is a much-needed and seemingly acceptable means of providing patients with access to an evidence-based treatment. It could provide immediate access to ORBIT for those who otherwise would not have access due to long waiting lists or their geographical location, which could also potentially free up existing resources and services for those requiring more complex treatment and assessment.102 Such cutting of costs and waiting times would represent a two-fold benefit for the NHS and patients alike.

In addition, one of the barriers to reach identified was the associated travel to the baseline assessment. It would therefore be sensible to have an initial remote assessment. As the use of remote medical assessments has increased due to the COVID-19 pandemic, this flexible approach would allow even more people to benefit from this intervention.102 As already discussed, it is also recommended that the term ‘therapist’ should be altered to avoid high expectations of their role. Furthermore, the ‘therapists’ could be employed on a part-time basis and would require very little training, which would cut costs even further. As the findings indicate, ‘therapists’ should engage the parents as much as possible to achieve successful outcomes.

Conclusions

This mixed-methods study is a comprehensive assessment of the processes underlying a complex online intervention delivered to CYP with tic disorders using MRC guidelines as a framework.150 Overall, ORBIT is an effective and acceptable means of delivering an evidence-based ERP treatment to CYP with tic disorders and supporting them in overcoming barriers to accessing this therapy. Whilst some CYP may require additional support from their parents to enhance their level of engagement with the intervention, there is substantial evidence that this online intervention is a promising means through which these debilitating and complex symptoms can be addressed.102