Feasibility and pilot randomised controlled trial

Overview

The final phase (phase 3) of the study consisted of a pilot RCT of the modified Video-feedback Intervention to promote Positive Parenting and Sensitive Discipline (VIPP-SD) programme [i.e. the Video-feedback Intervention to promote Positive Parenting and Sensitive Discipline, Foster Care (VIPP-FC)]. A new group of children was randomised to receive the VIPP-FC plus care as usual (CAU), versus CAU only. The VIPP-FC was delivered in-home by trained CAMHS practitioners or other appropriately qualified VIPP-trained interveners. Baseline and end-of-treatment measures included validated assessments of RAD and co-occurring conditions, security of attachment (including disorganised attachment), sensitivity of parenting, child adjustment, carer well-being, and the CA-SUS measure of service use. In addition to these outcome measurements, we also conducted qualitative interviews regarding carers’ experiences of the programme, the results of which are presented in Chapter 7. The specific aims of the pilot RCT were to:

1. test the feasibility of identifying sufficient numbers of appropriate cases through social care using appropriate screening instruments across several local authorities and locales (rural and urban)
2. test the feasibility of recruiting and consenting foster families with young children (including those with RAD symptoms) to a RCT (numbers contacted and consented)
3. conduct an initial evaluation of the feasibility of randomising to the VIPP-FC or CAU (numbers refusing randomisation)
4. document study throughput – uptake, numbers of sessions attended and numbers completing treatment, rate and stage of dropout, and acceptability of the treatment
5. establish the feasibility and acceptability of baseline and outcome assessments and level of data completeness
6. modify and test a version of the CA-SUS measure of service use to ensure suitability for pre-school/primary school aged fostered children
7. identify the most appropriate primary outcome
8. obtain initial estimates of variance of key outcome measures
9. explore the appropriateness of exclusion criteria, particularly concerning co-occurring conditions
10. establish the optimal and most acceptable screening tools for RAD, as recently redefined in the DSM-5.

Methods

Setting

The study was a multisite RCT involving eight NHS trusts and nine linked local authorities, and took place across four large and distinctive regions in England: Greater London (North, South, East and West), Hertfordshire, Peterborough, and the Yorkshire and Humber region. As noted previously, a substantial number of local authorities was necessary to ensure adequate recruitment and to capture important variability in organisational contexts that may be important to understand when preparing a future larger-scale clinical trial.

Recruitment

Participants were identified initially through a large community screening process in the nine local authorities, with a view to identifying ‘at source’ (rather than through referrals or CAMHS; however, these were also accepted as a recruitment route) children experiencing high levels of behavioural signs of RAD on the screening questionnaires. Positively screened cases were assessed using the DAI to establish, using a validated and rigorous investigator-led diagnostic schedule, whether or not children showed evidence of RAD. Serious challenges to recruitment mid-way through the study led to a decision, supported by the TSC and funder, to remove the RAD eligibility criterion so that there was greater opportunity to run a reasonable number of cases through the research and intervention protocol. The decision to remove the RAD eligibility criterion also reflected a view shared by the research team and TSC that children in foster care are at substantially heightened risk of not only RAD, but a range of attachment difficulties and behavioural and emotional problems, and may therefore benefit, as a group, from the VIPP-FC. The outcome measures continued to track impact on RAD symptoms as well as these other clinical domains. The number of participants and their characteristics are described in Results, as they represent important feasibility-related outcomes.

Changes to protocol between the case series and the pilot randomised controlled trial

For the most part, the research protocol proceeded smoothly during the case series, so no changes were made to the assessment battery. The primary challenges were the somewhat lower than hoped for return rates of the screening questionnaires and the modest numbers of cases meeting criteria for RAD on the DAI. A key challenge was the lack of capacity within the local authority to undertake follow-up telephone calls to foster carers to encourage return of the screening instruments. In response to this, and in discussion with the TSC, we revised the protocol to allow members of the social work team to obtain verbal consent from foster carers for the research team to contact them directly by telephone. In consultation with our foster carer advisory group, we also substantially revised the study leaflets and information sheets to make them clearer and more appealing. We also introduced an alternative online screening/agreement to be contacted form following recommendations from our foster carer study member and advisory group. The study team received e-mail notifications from the study database when a participant gave consent and completed questionnaires online. Both the paper and online consent forms required the carer to provide their contact details (including the child’s initials) so that the research team could contact them to thank them and invite them to the interview stage. As these changes generally did not affect the overall research protocol, the methods are presented succinctly here; see Chapter 3 for detailed information about the research instruments.

Consent

No changes were made to the consent process from that used in the case series. During screening, active consent was required from the foster carer only. Passive consent was obtained from parents. After screening, if the child met the eligibility criteria to take part in the trial, full written consent was obtained from all those holding parental responsibility and from the foster carer, as well as assent from the child.

Measures

No changes were made to the assessment battery between the case series and pilot trial. The assessments consisted of the following:

• screening – the DAWBA Relationships with Adults scale, the ASA and the DAI
• baseline and post treatment – the Strange Situation Procedure, the DAWBA (full instrument), the CBCL, the Parenting Stress Index and the Parental Self-efficacy Scale, the CA-SUS and a carer–child interaction task (for coding of parental sensitivity with the NICHD scales).

Process data and qualitative outcomes

Process data were collected during the pilot RCT to document the uptake, numbers completing treatment, rate and stage of dropout, and acceptability of the treatment to participating families. Records were kept of the number of families approached that refused to take part in the study; data were also obtained on the range of care pathways observed (for example distribution of care orders and placement types, e.g. family and friends care, long-term foster care), the numbers moving placements before or after treatment commenced, and the destinations and timing of any movements out of placement. Records were also kept of essential information on a child’s previous history, such as number and type of previous placements. These data were obtained from a child’s social worker and/or foster carer. Reasons for refusing to take part in the study and for dropping out were also recorded.

The acceptability data gathered through the conduct of qualitative in-depth interviews with all consenting families that received the intervention as part of the pilot RCT.

Randomisation

Participants were individually randomised in a 1 : 1 ratio to one of the two treatment arms: the VIPP-FC plus CAU or CAU only. Randomisation with minimisation was employed, with minimisation factors age (≤ 4 vs. > 4 years), sex and site. Minimisation included a random factor, as implemented in the minimisation program SiMin.70 Randomisation was requested by the research assistant using a standardised form following completion of all necessary screening and consent procedures, and was performed by an unblinded member of the trial co-ordination team. The allocation list was held in the Trial Master File. A member of the trial co-ordination team contacted the child’s social worker and VIPP-FC intervener (the latter for those allocated to VIPP-FC). The foster carer and the individual(s) giving consent on behalf of the child (the parent and/or local authority social worker with parental responsibility, as appropriate, see Chapter 5, Consent) were notified of the outcome by letter/e-mail by the unblinded member of the trial co-ordination team. The foster carer was also contacted by telephone to ensure a timely initiation of the intervention.

Blinding

All outcome assessments were performed by research assistants blind to trial arm allocation. All data handling and communication regarding unblinded information was undertaken by the Trial Co-ordinating Team. Prior to all research visits, the trial co-ordinator briefed the foster carer(s) about the importance of assessor blindness and any cases of inadvertent unblinding were recorded by the research assistant(s). By definition, foster families were not blind to the treatment arm to which they had been randomised.

Intervention

The VIPP-FC

The VIPP-FC is described in detail in Chapter 3 and the clinical protocol was not changed after the first phase of manual development and modification, as described in Chapter 2.

Care as usual

The VIPP-FC was compared with CAU, as there is no pre-existing integrated care pathway for children in foster care. Usual care varies widely between localities and good-quality data were not recorded that would allow precise specification of CAU prior to undertaking the study. Systematically documenting CAU was therefore a key objective of this study. Initial consultations with CAMHS teams in our study sites identified several potential treatments that may be offered within CAU: (1) Theraplay, (2) one to three sessions of foster carer consultation, (3) network consultation, (4) The Incredible Years^® (The Incredible Years, Inc., Seattle, WA, USA), (5) foster carer support, (6) 6–12 sessions of psychotherapy, (7) behaviour support and (8) foster parent groups.

As assessments of attachment disorders are rarely undertaken, it is not possible to determine the extent to which the type and intensity of services offered to this particular group are different from those offered to other young children in foster care referred to CAMHS. We used the CA-SUS to systematically describe and quantify the services received by children and carers in the comparator arm of the pilot RCT. The CA-SUS asks participants to confirm whether or not they have attended any health services, mental health services, support services or child-care services, and if so, how many times they have attended these services in a defined period of time.

Sample size considerations

The planned sample size of 40 participants (20 per arm) was based on common recommendations for feasibility studies, in which the aim is to test practical implementation parameters, rather than to test treatment effects. Using data obtained from the Department for Education from 201471 (which we have corroborated against a selection of figures obtained in 201671 from our participating local authorities), we expected the full population of children in foster care in our target age groups across our sites to be in the region of 700. Previous screening work with this population led us to aim for 70–80% of foster carers to return the initial screening questionnaires; on that basis, and bearing in mind cost implications, we aimed for a sample size for the screening stage of 500 participants. To identify cases likely to present RAD, we planned to choose cases above the highest 66th percentile for symptoms of RAD, leading to a sample size of 150 who would be invited to take part in the more in-depth assessment of RAD symptoms using the DAI. Based on the previous RCT of the Fostering Changes programme,72 we expected two-thirds of those to agree to take part and complete the assessment. From this pool of 100 children, we expected to have scope to identify the target 40 cases who would be eligible for treatment, as long as the prevalence of RAD in the high-scoring screened group was ≥ 50% (allowing for a further 10% dropout at this stage).

It is important to note that, at the planning stage of this study, two key pieces of information were subject to considerable uncertainty. First, although clinical screening work within routine services locally were able to obtain 70–80% return rates, these were not obtained in the context of a research trial and had not been tested in all sites. In the case series, we obtained a screening questionnaire return rate of 60% (based on 20 cases), which, although lower, was not drastically out of line with expectations. However, this was from a single site only. Second, we lacked reliable information concerning the prevalence of RAD in UK foster care; thus, the 50% prevalence within the population of children exceeding the symptom threshold was subject to considerable uncertainty. Consequently, we planned for an interim analysis after 15 cases had been assessed to establish whether or not the proportion of RAD cases was below this expected proportion. The lower bound of the 95% confidence interval (CI) around a proportion of 0.50 is 0.22 with a sample size of 15. Thus, identification of fewer than three cases out of 15 would suggest that the prevalence was lower than 0.50. In this event, we planned to widen the eligibility criteria to include children not meeting full RAD diagnostic criteria, subject to approval from the TSC.

A sample size of 40 participants provides a reasonable balance between precision and cost. Assuming that the proportion of participants successfully completing the key outcome assessment is 75%, this yields a 95% CI of 59% to 87%. The consent/uptake rates could be estimated at two stages: for consent to take part in the eligibility assessment, and, following eligibility, the proportion entering treatment/being randomised. The former rates would be estimated with higher precision as they are based on larger samples. Assuming 67% (100/150) of participants take part in the eligibility assessment, this yields a 95% CI of 59% to 74%. Similarly, assuming that 80% of cases identified with RAD (40/50 participants) go on to be randomised, we would obtain CIs around that proportion of 66% to 90%.

Data analysis

Point estimates and 95% CIs were calculated for the key feasibility parameters based on proportions (proportions of cases identified, consented, randomised, completing treatment and completing outcome measures). Approximate variance estimates were obtained for the main outcome measures using the upper 80th percentile of the relevant CI. Specifically, we tested the following feasibility parameters:

1. Test the feasibility of identifying sufficient numbers of appropriate cases using screening instruments.
   • Target n for screening tests returned, n = 500. Test a.1 = proportion and 95% CI of those returning screening instruments from total sent out.
   • Target n of diagnostic assessments, n = 100. Test a.2 = proportion and 95% CI of positively screened cases (above upper 66th percentile) agreeing to full assessment.
   • Target n for positive diagnosis from upper 66th percentile of screened cases, n = 50. Test a.3 = proportion and 95% CI of cases for whom diagnosis is confirmed.
   • Test a.4 – initial test of false negative rate = proportion (of 25 cases) and 95% CI of detected diagnoses in random sample of cases not meeting screening threshold.
   • Test a.5 – initial non-parametric receiver operating characteristic (ROC) curve analysis (target n = 125), based on assumption that the DAI diagnosis is gold standard.
2. Test the feasibility of recruiting/consenting foster families with children with RAD to a RCT.
   • Target n of consented cases, n = 40. Test b.1 = proportion and 95% CI of cases consenting to enter pilot trial from those meeting entry criteria.
3. Investigate feasibility of randomising to the VIPP-FC or CAU.
   • Test c.1 – proportion and 95% CI of cases accepting randomisation.
4. Document study throughput.
   • Test d.1 – proportion and 95% CI of cases who are randomised completing treatment (definition of completing = 80% of offered sessions).
   • Test d.2 – proportion of cases attending each session (six proportions and 95% CIs).
   • Acceptability of treatment – qualitative methods for client experience of treatment interviews (see Chapter 7).
5. Establish the feasibility and acceptability of baseline and outcome assessments.
   • Test e.1 – proportion and 95% CI of cases with complete data. See also qualitative methods for client feedback regarding measures.
6. Identify the most appropriate primary outcome. Key assessment was participant feedback from qualitative interviews. However, preliminary tests from tests a.3, a.4 and a.5, were also considered.
7. Obtain initial estimates of variance of key outcome measures for future power analyses. Test – standard deviation (SD) of RAD symptom counts and 95% CI.
8. Establish the optimal and most acceptable screening tools for RAD, as recently redefined in the DSM-5. Tested with qualitative interviews and tests a.4 and a.5.

Results

Recruitment sites and flow

The pilot RCT commenced in February 2018; recruitment ended in September 2019. All follow-up assessments were completed by the end of January 2020. Recruitment was attempted from a total of nine local authority children’s services and eight linked NHS trusts. The first site to open to recruitment was the site that took part in the case series. The second site opened in January 2018 and the remaining sites opened throughout 2018 and 2019, with the last site opening in April 2019. The first participant was recruited to the screening stage of the study in May 2018 and the first participant was recruited to the second-stage RCT in July 2018. Owing to serious challenges with recruitment and some delays in site initiations, recruitment was extended from the originally planned period of 12 months to 17 months.

Of the nine sites, we were able to recruit successfully from seven. Even with considerable effort, we were not able to progress with two London local authorities, despite their initial agreement and support. Throughout the recruitment period, we implemented several adjustments to the protocol to aid recruitment (see Chapter 1, Ethics amendments and protocol modifications and Appendix 1). The key changes that were implemented during the RCT period included the inclusion of additional sites and the dropping of the RAD-related eligibility criterion in March 2019. We illustrate the timeline and key events against progress on recruitment in Figure 2.

FIGURE 2 Recruitment rate over time. Reproduced with permission from Oliveira et al.73 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure. (Figure not reproduced here; see the original article.)

Recruitment flow increased substantially in the last period of the study, as can be seen in Figure 2. It is not possible to precisely determine the reasons for the improved recruitment rates, but the improved rates largely coincided with the removal of the RAD criterion, the addition of three new and engaged sites (notably, all of which were outside London), and our success in one site in obtaining an honorary contract for a member of the research team to work within the local authority children’s services to directly support recruitment. The rate of recruitment in this final phase was four cases per month. Most of the other sites were unable to maintain their recruitment efforts in the final stages of the study. Therefore, this final phase primarily reflected recruitment from four sites. By the end of the recruitment period, we were able to recruit and randomise 30 cases to the trial, which was 10 cases below the original target.

Screening response rates and recruitment

The screening process involved close collaboration with partner local authorities, which sent out information sheets, initial consent forms and screening measures to the foster carers in the study localities. In addition, local authorities sent out a letter to all parents explaining the study and the opting-out process, and the letter included the contact details of the research team. They all communicated to their staff members within the children’s social care teams to raise awareness of the study and draw attention to the support required from children’s social workers. For this screening phase, we aimed to invite all foster carers or special guardians registered with the local authority caring for children aged ≤ 6 years. In practice, different local authorities applied their own exclusion criteria at this point (e.g. excluding highly complex cases undergoing assessments for court or cases with restricted files), as we note in Discussion.

The overall response and retention rates are shown as a CONSORT flow diagram in Figure 3. As is evident from the figure, the overall response rate to the screening questionnaire was 29% (95% CI 24% to 34%), which is substantially below the original target of 80% and below the rate achieved in the case series (60%). Specifically, of the 336 foster carers invited to complete the screening, 96 returned completed questionnaires. The overall number of mailed-out questionnaires was also smaller than the original target (n = 500), largely due to significant numbers of children registered with the local authorities being considered inappropriate or ineligible for the study (e.g. very short placements, placed out of borough). It is important to note that the return rate varied substantially per site, ranging from 3% to 63% (median 30%).

FIGURE 3 The CONSORT flow diagram for the VIPP-FC pilot trial. LA, local authority. Reproduced with permission from Oliveira et al.73 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure. (Figure not reproduced here; see the original article.)

Approximate prevalence of reactive attachment disorder

The results of the DAI indicated an approximate prevalence of significant RAD symptoms in this sample of 33% (22/67 children assessed, 95% CI 22% to 44%). This proportion is based on cases scoring ≥ 3 points on the total RAD score, which indicates clinically significant difficulties (Charles Zeanah, Tulane University, 2015, personal communication). Furthermore, 41 out of the 67 foster carers’ DAIs received a score of at least 1 point (scores can range from 0 to 2 points for each item) on at least one RAD item, and of these participants, 33 received a total RAD score of ≥ 2 points (51%, 95% CI 37% to 61%). These can be seen as cases who show some difficulties in the areas of development tapped by the DAI-RAD items.

Participant characteristics

As demographic data were collected both at initial screening and at entry to the RCT, it is possible to compare the composition of the trial sample with that of the larger group of screened cases for indications of sampling bias. Table 1 shows the key demographic data for the larger and the smaller subsamples; as can be seen in the table, children and carers that took part in the trial (n = 30) did not appear to be different from the overall group of children who took part in the screening, except for legal status: a larger proportion of children on a full care orders were in the sample taking part in the trial. This is likely to be related to the fact that these children tend to remain in their current placement for longer and to the simpler consent process for these children, as their parents were not required to provide active written consent.

Timing of randomisation, baseline and follow-up assessments

The original study design aimed to allocate cases quickly after baseline assessments had been completed and ideally within 48 hours. The period between baseline and follow-up was expected to be approximately 4 months, which was the expected duration of the intervention. However, we expected this to be potentially quite variable in practice and so allowed the outcome assessments to be undertaken after treatment had ended for all cases in the treatment arm. To balance duration of follow-up across the arms of the trial and reduce the risk of unblinding, we trialled a procedure in which the follow-up period in the CAU arm was yoked to a case in the treatment arm. Matching was undertaken with a relatively wide window (up to 1 month) to not excessively constrain scheduling of follow-ups. The resulting timing of follow-ups is presented in Table 2.

As Table 2 indicates, in general, the time period between baseline and randomisation was comparatively short, but some delays occurred in a number of cases, with one very lengthy delay, which was due to a lack of availability of a VIPP-FC intervener in the relevant site. On average, the actual period between baseline and follow-up was 6.4 months. The duration of the follow-up period was relatively balanced between the two arms of the trial, although the small number of cases with long delays shifted the mean time period in the VIPP-FC arm to more than that of the CAU arm, despite the pairwise matching.

Evaluation of feasibility parameters

In this section, we summarise the data on the key feasibility questions we set out to address in the pilot trial, as detailed in Methods. Each question is presented in the order described in that section.

A. Feasibility of identifying sufficient numbers of appropriate cases using screening instruments

1. The proportion of screening questionnaires returned from the total sent out was 0.29 (95% CI 0.24 to 0.34).
2. The proportion of positively screened cases (above the upper 66th percentile) agreeing to full assessment on the DAI was 0.73 (95% CI 0.60 to 0.86). Scoring high on the screening questionnaires did not seem to make a significant difference to whether or not carers accepted the DAI. Specifically, the proportion of cases scoring below the 66th percentile on the screening questionnaires that agreed to participate in the DAI was 68%.
3. The proportion of cases scoring above the 66th percentile on either screening questionnaire that scored ≥ 3 points on the DAI was 0.86 (95% CI 0.71 to 1.0). The proportion of cases scoring ≥ 2 points on the DAI (the less stringent threshold) that were above the 66th percentile on either questionnaire was 0.76 (95% CI 0.61 to 0.91). We conducted a more fine-grained analysis of the detection rates for each questionnaire. Of the cases confirmed as presenting a RAD score on the DAI of ≥ 3 points, 77% were detected by the DAWBA alone and 68% were detected by the ASA alone, with each case again using the 66th percentile threshold.
4. The false-negative rate, that is the proportion of cases with a DAI RAD score of ≥ 3 points but not meeting the screening threshold on either questionnaire, was 0.09 (95% CI 0.00 to 0.21). If we use the less stringent DAI RAD threshold, the false-negative rate was 0.24 (95% CI 0.09 to 0.39). Considering each questionnaire separately, the false-negative rate (based on the 66th percentile threshold) was 0.23 for the DAWBA and 0.32 for the ASA. Using the less stringent DAI criterion, the corresponding rates were 0.36 and 0.45.
5. Initial non-parametric ROC curve analysis was undertaken based on the assumption that DAI diagnosis is gold standard. We used non-parametric ROC curve analysis to investigate the overall performance of the two screening questionnaires against the DAI. Considering the DAWBA first, non-parametric ROC analysis estimated an area under the curve (AUC) of 0.84 (95% CI 0.74 to 0.92). The ROC curve is shown in Figure 4a. In general, the DAWBA showed good sensitivity, with a raw score threshold of ≥ 2 points, yielding an estimated sensitivity of 0.96 and specificity of 0.48. At this cut-off point, the positive predictive value was 49% (95% CI 35% to 63%), the negative predictive value was 96% (95% CI 77% to 99%), the positive likelihood ratio was 1.74 (95% CI 1.33 to 2.27) and the negative likelihood ratio was 0.09 (95% CI 0.01 to 0.60). A threshold of ≥ 3 points (which is the 66th percentile in our sample) yielded a sensitivity of 0.77 and a specificity of 0.80. With a cut-off point of 3 points, the positive predictive value was 63% (95% CI 44% to 79%), the negative predictive value was 85% (95% CI 71% to 94%), the positive likelihood ratio was 3.01 (95% CI 1.77 to 5.13) and the negative likelihood ratio was 0.31 (95% CI 0.15 to 0.64). Depending on the objectives and, therefore, the preferred balance of false negatives and false positives, a threshold score of 2 or 3 points would seem appropriate using this instrument. The ASA scored similarly well, with an estimated AUC of 0.83 (95% CI 0.71 to 0.91). The ROC curve is shown in Figure 4b. Inspection of the graph suggests that after (1 – specificity) rises to around 0.26–0.30, improvements in performance level off. A threshold score on the ASA of ≥ 24 points yields an estimated sensitivity of 0.74 and specificity of 0.78. For this cut-off point, the positive predictive value was 57% (95% CI 39% to 74%), the negative predictive value was 84% (95% CI 68% to 94%), the positive likelihood ratio was 2.41 (95% CI 1.51 to 3.85) and the negative likelihood ratio was 0.34 (95% CI 0.16 to 0.70). A lower threshold of ≥ 21 points yields sensitivity of 0.91 and specificity of 0.66. A cut-off point of 21 points yielded a positive predictive value of 55% (95% CI 39% to 70%), a negative predictive value of 93% (95% CI 77% to 99%), a positive likelihood ratio of 2.17 (95% CI 1.53 to 3.08) and a negative likelihood ratio of 0.13 (95% CI 0.03 to 0.52). In our sample, a score of 21 points reflects the 51st percentile, which is considerably lower than our original screening cut-off point. Overall, there is indication in these data that the ASA may be marginally superior to the DAWBA for identifying cases, possibly reflecting its greater variability. Across all cases in the study, the DAWBA yielded 10 different recorded scores (reflecting the number of items), whereas the ASA showed a considerably wider range of scores (26 different values, 12–47 points).
6. Combining and standardising both questionnaires yielded an estimate of the AUC of 0.87 (95% CI 0.78 to 0.94). The ROC curve is shown in Figure 4c. An averaged z-score on this composite measure of –0.036 (close to the sample mean) yielded a sensitivity of 0.92 and a specificity of 0.78. For this cut-off point, the positive predictive value was 66% (95% CI 47% to 81%), the negative predictive value was 88% (95% CI 74% to 96%), the positive likelihood ratio was 3.45 (95% CI 1.99 to 5.98) and the negative likelihood ratio was 0.25 (95% CI 0.11 to 0.56). A z-score of –0.26 gave a sensitivity of 0.96, while retaining a comparatively good specificity of 0.65. In this case, the positive predictive value was 60% (95% CI 43% to 74%), the negative predictive value was 97% (95% CI 83% to 99%), the positive likelihood ratio was 2.66 (95% CI 1.80 to 3.92) and the negative likelihood ratio was 0.06 (95% CI 0.01 to 0.42). These results suggest that combining the DAWBA with the ASA is preferable if time and considerations of burden allow for this. Consistent with that inference, a logistic regression, based on z-standardised scores, showed that both the DAWBA and the ASA RAD scales independently contributed to the prediction of RAD diagnosis based on the DAI (DAWBA odds ratio 5.0; p = 0.003; ASA odds ratio 3.6; p = 0.004).

FIGURE 4 The ROC curves for screening questionnaires against the DAI, with 95% CIs. (a) DAWBA; (b) ASA; and (c) DAWBA and ASA combined. (Figure not reproduced here; see the original article.)

B. Feasibility of recruiting/consenting foster families with children aged ≤ 6 years presenting with reactive attachment disorder symptoms

The proportion and 95% CI of cases consenting to enter the pilot trial from those meeting entry criteria are as follows. For the cases who completed screening (including the DAI assessment) prior to the dropping of the RAD eligibility criterion, 56% went on to take part in the trial (23/41, 95% CI 41% to 71%). Of these, 11 (27%) declined to participate in the trial stage of the project and the remainder could not participate for other practical reasons. If we ignore the RAD eligibility criterion, then the proportion of the whole sample is 45% (30/67, 95% CI 33% to 57%). As a percentage of the total cases invited to take part (prior to screening), the conversion rate was 9% (95% CI 6% to 12%).

C. Feasibility of randomising to the VIPP-FC or care as usual

No cases refused randomisation.

D. Document intervention throughput and acceptability

1. The proportion and 95% CI of cases who were randomised and completed treatment (definition of completion = 80% of offered sessions). Four cases who had been randomised to the VIPP-FC did not proceed to receive it. Two failed to receive it because of a lack of clinical capacity in the study site and two because of social worker concerns about competing interventions that had begun since initial contact. All other cases completed 100% of the planned sessions.
2. Acceptability of treatment: this is addressed in Chapter 7 using qualitative interviews with recipients of the VIPP-FC.

E. Establish the feasibility and acceptability of baseline and outcome assessments

The proportion of participants who completed post-treatment outcome assessments was 0.93 (95% CI 0.84 to 1.0); missing outcome assessments post treatment were due to two families dropping out from the trial after randomisation (these were two children who had the same social worker who decided to withdraw these two families from the study). In two other cases, the children moved to a new placement; thus, only some of the outcome assessments were possible; taking this into account lowers the completion of outcome assessments to 0.87 (95% CI 0.75 to 0.99]). Finally, it is also important to consider the proportion of cases with complete data, defined as having completed all of the assessments; Table 3 details the number of missing data in the baseline and follow-up assessments. In general, there were few missing data. However, some difficulties were encountered with the post-treatment DAWBA (when we use the entire questionnaire), which was completed online. Several participants did not manage to complete this measure, owing to its length and some technical difficulties experienced by several participants in using the website.

F. Identify the most appropriate primary outcome

The primary basis for deciding on the most appropriate outcome measure was consideration of acceptability. The feedback (reported in detail in Chapter 7) generally supported the use of the DAI, as foster carers tended to prefer in-person assessments and found the interview format helpful. Although the sample size was too small to complete formal estimates of the sensitivity of these measures for detecting change, it was notable that the DAWBA (RAD scale) and the DAI showed an overall trend for reductions from pre treatment to post treatment (across both arms of the trial, including all participants with follow-up data; DAWBA, t = 2.35; p = 0.03; DAI, t = 1.95; p = 0.06), whereas the ASA did not (t = –0.07; p = 0.95). On balance, we judge that the DAI should be the preferred tool for assessing RAD symptoms for a trial of this nature.

G. Obtain initial estimates of the variance of key outcome measures

Table 4 summarises the pre-treatment and post-treatment descriptive statistics for the main outcome variables. To obtain an approximate estimate of the SD of the RAD symptom measures for future trial planning, we relied on the screening data, for which the sample size was largest. For the DAWBA-RAD subscale, the variance was 5.0 points, with an upper 80% confidence limit of 6.07 points (based on 101 screening cases). For the ASA-RAD subscale, the variance was 52.3 points, with an upper 80% confidence limit of 64.8 points. Finally, for the DAI-RAD subscale, the variance was 4.16 points (based on 73 interviews), with an upper 80% confidence limit of 5.2 points. We do not report tests of group difference, as these would not be meaningful given the very small sample size.

H. Establish the optimal and most acceptable screening tools for reactive attachment disorder as defined in the DSM-5

The analyses of screening test performance indicated that both the DAWBA-RAD and the ASA-RAD scales were adequate as potential screening instruments. Although feedback during the case series did not indicate difficulties with either of these measures, in the pilot trial some participants reported difficulties with the response format of the ASA and this also appeared to lead to a few missing data. The DAWBA-RAD subscale is also considerably shorter. From that point of view, we would recommend the DAWBA as a screening instrument. However, we would also note that we lost considerable numbers of DAWBA data when it was collected online at the post-treatment phase; therefore, we would not recommend this mode of data collection for this population. It is also worth noting that feedback from foster carers indicated that they preferred in-person interviews to questionnaires in general, and, in that regard, the DAI had a high level of acceptability. Thus, under some scenarios (e.g. when screening can be undertaken at a face-to-face meeting), the DAI may be preferable, as not only is it a rigorous and in-depth assessment procedure, but it is also preferred by foster carers and can help to build rapport and engagement.

Adverse and significant events

No adverse events were recorded during the course of the trial, but substantial numbers of significant events were noted. The significant events recording was designed to capture pertinent changes in a child’s health and circumstances tailored for children in foster care. At baseline, foster carers were asked whether or not any significant events had occurred to the child in terms of their health, their schooling (if applicable), their care order or contact arrangements (Table 5 shows all of the categories). In general, the most common significant events were changes in foster placement, contact arrangements with parents, schooling and care order. Table 5 shows the number of individual events recorded for each category for the two arms of the trial.

We did not plan any statistical analysis of the significant events data. Nevertheless, we note that there was indication of possible differences in the rates of significant events between the two arms of the trial at follow-up, which was not apparent at baseline. The mean number of events at baseline was 0.73 for both groups (VIPP-FC, SD 1.02; CAU, SD 0.96). At follow-up, the mean number of events was 0.61 (SD 1.12) for the VIPP-FC group and 1.5 (SD 1.25) for the CAU group. The mean difference at follow-up adjusted for baseline was 0.95 (95% CI 0.04 to 1.9). The apparent difference in rates largely concerns significant events linked to changes in the placement and care order, with fewer of these events reported for the children receiving the VIPP-FC. It is not possible to determine in all cases whether or not these were planned changes, or, indeed, whether they are positive or negative outcomes. Nevertheless, given the negative impact of placement instability, the smaller number of placement-related changes in the VIPP-FC group is quite notable, if very tentative, given the small number of cases involved.

Description of care as usual

One aim of this feasibility study was to find out which services are typically used by the target population. Given that difficulties in the realm of attachment problems in this younger cohort of looked-after children are often not assessed or treated, we anticipated that most children would not be receiving targeted psychological therapeutic interventions. On the other hand, emotional–behavioural difficulties often result in children and their families being offered other support services; therefore, it was important to take a broad perspective when asking about service usage to capture any kind of health, educational and community service (see Chapter 6 for a detailed analysis of all these types of services). In addition, given the multilevel difficulties often associated with being looked after, as well as the complex system involved in caring for these children, we expected that they would be involved with a large number of services and professionals.

The findings were in line with these expectations. We report details of the services that the whole group of 30 children were reported to use during the 4 weeks prior to being interviewed. Carers reported that children were seen by a general practitioner (GP) (n = 3 at surgery and n = 1 by telephone consultation); a nurse at the surgery (n = 2); a health visitor, district nurse or school nurse (n = 11); and a family partnership nurse (n = 1). A small number of children had been seen by a care co-ordinator or case manager (n = 2), and the same number had been seen by a family support worker (n = 2). The majority of children had seen their social worker in the previous 4 weeks (n = 22). Other related services that families had used were a foster support worker (n = 1) and an adoption social worker (n = 1). Seven children had received speech and language therapy in the previous 4 weeks. One child had been seen by a community paediatrician for their health and care plan. In terms of psychological services accessed in the previous 4 weeks, a few children had seen an art/drama/music/occupational therapist (n = 3), and others had seen a CAMHS worker (n = 3), a counsellor (n = 2) or a clinical psychologist (n = 2).

New service types that children received between baseline and follow-up were (1) a psychiatrist, (2) a community psychiatric nurse, (3) an educational psychologist and (4) a play therapist.

Children aged ≤ 6 years in foster or kinship care clearly receive a diverse set of services related to their development and well-being. The services that are offered can include universal services related to being in care (e.g. social worker visits), services for children identified as requiring additional support (e.g. carer consultation or one-to-one support at school), or services offered for reasons other than being identified as requiring additional support; our aim was not to determine whether or not the services related specifically to RAD, but to describe which services participants were using and how frequently. This will be important information to consider when planning a larger trial, as it will allow careful consideration of the comparator arm.

Intervention delivery process

Prior to conducting baseline assessments, a careful process of capacity-checking had to take place, considering participants’ and interveners’ locations and availability, so that, as far as possible, if a case was randomised to the VIPP-FC an intervener would be available to take on the case. As we had only one or two interveners in any one locality, small runs of cases randomised to the VIPP-FC or changes in staff availability (e.g. when staff move jobs) could lead to an inability to allocate or to lengthy delays. In the following sections, we summarise observations of the process of intervener training, the allocation process and the treatment delivery itself.

Training of interveners

We trained a group of practitioners from NHS CAMHS teams, clinical study officers, psychologists and social workers, as well as other appropriately qualified practitioners, in the delivery of the VIPP-FC. Each intervener was trained by an accredited VIPP trainer and was required to undertake supervised practice before becoming a study therapist. Accredited supervision face to face, by telephone and by e-mail was provided to all practitioners during the pilot case series and feasibility trial.

The VIPP-FC interveners

We engaged a total of 23 interveners. Of these, we provided the full training to nine professionals, that is we offered the VIPP-SD training (4 days plus practice families) and a VIPP-FC adaptation half-day training. These practitioners were from across CAMHS and local authorities. In addition, 14 practitioners who were already trained VIPP-SD interveners were then trained in the VIPP-FC adaptation for this study (half a day’s training). Out of our pool of 23 interveners, only 12 delivered a case (this includes the case series; only two interveners saw more than one case). Of the 11 who did not deliver a case, this was for the following reasons:

• Two left CAMHS before being allocated a case.
• One left CAMHS during a case.
• Two left the study because of heavy workload.
• Six could not be allocated cases that matched their locations/availability.

Intervention fidelity

We supported fidelity to the VIPP-FC model through supervision of three visits per case or all visits per case when the intervener was training. All interveners completed logbooks for each visit, outlining how closely they followed the manual, if they completed all of the filming tasks, and if they included key messages in their feedback to the carer, which was required for each different visit. They were also required to audio-record at least one of their feedback sessions per case.

Intervention delivery

Of the 30 randomised cases, 15 (50%) were randomised to receive the VIPP-FC. Of these, 11 received the VIPP-FC, but four did not. There were different reasons why each of these four families did not receive the intervention. In one case, the child left the placement straight after randomisation. Although the team took great care to invite to the trial only those children who were expected to remain in placement for the duration of the programme (approximately 4 months), this was sometimes difficult to predict and changes in circumstances could occur suddenly. For another two cases, professionals involved with the families decided, after randomisation, that it was not appropriate for the child to receive the intervention at that moment in time. For one child, the social worker considered that engaging in another intervention programme was too much for the child, who was already also receiving Theraplay. Similarly, for another, the child’s CAMHS worker felt that the foster carer and child were already engaged with too many services. The fourth case could not be allocated because of a lack of VIPP-FC intervener capacity.

Process observations regarding intervention delivery

The modifications introduced to the manual and delivery of the programme, including shortening the gaps between sessions, were received well. We received positive feedback from both interveners and foster carers about the programme overall. We also collected feedback about the VIPP-FC from the interveners’ logbooks, which they completed after every session. The key points that interveners noted were as follows:

• Foster carers enjoyed watching the videos and engaged with them.
• The programme helped them to notice, interpret and react to behaviours and signals of their child that they had not seen before.
• The programme helped carers notice changes and improvements in the child’s behaviour over time that they would not otherwise have noticed.
• The programme enabled carers to recognise how they can more sensitively respond to the child.
• It helped carers to see how strong their bond was with their child.

See Chapter 7 for the detailed impressions of foster carers.

The main challenges with the delivery of the intervention were the changing capacity of interveners, the small numbers of interveners in any one site (despite the large number overall) and, relatedly, difficulties in matching incoming cases to interveners’ locations and availability. This meant that, in addition to one child not being allocated to the intervention, six interveners whom we had trained were not allocated to any cases.

Challenges to clinical process

One common problem to emerge was that, because we were working with older children (up to aged 7 years) than is usually the case with the VIPP-SD, the older children often listened to the instructions the intervener was giving or to the feedback; this meant that interveners had to be sensitive to the wording and language used in front of the child. Accordingly, a recommendation for the future is that it would be ideal to plan for older children not to be present for the delivery of feedback. Another recommendation is to plan to have someone present to look after the other children at home when watching the clips. Sometimes foster carers said that the other foster child(ren) might be distressed by the target child having the programme and them not, or they would require their attention too much throughout the visit.

Relatedly, a frequent experience of interveners was a greater need to think ‘on their feet’ as the children could be particularly challenging and the carer was sometimes very busy/distracted by what else was going on for the child that day, or by other children, both when filming and during the feedback. It also became apparent how important it was for interveners to fully understand the stress that a lot of foster carers are under and the types of concerns they have. An issue raised by some interveners was that foster carers noted that some VIPP-FC tips can appear to contradict advice given by social workers, for example about whether or not to use physical touch with the child. In future, this could be incorporated into the intervener training so that they are more prepared to handle this sort of situation.

An important issue that emerged in this study related to the fact that some interveners had a dual role, both study intervener and holding another position within the local authority or CAMHS. It was reported to be difficult for some interveners to keep their VIPP-FC role separate from their main role, especially if the case was also being seen by their team independently of the VIPP-FC work. For example, sometimes the carers wanted to talk about non-VIPP-related issues, such as network meetings, in the visits, which then led to much longer visits. This also raises the question of who is best placed to deliver the VIPP-FC with this population. For a future trial, the advantages and disadvantages of different models of delivery need to be considered. Some advantages and disadvantages of having the VIPP-FC embedded in CAMHS, rather than delivered by, for example, external trial therapists, were noted.

Discussion

In this study, we ran a pilot RCT to test a number of key feasibility questions facing any future trial of VIPP-FC in a UK foster care context. Key issues to be addressed include the identification and recruitment process, retention rates in the research protocol and the intervention, and the quality and completeness of data collection. Issues of acceptability of the intervention and of the research are primarily dealt with in Chapter 7.

Overall, the screening tools performed well, demonstrating good convergence and sensitivity against the DAI, which is generally considered a gold-standard assessment. We concluded that, on balance, the DAWBA-RAD subscale is the most suitable screening instrument if a short questionnaire is required, but noted that the DAI itself was often preferred by foster carers. Given that in-person contact may be extremely important in engaging foster carers, home-based recruitment and initial screening using the DAI may even be preferable to a postal or online mail-out system for this population.

The return rate for the screening questionnaires was disappointing and, unfortunately, much lower than our first pilot run during the case series. Overall, the return rate was 30%, which meant that the total number of potential recruits in any one site was quite low from the outset. One possible reason for this is that, despite considerable effort, we were rarely able to effectively join social worker team meetings or foster carer meetings, and we lacked a mode of direct contact with foster carers. In several sites, it was not possible to find a person who could consistently act as the study champion. Relatedly, in only one of our study sites was a member of the research team granted access to the local authority through an honorary contract, despite efforts to achieve this in all sites. Crucially, the effectiveness with which we were able to implement processes to support the follow-up and encouragement of carers to return the questionnaires in each site was a major determinant of the return rate. When this went comparatively well, return rates were between 40% and 60%.

Our preliminary data suggest that approximately one-third of children in the sample meet the criteria for RAD, according to the DAI. However it is unclear, of course, if the prevalence in this sample is representative of the population of children in foster and kinship care or placed with guardians in the UK, as non-responses may not be random and our sample may selectively over-represent low- or high-scoring cases. This rate was close to our initial projections, albeit based on very limited epidemiological data. In that sense, the primary reasons for low recruitment rates concerned the smaller than expected number of cases deemed suitable (by the local authority) prior to screening and the relatively low return rate for the screening questionnaires [as well as impacts caused by delays, sometimes related to Office for Standards in Education, Children’s Services and Skills (Ofsted) inspection processes, and two sites becoming unworkable]. It was not possible for us to access data on the children who were excluded from the initial screening lists, so we lack vital information on the validity of this initial triaging process undertaken by the local authorities (which no doubt varied across sites). Nevertheless, anecdotally, a considerable number of cases were excluded from the screening lists for valid reasons, often related to the short-term nature of the placement or complex upcoming court hearings.

Even though the prevalence of RAD symptoms was not substantially below our expectations, it nevertheless meant that two-thirds of foster families who were engaged in the screening process could not take part in the trial. The poor recruitment rate threatened our ability to run a sufficient number of cases through the intervention and research process and jeopardised our ability to study a number of other key feasibility questions. In the light of these issues, and in view of the high level of vulnerability to more broadly defined attachment and emotional difficulties of children in foster care, we chose to remove the RAD criterion and widen the group of children allowed to take part in the trial. This, combined with the inclusion of three additional sites, led to a substantial improvement in recruitment rates. Notably, the majority of children who entered the trial who would not have qualified according to the RAD criterion presented with other (non-RAD-related) significant concerns regarding their emotional development.

It is also critical to note that the inclusion of the three additional sites probably made a significant contribution to the later improvements in recruitment. Our informal observations suggest several characteristics of successful sites. First, we generally had greater success in sites outside of London. Second, sites tended to be more successful when we were able to identify a single energetic individual in the local authority who could take responsibility for moving the project forward and who had strong support from their management team. Third, sites tended to work well when CAMHS colleagues worked very closely with the local authority and/or when a member of the research team held a local authority honorary contract. Having all of these factors in place consistently is likely to be essential for a successful larger-scale trial.

Other factors that had a negative impact on recruitment in the local authorities were significant changes in the senior management teams, Ofsted inspections and high levels of staff turnover. Indeed, organisational changes and frequent staff turnover also had a negative impact on the delivery of the intervention. In the course of the study, we trained a large number of interveners to replace those who became unavailable as a result of changes in their availability or because they moved jobs. Simultaneously, the very limited time that interveners were able to dedicate to the study in each site meant that sites could easily run out of capacity to deliver the VIPP-FC; in two instances, this meant that cases randomised to the VIPP-FC were not actually provided with the intervention. Our reliance on the goodwill of CAMHS staff to dedicate their unfunded time to deliver the VIPP-FC was a limiting factor, leading services to offer very small numbers of practitioners and limiting their involvement to a very small number of cases (usually only one case at a time). All of these observations speak to the value of having a comparatively small number of large and well-functioning sites, and finding mechanisms to adequately support clinical services in providing the intervention (or having university-based trial therapists). The new NIHR funding formula for treatment costs, which came into effect after this study, may make this more feasible in the future.

The research protocol itself proved to be feasible, with no refusals of randomisation and high levels of data completeness. Missing data were more of an issue for the post-intervention DAWBA, which was conducted online; for this population, this is not an approach we would recommend. Retention in the intervention was very high; indeed, in all cases the reason for not proceeding with the intervention was resolvable process issues, rather than dropout or refusal. The study visits were relatively straightforward to arrange, although it was notable that an essential ingredient in this was the research team’s flexibility regarding the location, date and time of day for the research visits. Children who attended primary school (as opposed to nursery) or who had contact with the birth family several times per week tended to be more difficult to schedule. Overall, carers’ feedback regarding the assessments was positive, particularly regarding interviews and observational measurements. In addition, carers often commented on how the children enjoyed coming to the study assessments. However, a common piece of negative feedback was that carers felt that they completed many questionnaires and forms as part of their role, and so the study measures represented a tiresome additional burden. Keeping questionnaires, and forms more generally, to a minimum is recommended.

In conclusion, the pilot trial demonstrated that many aspects of a trial for the VIPP-FC in the foster care context were feasible, but fundamental questions arose about the feasibility of recruitment.