Work package 2: assessing validity of parent-interaction observational tools

What is the most reliable, valid and clinically feasible observational measure for routinely assessing parent–infant interaction quality in PMH services?

Background

Parts of this chapter are reproduced with permission from Sharp et al.30 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. The text includes minor additions and formatting changes to the original text.

Specialist PMH services require guidance as to how to meaningfully assess, target and evaluate perinatal interventions that aim to improve the quality of parent–infant interactions (e.g. maternal sensitivity, intrusiveness, withdrawn behaviours, warmth/positive regard). These early parent–infant interactions are important mediators or moderators of PMH on child mental health outcomes, and also represent important early modifiable risk factors for poor child outcomes in their own right. Observational measures are considered the ‘gold standard’ index of parenting quality, superior to and generating different information from parental self-report. However, the recent Royal College of Psychiatrists working group on outcome measures were unable to recommend an observational tool for this purpose31 because existing measures either are too training and time intensive, lack evidence for their predictive validity to later child mental health outcomes or require a lengthy filmed observation period not commensurate with routine clinical practice. In this WP, we aimed to evaluate the psychometric properties of two parent–infant interaction observational coding schemes, selected from the research and clinical literature, using video archive material from a prospective, longitudinal Medical Research Council-funded study with follow-up from pregnancy to school age [The Wirral Child Health and Development Study (WCHADS)32].

The two measures we selected are described briefly below, along with the reason for their selection:

1. The Parent–Infant Interaction Observation Scale (PIIOS).33,34 Developed as a clinical screening tool for coding 3- to 4-minute parent–infant interactions between age 2 and 8 months, its concurrent validity has been established against the maternal sensitivity subscale of a detailed measure of mother–infant interaction (the CARE-Index). Training has been rolled out in specialist PMH teams in England on the basis of this, but it is not yet routinely used. However, predictive validity for the PIIOS had not yet been established to later meaningful child outcomes like attachment security or child mental health indices.
2. The revised ‘Qualitative Ratings for Parent–Child Interaction’.35 Adapted from the NICHD Study of Early Childcare and Youth Development (SECCYD) system,36 this system has been widely used internationally in research for infants age 3–15 months. This coding system has established predictive validity to attachment security and later child psychopathology using 10–15 minutes of mother–infant observation.37,38 However, the reliability and predictive validity from a shorter observation period with ‘clinical utility’ are not known and perinatal clinicians view a 10- to 15-minute observation period as too lengthy in the context of parents experiencing moderate to severe mental health problems.

We therefore tested the inter-rater reliability and predictive validity of both the NICHD system and PIIOS system. In the original plan, child outcomes were attachment security and disorganisation (age 14 months) and child externalising and internalising problems at school entry using Pre-school Child Behavior Checklist (CBCL)39 (age 4–5 years). However, we completed additional work during the course of WP2 in which we also examined predictive validity to child mental health outcomes assessed using Brief Infant-Toddler Social and Emotional Assessment (BITSEA)40 at age 1 and 2, also available within the WCHADS data set. In order to further inform clinical practice, we also tested whether brief forms of each tool (subsets of items) could be identified that retained acceptable psychometric properties, and whether the reliability and validity of the measures varied depending on length of observation on which ratings were based.

The full NICHD-SECCYD and PIIOS systems are similar, in that they each involve coding multiple dimensions of parent–infant interaction and both aim to capture parental sensitive-responsiveness. Researchers have typically previously used the NICHD system in a hypothesis-driven manner to examine specificity of prediction from specific parenting dimensions to later child outcomes of interest. Internationally, some research groups have used a single coding dimension (e.g. global sensitivity or sensitivity to distress) and others have used subsets of codes. Commonly a three-item composite has been used to yield a broader ‘sensitivity’ composite, using sensitivity to non-distress or global sensitivity (which includes non-distress and distress episodes during play), positive regard (also known as warmth) and intrusiveness (reverse coded) examining prediction to later child mental health or attachment-related outcomes.41–45 In contrast, for PIIOS, there is a limited research literature with only one publication describing the development of the measure.4 The manual5 recommends calculation of a total score across 13 parent–infant interaction dimensions to index overall sensitive-responsiveness and then conversion of this score into three categorical domains (no concern, some concern, significant concern), designed to drive clinical decision-making and intervention level (e.g. Universal, Universal Plus and Universal Partnership as part of the UK Department of Health’s Healthy Child Programme).46,47 To our knowledge, no previous research has been conducted to examine whether or not the whole scale or a subset of codes within PIIOS has validity in the prediction of later future child outcomes.

In summary, in order to determine which measure or combination of codes within each scale had optimum predictive validity and brevity, and thus clinical utility, we examined whole scale performance and item-subset performance, determined a priori from the research literature or guidance manual for the tool, or from factor analytic or machine learning analyses conducted within this study. Since parents with moderate to severe mental health problems may not tolerate lengthy observation, we further tested whether 3-, 5- or 7-minute observation periods incrementally improved inter-rater reliability and predictive validity to child outcomes or not.

Original aims

1. To derive indices of clinical utility
   1. Time to train staff*
   2. Costs to train staff*
   3. Length of reliable observation period
2. To test the comparative predictive validity of the two coding schemes over varying observation periods (3, 5, 7 minutes) to gold-standard measures of attachment security at age 1 and total child mental health symptoms (internalising and externalising) at school entry (age 4–5 years).

Additional aim addressed during course of work

1. To use the same study sample and methodology to test the comparative predictive validity of the two coding schemes over varying observation periods (3, 5, 7 minutes) to shorter-term mental health outcomes (total internalising and externalising problems) at age 1 and 2 years.

*Note: time and costs to train staff were linked to our original bid proposal which included a new cohort study in which multiple staff were to be trained in order to generate reliable estimates of time taken for a range of multidisciplinary healthcare staff. Since this element of the original proposal was not funded, we are not able to report here on time and costs to train clinical staff.

Findings related to Aim 1 length of reliable observation period, Aim 2 predictive validity in relation to attachment outcomes and Aim 3 to the shorter-term mental health outcomes are reported in detail in Sharp et al.30 Here we report a brief summary of these findings, and we report on the findings in relation to Aim 2 prediction of school entry mental health outcomes (age 4–5 years).

Methods

Overall design

Data were drawn from a longitudinal data set in which film footage was gathered of parent–infant interaction in a play-based task at 6–8 months of age and follow-up outcome data were gathered from cohort members at age 1, age 2 and age 4–5 years (see detailed methods in Sharp et al.30)

Sample size and estimated power

The Consensus-based Standards for the selection of health Measurement Instruments (COSMIN)48 recommendation is for a sample size of at least 50 for internal consistency (using Cronbach’s alpha),49 which is sufficient also for inter-rater reliability [agreement coefficients, such as the intraclass correlation coefficient (ICC)].

Power analysis

For the mental health problem score, we will report univariate measures of association (for which we will have 90% power for explaining 4% of behaviour variance) and test for a difference in predictive power of the two ratings (for which Stata powerreg calculates 80% power is for a difference in explained variance of just 2%).

Sample

A stratified high-risk community sample of 250 mother–infant dyads from UK WCHADS completed a 7-minute play-based interaction at 6–8 months. The stratification variable, interpartner psychological abuse reported by a larger sample of women50 in pregnancy, was chosen for its known association with a variety of risk factors for early child development. The intensive sample comprised a higher risk community sample of first-time mothers in which 53% and 47% represented high- and low-risk strata, respectively. The stratification variable was effective in generating a higher-risk perinatal sample, as by the fact that the mean prenatal Edinburgh Postnatal Depression Scale scores51 in the low- versus high-risk strata were 6.67 [standard deviation (SD) 3.97] versus 9.86 (SD 4.80), Cohen’s d = 0.68, p < 0.001.

Of these 250 dyads, 231 completed the attachment assessment at mean age 14 months. Parents reported child mental health outcomes at ages 1 year (N = 231), 2 years (N = 219) and age 4–5 years (N = 225). Socioeconomic conditions on the Wirral range between the deprived inner city and affluent suburbs, but with very low numbers from ethnic minorities. The demographic characteristics of the sample are given in Table 4. Just under 40% of the sample were living in socioeconomic conditions equivalent to the bottom quintile of UK neighbourhoods52 and just 3.6% were from non-White British ethnicities which is representative of the local area from which participants were drawn.

Measures

Mother–infant observation

The Wirral Child Health and Development Study video archive material of 7-minute play-based mother–infant interaction recorded at 6–8 months of age was coded using the PIIOS33,34 and the revised NICHD-SECCYD6 system. The procedure for the observation of mother–infant interaction followed the manual from NICHD-SECCYD research network.53

The NICHD coding scheme

The revised manual for The Qualitative Ratings for Parent–Child Interaction for 3–15 Months of Age,35 which uses 5-point global ratings adapted from the 4-point NICHD-SECCYD system,36 was used. This yielded ratings for nine maternal and four infant-focused scales and one dyadic scale. Each is rated on a global 5-point scale, ranging from 1 (not at all characteristic) to 5 (highly characteristic). Two ‘a priori’ summary indices were also created for the analysis which combine NICHD dimensions, namely the three-item composite score (NICHD-3) and the NICHD total score (NICHD_total). These were guided by previous empirical research using the NICHD system. The NICHD-3 score (range 3–15) was created by summing 3 codes: global sensitivity, intrusiveness (reversed) and positive regard. The NICHD_total score was the sum scale of all seven maternal codes plus the one dyadic code: global sensitivity, intrusiveness (reversed), detachment (reversed), positive regard, negative regard (reversed), animation, stimulation and dyadic mutuality (range 8–40). High scores indicated more optimal interaction. The internal consistency in the current study for NICHD_total was high at α = 0.84 and for NICHD-3 it was acceptable given the scale consists of only three items, α = 0.67.

Reliability and validity analyses first included examination of individual codes within the NICHD system and the two summary indices described above. The Data analysis section describes further analytic approaches taken to examine whether there might be other more optimal approaches to combining codes to maximise predictive validity of the tool.

The PIIOS coding scheme

The PIIOS34 manual was used to record the interactions and maternal responses on 13 dimensions which together represent overall sensitive-responsiveness according to the authors. The coder rated the interaction based on descriptors given for allocation to a 3-point categorical scale, assigning a score of 0 (no concern), 2 (some concern) or 4 (significant concern) for each dimension, with the lowest score indicating the most optimal interaction and the highest, the least optimal. Once all dimensions were assigned a score, a total score was calculated, hereafter referred to as the PIIOS_total. The total score was then used to create the PIIOS domain score PIIOS_domain, which indicates which category the dyad’s quality of interaction should be allocated to overall according to the manual; no concerns (PIIOS_total scores between 0 and 17), some concerns (PIIOS_total scores between 18 and 25) or significant concerns (PIIOS_total scores of 26+). The internal consistency in this study for the PIIOS_total was high, α = 0.82.

Reliability and validity analyses first included examination of individual codes within the PIIOS system, and the two summary indices described above, total score and domain of concern. The Data analysis section describes further analytic approaches taken to examine whether there might be other more optimal approaches to combining PIIOS codes to maximise predictive validity of the tool.

Outcome measures

Outcomes were:

• infant attachment security and disorganisation (Strange Situation54) at 14 months
• maternal report of internalising and externalising child mental health symptoms (BITSEA40) at 14 and 30 months
• internalising and externalising child mental health symptoms (Pre-School Child Behaviour Checklist39) at age 4–5 years.

For details of all attachment and mental health outcome measures at age 1 and 2, please see Sharp et al. 2024.30 In the absence of UK norms for the BITSEA and CBCL to define clinical level symptoms, we used a cut-point of being in the top decile range for total subscale scores (1) versus below (0) to indicate mental health outcome in receiver operating curve (ROC) analyses.

Covariates

• Socioeconomic status, which was derived from post code data using the English Index of Multiple Deprivation (IMD52) and converted to quintile categories with a binary variable (1 = most deprived, 0 = all 4 other quintiles), was used for analysis.
• Mother’s age, education beyond school and marital status were recorded at consent.
• Child sex was recorded at birth.

Procedure

An incremental approach was taken to coding each video. Each rater coded the 3, then 5, then 7-minute interaction clip for the same dyad to enable examination of any incremental improvement afforded in reliability or predictive validity by increasing the observational period. Rating each interaction in this incremental manner using the full NICHD coding scheme took approximately 30–45 minutes and took 20–30 minutes using the full PIIOS coding scheme.

Inter-rater reliability between each naïve rater and a gold standard rater was calculated for the study data set for 40 randomly selected filmed interactions using PIIOS and 40 using the NICHD system (see Sharp et al. 202430 for additional details of training procedure and main study rating procedure).

Data analysis

We evaluated the psychometric properties of the two mother–infant interaction measures based on COSMIN48 and contemporary psychometric guidelines.55

Reliability

Agreement coefficients were used to evaluate the agreement between the raters (inter-rater reliability) and subsequently agreement of ratings within rater for different observation duration (stability). Raters were Rater 1 (R₁), Rater 2 (R₂) and a golden criterion rater. Data from all three raters were used. As the PIIOS and NICHD codes are rated in skewed three and five points scales respectively, we used the nonparametric Psi coefficient [R package (The R Foundation for Statistical Computing, Vienna, Austria) nopaco56], which can be transformed to the ICC.57 For completeness, we also present the percentage of agreement and Cohen’s weighted kappa coefficient58 where appropriate. Guidelines59 were followed for interpreting the results (values < 0 no agreement; 0–0.20 slight agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial agreement; and 0.81–1 almost perfect agreement).

Exploratory factor analysis

Exploratory factor analysis was carried out to assess the latent constructs of the NICHD and PIIOS, for ordered categorical items, using the weighted least squares mean and variance adjusted estimator.60 Measures of relative and absolute fit were used to assess the goodness of fit of the emerged structures (see Sharp et al.30 for detail). Latent variable analysis was conducted using the Mplus (Muthén & Muthén, Los Angeles, CA, USA) software.61 This data-driven approach was taken to determine if coding dimensions might form factors or subscales for each measure. Any such subscales might then be examined for their predictive validity in relation to the outcomes of interest in this study, alongside those selected on an a priori basis from the literature or original scoring in the manual (e.g. NICHD-3, NICHD_total, PIIOS_total and PIIOS_domain).

Predictive validity

We first examined the predictive validity of the two tools in relation to attachment outcomes using logistic regression. Analysis tested the predictive validity at individual code level, as well as subscale or summary level (NICHD-3, NICHD_total, PIIOS_domain and PIIOS_total).

We next created ROC33 to test the discriminant validity of NICHD-3, NICHD_total, PIIOS_domain and PIIOS_total in the prediction of attachment outcomes and top decile scores on each mental health outcome at different durations of observation. We further tested the discriminant validity of factor scores derived from the exploratory factor analysis above for PIIOS and NICHD systems in relation to top decile mental health outcomes. Discriminative validity was evaluated by the area under the curve (AUC) where values 0.9–1 indicate very good validity, 0.8–0.9 good, 0.7–0.8 fair, 0.6–0.7 poor validity and 0.5–0.6 failed to provide evidence for validity. We also tested the ways in which demographic covariates might affect the ROC curves. First, we tested if a covariate (child’s sex, maternal age and level of deprivation) affects the ability of the measure to discriminate between cases and controls. Second, we tested if the ROC curve is biased by the levels of the covariate. We used the ROC regression (rocreg) process for the testing of significant covariates.

Finally, we used regularised methods (Lasso regression) and cross-validation (machine learning) methods to examine the prediction of attachment classification and symptom outcomes. Both the NICHD and PIIOS coding schemes provide ratings of a range of behaviours, some principally of the mother, some of the child and some that are intrinsically dyadic in nature. It is possible that there are particular coding dimensions and combinations of those dimensions that might be of particular salience for later development of secure attachment and behaviour problems. Such a search requires choosing from among a large number of potential prediction models and requires special attention being paid to the problems of overfitting. Overfitting leads to coefficient estimates being upwardly biased in magnitude, and an exaggerated assessment of prediction success. We applied lasso and split-sample cross-validation methods to reduce the extent of these biases. We compared the performance of the two coding schemes using several sets of predictors in each case.

A priori 8: The first was the single item (appropriately signed) sum score from each scale. There being no model selection, this used regular logistic regression informed by the literature. In the case of the PIIOS, the total score was used (PIIOS_total) as the instrument was developed with the intention of a total score being derived to reflect overall quality of interaction. In the case of NICHD, the literature has typically focused on ratings derived from combinations of the seven adult scales [global sensitivity, detachment (reversed), intrusiveness (reversed), animation, stimulation, positive regard, negative regard (reversed) AND one dyadic scale (dyadic mutuality)] to predict child development outcomes such as behavioural problems. Accordingly, we selected to examine prediction to total internalising and externalising problems at age 1 and 2 from (1) a total score (NICHD_total) representing all eight NICHD dimensions of adult behaviour including dyadic mutuality and (2) a commonly used trio of codes (NICHD-3); global sensitivity, intrusiveness (reversed) and positive regard, identified below as the minimum set.

A priori 3

The set of three NICHD items (NICHD-3), global sensitivity, intrusiveness and positive regard most commonly identified in the literature. There is no equivalent evidence base for PIIOS.

Main

This set included all the individual item scores as main effects and likely freely available demographics (maternal age, child sex, IMD deprivation quintile associated with postcode, maternal education beyond school, marital status).

Interaction

This set included both main effects included in Main but also all pairwise (two-way) interactions, for example allowing for the possibility that maternal intrusiveness in a context of low maternal warmth might be of particular concern.

All analyses were conducted in Stata Version 16 (StataCorp. Stata treatment‐effects reference manual. College Station, TX, USA: StataCorp LLC; 2017) unless otherwise stated.

Results

Psychometric evaluation

The descriptive indices of the measuring tools (BITSEA, NICHD and PIIOS) are presented in Table 5.

Length of reliable observation period

Inter-rater agreement

For all NICHD individual codes, the two raters had excellent inter-rater agreement with the golden rater, regardless of the length of observation. The percentage of agreement across codes was at least 84%, the PSI coefficient varied between 0.72 and 0.94, leading to an ICC coefficient of at least 0.83 (excellent agreement).

For all PIIOS individual codes there was high inter-rater agreement between the two raters and the golden rater for all observation periods, across all codes. The percentage of agreement was at least 84%, the PSI coefficient varied between 0.71 and 0.96, leading to ICC coefficients of at least 0.83 (excellent agreement).

Intra-rater agreement (stability of ratings)

Table 6 shows the agreement coefficients between ratings for different observation periods. For the NICHD_total, the ICC between 3 and 5 minutes, 5 and 7 minutes, and 3 and 7 minutes observation, agreement was excellent for all comparisons though highest between 5 and 7 minutes. Similar results were found for the composite NICHD-3.

For the PIIOS_total, the ICC between 3 and 5 minutes and the ICC between 5 and 7 minutes showed excellent agreement, while the ICC between 3 and 7 minutes was lower at 0.69. These results indicate that adding the first 2 minutes makes a small difference in the reliability scores, but adding the second 2 minutes does not actually make a difference in the scores. With respect to the PIIOS_domain score, as expected, the agreement was lower (due to the categorical classification). Using weighted kappa, the agreement coefficients indicated that the observations at 5 and 7 minutes were in highest agreement. See Sharp et al.30 for coefficients of agreement for different scenarios.

Five minutes observation was optimal to achieve an excellent level of stability in mother–infant interaction ratings.

Exploratory factor analysis

For detailed information, see Sharp et al.30

Factor analysis for categorical data was performed for PIIOS and NICHD systems separately to establish whether or not a new combination of parenting codes might prove reliable and predict later outcomes. Factor analysis led to a two-factor model in each case. The factors derived for each measure were of good or satisfactory internal consistency and content validity. However, when ROC analyses were conducted to assess predictive validity to attachment and mental health outcomes, factor scores had poor discriminant validity in the case of prediction to each mental health outcome at age 1, 2 and 5 and attachment outcome at age 1 (AUC < 0.70).

We concluded that the predictive validity was not increased if the information from NICHD or PIIOS was summarised using a different clustering of the codes, to that proposed a priori in the Measures section above, for either tool. See Sharp et al.30 for more detail.

Predictive validity to attachment outcome at age 1

Although logistic regression analyses showed multiple individual codes and the summary indices in both measures significantly predicted later attachment security, and for NICHD some codes also predicted attachment disorganisation, ROC analyses revealed neither the NICHD_total, NICHD-3 nor the PIIOS_total, PIIOS_domain predicted attachment outcomes at the level required for a screening tool (AUC > 0.70). For more detailed information, see Sharp et al.30

Predictive validity to mental health outcomes

For 5 minutes observation using NICHD-3, ROC curve analysis demonstrated fair discriminant validity to later BITSEA externalising problems (threshold – top 10%) at age 1 and 2 (AUCs = 0.74), but values were < 0.70 threshold for prediction of internalising problems and age 4–5 CBCL outcomes. Values of AUC from NICHD_total fell < 0.70 threshold for outcomes at age 2, indicating use of brief NICHD-3 is optimal for the NICHD-SECCYD system.

For 5 minutes observation using PIIOS_total, AUCs ranged from 0.72 to 0.82 indicating fair to good discriminant validity in the prediction BITSEA internalising and externalising problems, respectively, at age 2, but values were < 0.70 threshold in relation to age 1 BITSEA and age 4–5 CBCL outcomes. For PIIOS_domain, AUC was 0.77 indicating fair discriminant validity in the prediction of age 2 BITSEA externalising problems only, but values were also < 0.70 threshold in relation to age 1 BITSEA and age 4–5 CBCL outcomes, indicating use of PIIOS_total, rather than the domain-based scoring system, is optimal for this tool.

In ROC regression analyses, ROC curves at specific values of the covariates were not implemented, as no covariates (maternal age, gender, IMD quintile) were identified as having a significant effect on the discriminatory ability of either measure. Adjustment for significant covariates (maternal age for NICHD and all three covariates for PIIOS) resulted only in a marginal fall in AUC.

Predictive validity in relation to length of observation

For both NICHD-3 and PIIOS_total, optimal prediction to age 2 mental health outcomes was reached at 5 minutes observation, with values of AUC falling below threshold for the shorter 3 minutes of observation in the case of NICHD-3 in relation to age 2 externalising problems, and in the case of PIIOS_domain and PIIOS_total in relation to age 2 internalising problems.

Derivation of cut-points from receiver operating curve analyses for use in clinical practice

Using data from 5 minutes observation for NICHD-3, a score of 11 or below had 92.8% sensitivity and 52.2% specificity to detect membership of the top decile at age 1, correctly classifying 54%. A score of 10 or below had 72.7% sensitivity and 66.3% specificity to detect membership of the top decile for externalising problems at age 2, with correct classification of 67%.

Using data from 5 minutes observation for PIIOS_total, a score of 20 or above had 81.8% sensitivity and 74.5% specificity to detect membership of the top decile for externalising problems, with 75% correctly classified, and had 66.7% sensitivity and 73.3% specificity (73% correctly classified) to detect membership of the top decile for internalising problems at age 2.

Prevalence and characteristics of high-risk dyads

Using the cut-points for sensitivity and specificity derived above identified 28.3% of dyads as at risk according to PIIOS_total and 24% dyads as high risk on NICHD-3 in our higher-risk community sample. Although the overall agreement level in classification of dyads as high or low risk was 73% so a dyad that was rated high risk on PIIOS was four times more likely to be rated as high risk on NICHD [odds ratio (OR) = 4.0, p < 0.001], the two scales also appeared to detect slightly different forms of risk.

Logistic regression revealed that those dyads identified using NICHD-3 were at raised risk of insecure attachment (OR = 2.7, p = 0.004) and top decile level externalising problems at age 2 (OR = 1.9, p = 0.14) which suggests the threshold for externalising problems derived above can be used to identify dyads at risk for attachment insecurity, whereas dyads identified using PIIOS were at raised risk for externalising (OR = 3.75, p = 0.002) and internalising problems at age 2 (OR = 2.16, p = 0.073) but not so clearly for insecure attachment (OR = 1.5, p = 0.15, ns). However, these results need to be considered with caution. ORs are best used as the index of effect size here since the small sample size constrained our power to detecting moderate effects as significant.

We did calculate the comparative likelihood of being classified as insecure based on above/below threshold status on PIIOS_total and NICHD-3. Table 7 shows that the odds of being insecure are threefold higher for dyads scoring above threshold on both tools, compared to dyads scoring as low risk on both measures. However, the raised risk for attachment insecurity is driven by the NICHD-3 rating as we can see that the risk of insecure attachment is not raised in those dyads who only score high on the PIIOS are those low risk on both measures [Chi (3) = 9.08, p < 0.05].

Predictive validity: Lasso regression using scales and scale items from 5-minute ratings and background/demographic variables

Regularised methods (Lasso regression) and cross-validation (machine learning) were used to examine the prediction of attachment classification and mental health symptom outcomes. Both the NICHD and PIIOS coding schemes provide ratings of a range of behaviours, some principally of the mother, some of the child and some that are intrinsically dyadic in nature. We considered the possibility that there might be coding dimensions and combinations of those dimensions within each measure, including interactions between codes, that have not yet been identified in the literature, which might be of salience for later development of secure attachment and mental health problems at age 1, 2 and 4–5 years. However, in almost all cases, analyses revealed that the a priori assumed scores (PIIOS_total, NICHD_total and NIICHD-3) performed better than any other new combination of codes. The results of these analyses are given in Sharp et al. 2024.30

Discussion and conclusions

• Using film footage recorded at 6–8 months of age in a high-risk community sample, we provide evidence for the reliability and predictive validity of two observational tools which we conclude are suitable for routine use in perinatal clinical practice to identify parent–infant dyads who may benefit substantially from interventions designed to improve interaction quality.
• These measures comprise (1) a short form of the NICHD coding system, namely the NICHD-3, or (2) the PIIOS in its full form using the total score (PIIOS_total). No shorter form of PIIOS was identified with sufficient predictive validity.
• Optimal reliability (stability of ratings) and predictive (discriminative) validity to age 2 mental health outcomes were achieved by using 5 minutes observation for both NICHD-3 and PIIOS_total. In clinical practice we therefore advise filming for a 5-minute period, rather than shorter or longer periods which confer, respectively, disadvantage or no advantage in terms of predictive validity.
• Neither observational tool predicted longer-term mental health outcomes at school entry, but this is not surprising if one considers the many intervening events or circumstances in children’s lives that might influence outcomes between 6 and 8 months of age when parenting was assessed and school entry at age 4–5 years.
• While PIIOS had broader predictive validity to age 2 internalising (emotional) and externalising (behavioural) mental health outcomes, it was developed for use with infants between 2 and 8 months which limits application in services with a broader age range of infants. The full system needs to be used as we found no short form. Training takes 3 days, plus time to practise and complete the reliability set of observations. Once trained, coding a 5-minute interaction takes 20–30 minutes depending on complexity.
• The NICHD-3 which focuses only on parental sensitivity, intrusiveness and positive regard (warmth) may have greater clinical utility due to its comparative brevity, its predictive validity to both attachment security and externalising problems and application from 3 to 24 months of age which matches the broader needs of specialist PMH services. We estimate the time to train to reliability will require 2 days, rather than 3 days for the full system, plus time to practise and complete the reliability set of observations. Once trained, coding a 5-minute interaction takes 15–20 minutes, rather than 30–45 minutes for the full system.
• Services who have already invested in staff training on these observational tools can use them with confidence for the infants who are within the age ranges that each respective tool has been validated on.
• Future research needs to establish the sensitivity to change following intervention for each tool.
• Our findings used a higher-risk community sample drawn from a population sample of consecutively registered mothers for antenatal care on Wirral and, as such, represented the local population well, with higher levels of socioeconomic deprivation than the national population of England. However, few families were non-White British and so the findings reported here are limited in this respect in terms of generalisation to more diverse populations.