The aim of WP1 was to adapt an existing quality assessment tool and an existing patient-rated outcome measure for use in supported accommodation services. The QuIRC26 is an international, standardised tool that assesses quality of care in longer-term mental health facilities. It is completed by the service manager and provides descriptive data and quality ratings, expressed as percentages, for seven domains of care (living environment, therapeutic environment, treatments and interventions, self-management and autonomy, social interface, human rights, and recovery-based practice). It has excellent inter-rater reliability and domain ratings are positively associated with standardised measures of service users’ autonomy and experiences of care.26,27 Therefore, it can provide a proxy assessment of service users’ views of a facility even though it is completed by the unit manager. It is freely available as a web-based resource (www.quirc.eu). Table 1 provides a summary of the tool content and structure.
| Domain | Number of items scoring domain | Examples of areas covered | Example of item |
|---|---|---|---|
| Living environment | 22 | Privacy, décor, cleanliness, meals, mobility access, access to laundry facilities, access to outside space | Is there a private room for patients/residents to meet with their visitors? |
| Therapeutic environment | 35 | Staffing, aims of service, therapeutic optimism, service user involvement in decisions about the service, staff supervision | How hopeful are you that the majority of your current patients/residents will show improvement in their general functioning over the next 2 years? |
| Treatments and interventions | 27 | Staff training, facilitating access to evidence-based interventions, managing challenging behaviour, processes for review of treatment and care | How often are therapeutic effects and side effects of psychiatric medication reviewed? |
| Self-management and autonomy | 27 | Facilitating service user involvement in decisions about own care, supporting service users to gain skills for more independent living, avoidance of ‘blanket restrictions’ | Is there a process for supporting patients to manage their own medication? |
| Social interface | 10 | Facilitating access to community resources, engaging with family, supporting service users’ friendships, providing support to vote in elections | How many of your patients/residents regularly take part in activities in the community? |
| Human rights | 24 | Access to advocacy and legal representation, formal complaints process in place, case notes kept securely and confidentially | Is a welfare/benefits advice service available to your patients/residents? |
| Recovery-based practice | 19 | Individualised and collaborative care planning, ex-service users employed in the service, working towards successful move-on to more independent accommodation | Please estimate the number of your patients/residents who have moved on from your unit to more independent accommodation in the last 2 years? |
The CAT scale is a seven-item, international, standardised, patient-reported outcome measure with good psychometric properties28 that was originally developed for inpatient mental health care (see Appendix 1). Each item is rated from 0 (not at all satisfied) to 10 (entirely satisfied).
Methods
Tool content review and adaptations
The content of the QuIRC was first reviewed by the research team to identify obviously problematic items (e.g. irrelevant items or those requiring rephrasing). Three focus groups with staff and service users recruited from North London were held, one of each focus group from each of the three main types of supported accommodation in England (residential care, supported housing and floating outreach) to gain participants’ views on the relevance of individual QuIRC (staff) and CAT (service user) items. Focus groups were facilitated by the researchers under the supervision of Gerard Leavey. The relevant QuIRC items (staff focus groups) and all seven CAT items (service user focus groups) were used to structure and focus the discussion, which aimed to identify items that required amendment/deletion and additional items. The focus groups were recorded and transcribed and the researchers collated responses.
The findings were supplemented by the advice of three panels of experts who also reviewed the QuIRC and CAT. The first panel comprised five members with expertise in supported accommodation (two senior clinicians, a service manager, a senior policy advisor and a Care Quality Commission senior adviser). The second expert panel was our QuEST study service user reference group, which comprised three members with lived experience of specialist mental health supported accommodation and services. The third expert panel was the North London SURF, which comprised 12 members with lived experience of mental health problems and expertise in mental health services research. All three expert panels were sent the original QuIRC and CAT and a document summarising the comments from the focus groups. They were asked for their opinion about the suggested amendments and any additional amendments. The researchers collated all responses from the focus groups and expert panels, identifying items where there was consensus for adaptation, deletion or a new item. These were reviewed by the QuEST PMG to gain final agreement on changes. The revised QuIRC was then piloted with three service managers (one of each of the three types of supported accommodation) in North London and final amendments to wording were made.
Psychometric property assessment of the adapted Quality Indicator for Rehabilitative Care
To assess the psychometric properties (item response spread, internal consistency, inter-rater reliability, sampling adequacy) of the adapted QuIRC, supported accommodation services were randomly selected from a Microsoft Excel® (Microsoft Corporation, Redmond, WA, USA) database of all supported accommodation services (residential care, supported housing and floating outreach) in each of 14 nationally representative areas of England (see Work package 2 for further details of area selection and scoping of services). Two services of each of the three types were randomly selected from each of the 14 areas with the aim of recruiting 20 managers from each type of service (60 in total). The researchers contacted service managers to gain their informed consent for participation. Two researchers attended a face-to-face interview with participating service managers. One researcher led the interview and both researchers independently rated the adapted QuIRC from the answers given by the unit manager.
Data analyses were conducted by Sarah White, statistician from St George’s University London, who was involved in the development of the original QuIRC. Items were considered to have inadequate response spread if > 90% of service managers gave the same response. Internal consistency of domain scores was assessed using Cronbach’s alpha and considered acceptable if > 0.6. Inter-rater reliability was assessed using kappa coefficients for categorical data (weighted kappa if more than two categories) and intraclass correlation coefficients (ICCs) for normally distributed, continuous data, and considered acceptable if > 0.7.
Psychometric property assessment of the adapted Client Assessment of Treatment
The psychometric properties of the adapted CAT were assessed in the 618 service users who participated in the national survey of supported accommodation services in WP2 (see Work package 2 for full details of the approach to sampling and recruitment). Internal consistency was assessed using Cronbach’s alpha. Convergent validity was assessed through correlation with ratings of participants’ subjective quality of life using the Manchester Short Assessment of Quality of Life (MANSA),29 which has a total maximum mean score between 1 and 7.
Results
Each staff focus group comprised four members (the service manager plus three support workers). The three service user focus groups had five or six members (residential care, n = 6; supported housing, n = 5; floating outreach, n = 5).
Adapted Quality Indicator for Rehabilitative Care
A total of 28 QuIRC items were rephrased, 20 items were deleted and 10 items were added. For example, item 77 from the original QuIRC was adapted to better reflect the nature of support within these settings (original item: ‘How many of your staff are trained in control and restraint techniques?’; amended item: ‘How many of your staff are trained in breakaway techniques?’), whereas item 116 was simply omitted (‘Are patients/residents free to send and receive uncensored mail or email?’). The final version had 143 items. It was also agreed that because floating outreach services are not ‘building based’ but provide support to people living in an independent tenancy, the items relating to the living environment of the service were not relevant and, therefore, the adapted QuIRC would not be able to provide a rating on this domain for these services.
Inter-rater reliability of the QuIRC was carried out with managers of 14 residential care homes, 21 supported housing services and 17 floating outreach services (52 services in total).
Only 16 out of the 143 items showed a poor response spread. Internal consistency was inadequate for all domains except therapeutic environment and treatments and interventions. However, the analysis was limited by a relatively small sample size and the lack of variability in response to some items. With regard to inter-rater reliability, 70 ICC analyses were conducted and only one item was found to be unreliable. A total of 186 kappa coefficient analyses were conducted and four items were found to be unreliable (κ < 0.7). In addition, there were 14 items where analyses could not be conducted because there were too few cases (five items), zero variance (two items) or where variables were constants (seven items). The full results of these assessments are available in Killaspy et al.20
The PMG agreed amendments to the adapted QuIRC in response to the results. It was decided to keep items with inadequate variance because (1) to drop them would have disrupted the logical flow of the tool content and (2) greater variance might be achieved in future development of the tool for use in settings outside the UK. Additional explanatory information was added to improve the reliability of one item, one item was dropped completely and unreliable response options were dropped for three items.
It was also agreed that internal consistency would be reassessed using the larger sample of 87 services participating in WP2 along with an assessment of sampling adequacy (the proportion of variance among the variables that might be common variance) using the Kaiser–Meyer–Olkin (KMO) statistic. Hair30 suggested a KMO value of ≥ 0.5 to evaluate whether or not a sample of data has sufficient common variance to make exploratory factor analysis appropriate.
Internal consistency improved on all domains with the larger sample (Table 2). All domains met the KMO > 0.5 criterion of sampling adequacy. It should be noted however that both Cronbach’s alpha and the KMO statistic are influenced by sample size and in this analysis the sample size was still smaller than desirable for robust estimates of both properties (generally 300 observations). However, we were able to investigate whether or not there was adequate common variance within the domain items to assume that the domain had coherence. We replicated the approach taken in the original development of the QuIRC, where items were considered to load onto a factor (domain) if they scored > ± 0.3. As some items had zero variance, they were removed before analysis (living environment: two items; self-management and autonomy: three items; human rights: four items). All items loaded onto a factor within that domain at the > ± 0.3 level (i.e. each item was contributing to the common variance within the domain).
| Domain | Number of items scoring domain | WP1 services (n = 52) | WP2 services (n = 87) | |||||
|---|---|---|---|---|---|---|---|---|
| Mean (SD) domain score | Range | Cronbach’s alpha | Mean (SD) domain score | Range | Cronbach’s alpha | KMO | ||
| Living environment | 20 | 81.0 (7.1) | 62.3–94.3 | 0.39 | 81.2 (8.7) | 53.9–96.2 | 0.56 | 0.58 |
| Therapeutic environment | 33 | 62.2 (7.3) | 48.5–78.9 | 0.66 | 61.4 (6.9) | 38.2–75.4 | 0.66 | 0.51 |
| Treatments and interventions | 27 | 55.1 (8.4) | 36.7–76.3 | 0.66 | 54.2 (8.1) | 35.1–73.2 | 0.64 | 0.61 |
| Self-management and autonomy | 33 | 69.0 (5.8) | 53.7–81.8 | 0.40 | 68.0 (6.9) | 39.3–83.8 | 0.62 | 0.58 |
| Social interface | 7 | 59.0 (10.8) | 33.9–89.7 | 0.27 | 58.9 (12.1) | 37.6–85.6 | 0.49 | 0.56 |
| Human rights | 21 | 86.7 (5.0) | 71.4–96.7 | 0.09 | 85.5 (6.9) | 66.1–97.5 | 0.37 | 0.53 |
| Recovery-based practice | 18 | 71.7 (8.2) | 51.9–91.4 | 0.53 | 69.2 (9.9) | 31.8–90.5 | 0.67 | 0.57 |
We engaged the same information technology specialist who developed the web-based version of QuIRC to develop a similar application for the adapted QuIRC for supported accommodation services. This increases its accessibility, reduces the time required to complete it compared with a face-to-face interview, and provides a built-in scoring algorithm. It has a similar facility to the original QuIRC application in producing a printable report for the service manager about the performance of their service on the adapted QuIRC domains, comparison benchmarking data for similar services and suggestions for how to improve performance. The final version of the adapted QuIRC was named the Quality Indicator for Rehabilitative Care: Supported Accomodation (QuIRC-SA).
Adapted Client Assessment of Treatment
All seven CAT items were considered relevant by the focus group participants and expert reference group members, with only slight modification of the wording required (e.g. ‘treatment/care’ was changed to ‘support/care’). No items were deleted or added. Cronbach’s alpha was 0.89. With regard to convergent validity, scores on the adapted CAT were positively correlated with the mean total MANSA (quality of life) score (rs = 0.35; p < 0.001). Full results are available in Sandhu et al.21
Links to other work packages
The QuIRC-SA and CAT-SA were used to assess service quality and service user experience in the national survey of supported accommodation services (WP2i), the prospective cohort study (WP2ii) and the feasibility trial comparing supported housing and floating outreach (WP4).
Limitations
Quality Indicator for Rehabilitative Care: Supported Accommodation
One of the limitations of the QuIRC-SA was the inadequate internal consistency. Although the domains were coherent [as measured by sampling variance (KMO statistic)], it is possible that the estimates of internal consistency (Cronbach’s alpha) may have been influenced by our sample size; a sample size of 300 to 400 is desirable in order to obtain robust estimates of internal consistency. The decision to retain items with inadequate response variance may be considered a limitation of the tool. Although it could be argued that the retention of these items unnecessarily increases the size and administration time of the QuIRC-SA, omitting them would have disrupted the logical flow of the tool. We were not intervening to improve service quality during the QuEST programme and we therefore did not assess the tool’s sensitivity to change. Test–retest reliability was not assessed as we felt that this was too onerous for busy service managers (as the tool takes 1 hour to complete). Both should be assessed in future studies, but their omission had no influence on the findings of later phases of the QuEST programme as we used only baseline QuIRC-SA assessments.
Client Assessment of Treatment for Supported Accommodation
Although the development and testing procedures for the CAT-SA were rigorous, a number of limitations must be acknowledged. First, we did not collect data on symptoms or clinical profile when assessing the psychometric properties of the measure, which negates the possibility of investigating potential relationships between CAT-SA ratings and symptomology. Second, owing to the nature of our investigations, test–retest reliability and sensitivity to change have not been assessed. Third, additional validation procedures are needed to establish its validity in countries outside England. Finally, service user experience/appraisals on the CAT-SA may be influenced by support received outside the supported accommodation setting.
Key findings
Quality Indicator for Rehabilitative Care: Supported Accommodation
- The QuIRC-SA is the first standardised tool for quality assessment of specialist mental health supported accommodation services.
- Of 143 items, only 18 showed a narrow response range, and five had poor inter-rater reliability.
- The QuIRC-SA had excellent inter-rater reliability and exploratory factor analysis showed that items loaded onto the domains to which they had been allocated.
- The QuIRC-SA can be recommended as a self-report quality assessment tool for supported accommodation services.
- A digital version is available, free to use, at www.quirc.eu.
Client Assessment of Treatment for Supported Accommodation
- All items of the CAT were considered relevant to mental health supported accommodation services; slight changes to wording were made.
- The CAT-SA demonstrated good internal consistency and convergent validity.
- The CAT-SA can be recommended as a ‘patient-reported outcome measure’ for use in mental health supported accommodation services.