Work package 1: LIAISE

Participatory research methods

The VOICE and VOTE measures were generated using a model of stakeholder involvement which holds at its core the concept of empowering under-represented groups in order to create well-balanced, responsive services. This method, pioneered in service user research,17,19–21 is novel because a researcher from the same group as the participants guides the process and the data are allowed to emerge by following the themes that the participants feel are the most important. This was also adapted for a nursing setting, with the emphasis being on a shared professional understanding, which fostered a non-judgemental research forum on which staff could express their experiences safely. A more balanced power dynamic between the researcher and the researched led to data and, subsequently, items on the scales that more accurately represented the views of services users and staff.

Background

In the UK and across the world, mental health hospitals have seen nursing staff shortages, bed shortages and an increase in complex presentations, especially in urban areas of high demand.22–24 Indeed, mental health nurses perceive these issues as barriers to change in acute ward settings as well as poor/indeterminate leadership and incidents.25,26 The effects of current working practices and the ward environment on the well-being of both staff and service users should be explored in more detail because an untherapeutic environment may negatively affect service user satisfaction and staff morale, may have a direct impact on the quality of the interactions between service users and staff and, moreover, may be detrimental to the overall quality of care.27

Dissatisfaction with adult acute inpatient care is not a new issue and is well documented both in Britain and internationally. Negative service user reports of acute inpatient settings have emerged, describing the experience of hospital care as non-therapeutic and coercive.28–30 Limited interaction between staff and service users is commonly reported and users express a need for good interpersonal relationships and support that is sensitive to individual needs.31–33 Poor levels of involvement and a lack of information associated with medication, care and treatment have also been identified.34 On many wards, there is little organised activity and service users experience intense boredom.10 Security is of particular concern as many service users feel that they are not treated with respect or dignity, have significant safety concerns and report high levels of verbal and physical violence.10

There is a rather sparse literature written by service user researchers themselves on acute inpatient settings, but it is important because information given to academic or clinical researchers may be influenced by the differential status of the two parties. Both Walsh and Boyle34 and Rose19 report that service users experience low levels of involvement and a lack of information about their care and treatment. This mirrors the nursing literature. Rose’s19 participants reported a lack of activities leading to crushing boredom and, like Mind’s Ward Watch campaign, patients reported feeling unsafe.10

Alison Faulkner, a service user-researcher who spent time in acute wards, combines her systemic knowledge of mental health services with her own experience and that she has witnessed of others.35,36 Drawing on the grey literature,30 she confirms the ‘shabbiness’ of wards, the intense boredom felt by the inpatients and the ‘petty’ rules and regulations.35 She further concludes that in the inpatient wards she has frequented she was treated as ‘less than human’ and witnessed other patients also being treated as if they were a different species.

Studies in which the researcher was a user37,38 found that what was most important to service users was the quality of the therapeutic relationship. This is entirely in line with both UK policy and some of the nursing literature. Interestingly, good therapeutic relationships were prized whether the user was in a traditional acute ward or an alternative style of service, such as a crisis house.

Turning to the work environment for nurses, a growing body of literature links the stress of working on a ward to low morale, often measured as ‘burnout’ and poor job satisfaction.39–41 Stress, burnout and lack of social support have been described as symptoms of a negative work environment.26,42–44 Stressors such as high caseloads of patients, high volume of work, violence on wards45 and management issues such as poor leadership and low staffing levels42,46,47 have been linked to low morale. Different staff groups also seem to respond differently to work-related stressors. Jenkins and Elliott43 reported differences in perceptions of work-related stressors according to the occupational status of nursing staff. In their study, qualified staff cited poor staffing levels as the main stressor, while nursing assistants reported the main stressor as difficult interactions with distressed clients. Burnout was common to both groups. Cushway et al.48 noted that occupational stress led to poorer mental health outcomes in male nursing staff than in females, perhaps because male staff are more likely to respond to violent situations. Burnout has also been described as affecting job performance and contributing to poor-quality care.46,47,49 Both low morale and low job satisfaction result in poor staff retention, which again affects care quality.50

The methods through which organisational outcomes are decided and the effects of these methods on the perceptions of the workforce may also be implicated in this negative trend. Organisational change in the NHS is often imposed via a top-down approach, which may not take into consideration the views of nursing staff. Indeed, change process issues, such as poor involvement in planning, implementation and control of a project, have been highlighted as potential barriers to success.51 An inequitable stance to decision-making, perhaps as a result of poor staff involvement, may shape attitudes to planned changes if staff feel such changes are unfair to them or do not meet the needs of their client group.52 This argument is supported by evidence from both the nursing and wider literature, which shows that staff who are in leadership roles may view changes more optimistically than staff who are involved in direct care with service users because they form part of the consultation process for the proposed changes.53,54 The emotional response of staff to proposed changes may be an important barrier to the success of implementation.

Therefore, we propose a model where negative staff perceptions of the daily pressures of working on an acute ward and negative staff perceptions of barriers to change lead to burnout and low job satisfaction, and this affects the quality of care delivered. However, in order to test this model, the specific pressures for mental health nursing staff in maintaining a therapeutic environment and high-quality client relationships and the quality of care as perceived by service users must first be given more detailed consideration because, to date, these constructs remain underexplored. This is perhaps because the ward milieu is responsive to continuous organisational and social changes, that is, the client group and staff mix change on a daily basis and so this is a complex construct to measure. This model was considered in an add-on study carried out by a nurse researcher as part of her PhD and is presented in the appendix (see Appendix 3).

Objective measures of inpatient care, reviewed by Sharac et al.,15 have many drawbacks. They neither capture the complexity of ward dynamics nor reflect the quality of inpatient care. Measures of perceptions have an important role to play in evaluating complex situations because they allow for objective stressors and their appraisal by subjects within the social environment, as well as linking cognitions with affective and behavioural components.55

Recently there has been a focus on patient-reported outcome measures (PROMs) as a measure of quality and appropriateness of services and therapies. Despite service user involvement being considered an essential element in improving mental health services,56 PROMs are rarely developed using an inclusive methodology and research suggests user dissatisfaction with many outcome measures currently in use.57 Service users can often have different perspectives to professionals and can provide insight into how services and treatments feel.58 Redefining outcomes according to users’ priorities can help to make greater sense of clinical research and develop a more valid evidence base.36,59 Studies comparing the impact of traditional and user researchers in conducting research show some differences in qualitative data analysis60 but none on quantitative research findings.17,61 Given this, we believe that research methodologies should aim to be as inclusive as possible.

There are also some staff measures in existence that address attitudes to occupational stress (e.g. The Mental Health Professionals Stress Scale48 and occupational stress indicator62) that highlight staff reactions to organisational pressures and team and client relationships. The well-known Daily Hassles Scale63 has also been used to appraise these stressors. However, these measures neither adequately explore whether staff view daily pressures as having an impact on the therapeutic milieu (and, therefore, on the quality of the therapeutic relationship) nor have they been developed using qualitative data generated by staff working within those areas.

In nursing studies, there has been limited focus on assessing how social, emotional and psychological barriers affect the implementation of new innovations. The relationship between psychological well-being and the acute ward setting in relation to planned changes may, in nursing, be distinct from other professions because nurses remain on the ward for the duration of their shift and are often expected to play a key role in delivering changes. There are several measures developed in health care that focus on the uptake of evidence-based practice in clinical areas by mental health providers and nurses.64–71 However, there are no measures that focus on general changes in a mental health setting.

What is needed in the literature on acute care are psychometrically robust, brief, self-report measures reflecting service users’ experiences of receiving care and staff experiences of delivering/innovating that care. Measures of this type would allow a clear assessment of changes to inpatient care following specific interventions to improve the environment and therapy provided. Our study was designed to generate such measures.

The purpose of this section is to:

• describe a model of service user and nurse involvement in developing two psychometrically sound measures. The service user measure captures service user perceptions of acute inpatient wards (VOICE) and the nurse measures capture staff perceptions of the daily pressures working on an acute ward (VOTE)
• provide a secondary qualitative analysis of the data from VOICE and VOTE.

Methods

This study was designed to develop and test two self-report measures. The measures were generated through a process of stakeholder (service user and nursing staff) involvement.17,19–21,59 This approach directly involved users of the mental health services and nursing staff in order to ensure measures were produced that captured an accurate picture of an acute care ward from both perspectives. Following the measure generation phase that used qualitative methods, prototypes of the new measures were assessed for their content validity, acceptability and feasibility prior to psychometric testing on two separate large samples of inpatient nurses and service users. Ethics approval was awarded by Bexley and Greenwich NHS Research Ethics Committee (07/H0809/49) for the study to be carried out in four boroughs within a London NHS Mental Health Trust.

Focus groups were conducted to generate data for VOICE and VOTE.

Phase 1: mapping out the dimensions of inpatient care

In this initial phase, three stakeholder/reference groups were held to map out the dimensions of inpatient care as a topic guide. In developing VOICE, the stakeholders were service users with experience of research and inpatient care who were not currently using local services. National voluntary organisations were also represented. In developing VOTE, the stakeholders consisted of senior nursing staff. Each group built a topic guide that was flexible in order to encourage as creative a list as possible. Members of these two groups also then constituted the reference groups for the user-led and staff parts of the study.

Phase 2: measure generation

For both VOICE and VOTE, purposive sampling was used to reflect local inpatient demographics with regard to gender, age and ethnicity when recruiting participants who were either staff or service users of an inner London UK mental health trust.

VOICE sample

The sample consisted of people who had used inpatient services within the previous 2 years. One of the groups was specifically for participants who had been detained under the Mental Health Act (2007)72 as it was anticipated that they may have had different experiences.

VOTE samples

Nursing staff from all grades (health-care assistants, entry-level qualified staff, clinical charge nurses and team leaders) were asked to participate in the development of VOTE. They were purposively sampled from acute inpatient mental health wards in an inner London UK mental health trust.

Focus groups

A pilot focus group was used to test the items on the topic guide. Then four two-part focus groups were run to enable respondent validation. This meant that the data from the first focus groups were analysed and results presented back to respondents in the second wave of focus groups to see if they agreed with the analyses or wished to amend, add to or change them. The aim was to ensure that the analyses were an accurate representation of the participants’ views.

The service user and nurse researchers then refined the thematic analyses and constructed the draft measure.

Expert panels

As a penultimate step, two expert panels met to discuss the design of the new measures and to inform the ‘instructions for use’. For the service users, one expert panel was made up of focus group members and the second was independent. Changes were made to the items, the layout of the measure and the language on the basis of this feedback.

Final consultation

The draft measure was finally presented back to the original reference group for their comments.

Psychometric assessment

The framework for psychometric assessment was that reported for the Health Technology Assessment on adequacy of outcome measures which used 10 criteria, including feasibility and acceptability as well as the usual categories of reliability and validity.73

Samples

Service user participants for the feasibility study were recruited from acute wards and psychiatric intensive care units (PICUs), and test–retest participants were engaged on acute wards. For the larger psychometric testing phase, participants were recruited from acute wards. Service users were eligible if they could provide informed consent and they had been present on the ward for at least 7 days during the 4-week data collection phase. All nursing staff from all grades (health-care assistants, entry-level qualified staff, clinical charge nurses and team leaders) were asked to participate in the psychometric testing of VOTE. All potential participants gave written informed consent following an explanation of the study.

Phase 3: feasibility and acceptability

The final measures were intended as self-report tools, so studies of feasibility and acceptability were conducted to evaluate the burden of administering and completing the measure. In the feasibility study, participants completed the measure including two additional questions assessing whether or not the measure was easy to complete and understand. Acceptability was assessed using two additional questions collected from participants. Two tests which indicate readability, the Flesch reading ease test (the recommended score is between 60% and 70%), were assessed.

Phase 4: psychometric assessment

Scoring

Scoring methods for each measure were decided at this stage.

Reliability

Test–retest reliability was assessed with participants who completed the measures on two occasions with an interval of 6–10 days. The test–retest reliability was assessed by Lin’s concordance correlation coefficient,74 kappa and proportion of maximum kappa75 to measure the level of agreement between total scores and individual item responses at times one and two.

Generally, kappa scores of 0.21–0.4 indicate fair agreement, scores of 0.41–0.60 indicate ‘moderate’ agreement and scores > 0.60 indicate substantial agreement.76 Lin’s concordance correlation coefficient can be interpreted similarly to a correlation coefficient.

Internal consistency was assessed using Cronbach’s alpha, which refers to the correlation between items on the scale. Low scores would suggest that the items are not contributing to the same latent construct, those with a score of ≥ 0.7 are considered acceptable.77

Face and content validity

Whether or not the new measures truly reflected the experiences of those receiving care (VOICE) and working (VOTE) in the acute inpatient services (face validity) and covered the full spectrum of views (content validity) was explored through participatory methodology during the instrument development phase.

Criterion validity

Criterion validity was assessed by testing hypothesised criteria against other relevant measures.

VOICE

We hypothesised that those with positive perceptions of acute inpatient services would also be satisfied with services. This was tested using the Service Satisfaction Scale – Residential Services Evaluation (SSS-RES).78 This is adapted from the Service Satisfaction Scale-30 (SSS-30),79 designed to evaluate residential services for people with serious mental illness. The original SSS-30 has been used in a variety of settings and demonstrates sound psychometric properties.80

VOTE

We hypothesised that those with negative perceptions of the daily ward pressures (VOTE) would have low levels of job satisfaction. This was tested using the Index of Work Satisfaction (IWS). This is a 44-item scale that can be totalled to produce a score that measures the level of job satisfaction for health professionals. Each item is answered on a seven-point Likert scale.81

Construct validity

The construct under study for each measure was explored using factor analysis, with both orthogonal rotation using varimax and oblique rotation using promax. The final solution was chosen from the scree plot and the factor structure was chosen from the analysis that produced the highest loading. This analysis followed preliminary checks on the strength of the pairwise correlations between items and Bartlett’s test of sphericity.82 The subscales produced were explored for internal consistency using Cronbach’s alpha.83

Exploratory analyses

VOICE: Do demographic characteristics affect service user perceptions of the quality of acute ward care?

We expected differences in views between service users from different populations and clinical settings. Using one-way analysis of variance (ANOVA), we assessed whether or not service users’ perceptions differed by borough, gender, ethnicity, age, diagnosis, admission and legal status. The majority of the analyses were exploratory. However, we had specific hypotheses relating to ethnicity and legal status, for which we expected poorer perceptions from participants who were compulsorily admitted and those from minority ethnic communities.

VOTE: Do demographic characteristics affect staff perceptions of the daily pressures of working on an acute ward?

A regression analysis with the VOTE total score as the dependent variable and all demographic characteristics as the key independent variables was carried out using random-effects regression modelling (clustering on ward) to take into account the multilevel nature of the data. The purpose of this model was to identify any significant demographic predictors of negative staff perceptions of the daily stressors in the working environment (VOTE). The demographic variables were length of employment (> 3 years or < 3 years), qualified nurse or not, education (degree level or not), ethnicity [white, or black or minority ethnicity (BME)], country of origin (UK or not), gender and age (median split at age 39 years).

Results

Phase 1: mapping out the dimensions of inpatient care

VOICE

The stakeholder group recommended five core topics:

1. the effect of acuity on service users
2. security issues
3. medication
4. the experience of admission and discharge
5. how ward routine is experienced.

VOTE

The stakeholder group recommended six core headings for the topic guide:

1. patient care
2. core interventions
3. teamworking
4. change
5. safety
6. ethical issues.

Phase 2: measure generation

VOICE sample

As Table 1 shows, a total of 397 participants were recruited for the study: 37 for the measure generation phase and 360 for the feasibility study and psychometric testing. Schizophrenia and schizophrenia spectrum disorders were the most frequent diagnoses for both groups and approximately half of each set of participants was from the BME community. In the measure development phase, 43% of participants were men and the median age was 45 (range 20–66). In the psychometric phase, 60% of participants were men and the mean age was 40 (range 18–75).

VOTE sample

A total of 376 nursing staff participated in the measure development (Table 2). The majority of nurses were qualified nursing staff, while approximately one-third were health-care assistants. In the psychometric testing phase, 47% of participants were women and the median age was 39 years (range 20–67 years).

Thematic analysis of the full data set resulted in an initial bank of 34 items, which were formed into brief statements and grouped into domains. A six-point Likert scale was chosen, ranging from ‘strongly agree’ to ‘strongly disagree’ and optional free-text sections were included to capture additional qualitative data. The items were unweighted and one question was reverse scored. The self-report measure was designed to provide a final total score, with a higher score indicating a more negative perception. The inter-rater reliability of the focus group data coding was assessed using an NVivo 7 (QSR International, Warrington, UK) coding comparison report and showed between 97% and 99% agreement. Item reduction based on relevance and preventing duplication produced 22 items.

The pilot study, which comprised a mixed group of health-care assistants and qualified staff, agreed that these topics were broad enough to allow flexibility in bringing in new ideas. Interim analysis of the pilot focus group suggested that occupational seniority might interfere with a full discussion so, in order to allow for the maximum emergence of key themes, one ‘health-care assistant only’ group was included.

Thematic analysis revealed that the staff participants found the following core themes the most important for inclusion on the measure: teamworking, patient care, core interventions, safety, bed management and continuing professional development. These themes provided the structure for the measure. The individual items were developed from sub-themes within these broader domains. A five-point Likert scale was reviewed at the expert panel stage, but those staff participants felt that a wider range of response options would provide more scope and reduce omissions. At this stage there were 26 items.

Themes from the focus groups

The main themes identified in the focus group data were staff/patient interaction and coercion. Often, staff and service users conceived these differently.

Staff and service users’ interactions on acute wards

Service users

In terms of the focus of group discussions, the overwhelming perception of service users was that the ward was ‘untherapeutic’. A crucial contributor was the lack of available staff and helpful staff. For patients, it appears that staff are ‘stuck in the office’ and cannot help even when the situation is urgent, the situation is escalating or the service user is in crisis. This lack of availability was compounded by the use of ‘bank’ or ‘agency’ staff who were brought onto the ward in times of staff shortage but were usually unfamiliar with the ward and did not know the patients. They were described as ‘sitting and staring’ and not prepared to have a conversation with users or even ask them how they were feeling.

In the UK, every service user should have a care plan and documented notes on meeting that care plan. In fact, this is the main form of ‘structured interaction’ that is meant to take place on wards. But even care planning was low in reference density in the data. One of our participants had been given his discharge summary which included this documentation. It implied that he had one-to-one conversations with a named nurse every day throughout his stay; however, this was not his memory of events.

Many of the service users understood that work pressures such as under-staffing, paperwork and work-related stress impacted on staff’s lack of interaction. Nevertheless, this resulted in an untherapeutic climate where emotional distress was suppressed rather than explored.

It is evident in the accounts that routine and therapeutic interaction was rare in ward life. Participants were more positive about talking therapies, although they gave instances of them being unhelpful, especially groups in which everyone, including the facilitator, sat mute for an hour. Although talking therapies were described as being helpful, very few people had actual experience of such therapy while on the ward. They reported in prospect that talking therapies would be a good way to deal with distress. One participant had observed a suicidal fellow patient and thought that if there had been group therapy available she might have got better. Another favourably contrasted talking therapies with the ever-present use of medication.

Nevertheless, there were instances when the service user recognised that the ward had done them some good or, at least, were ambivalent about it.

Nurses

The nurse participants were aware that the care they were providing was less than perfect and that they did not spend enough time with patients. They perceived both administrative tasks and bed management as a barrier to interaction and the ‘pressure’ of these daily tasks was very frequently referenced. Instability as a result of both the distress of individual service users and the overall ward milieu were of concern.

Bed management, which involved early discharge, making space and moving clients around, provoked a strong emotional response among nurse participants as they knew it led to poor interactions with patients. The practice of ‘sleeping out’ when a patient spent the day on the ward and then was sent to another ward, or even a bed and breakfast, to sleep evoked angry responses. Using periods of leave to free up beds also provoked feelings of guilt and anxiety alongside the view that senior management did not care.

Nurses had complex emotional responses to the fact that they could not spend enough time with service users. They expressed frustration and concern that they were becoming deskilled. Anxiety, arising from the lack of interaction, was a common response. Explicitly, staff remained committed to the need for interaction despite the daily pressures of working in an acute setting. Implicitly, there was an underlying theme of avoidance or interaction anxiety which suggested that many staff had withdrawn from interaction with the service users. The main cause of this ‘burnout’ appeared to arise from limited internal coping skills and from the need for staff to protect themselves emotionally from the complexities of individual service users in their care.

Coercion and control

As a barrier to therapeutic interaction and within the wider theme of violence, we will illustrate how service users and staff perceive their own power to be manifested on the ward. The two groups have differing perspectives: users feel coerced, whereas staff feel they are delivering a legitimate response to violence.

Service users

For service users, coercion is a complex concept. Here we focus on control, restraint and forced medication (rapid tranquillisation), which had high reference density in the data and our patient participants devoted much time to this theme. Medication was the main (or sole) treatment but here we are concerned with medication that is given by force. Other countries use mechanical restraint but this is not deployed in the UK.

One of the main reasons given by patient participants for behaviour that might elicit restraint or forced medication was that users were cooped up in the ward and not allowed to go outside and get fresh air. Even those granted time off the ward would often find themselves without a suitable escort. Others put it more strongly, likening the environment to a prison or a cage for an animal. This, they said, provoked extreme frustration and anger which was responded to by nurses in a way they thought aggressive and unnecessary. Our participants conceptualised forced medication as violence and had a whole vernacular to describe it: ‘jump on you with the needle’ or ‘pound on you with the needle’. However, they did not see their own behaviour as unnatural. For these participants, forced medication was not medical treatment but control.

Coercion can be experienced in more subtle ways and examples were common in the data. Medication does not always have to be given forcibly, there can be pressure to take medication under the threat that if the patient does not then it will be given by force.

There is a dichotomy in our data. Inpatient wards were routinely described as unsafe, fearful places where users felt unprotected by staff. Perceptions of coercion regarding control, restraint and medication resulted mainly from ‘being done to’. However, within the context of an often chaotic ward, coercion was sometimes perceived as an appropriate staff response to other violent service users.

Nurses

Both service users and nurses reported a sense of ‘them and us’ on the ward. In the management of violence, nurses used interventions that were perceived as coercive by service users. However, nurses expressed some awareness and sympathy towards service users around this issue and referred to the therapeutic relationship as a protective factor in the management of violence.

At the same time, it was clear that control was not absolutely one sided in favour of nurses. They reported that violence as well as intimidation was a tool that service users could use to express their resentment at their situation on the ward. The lack of support felt by nurses in the face of violence was strongly expressed in the data. This is despite the provider trust having a ‘zero tolerance’ policy with respect to abuse, which would imply that police intervention was sometimes warranted. Nonetheless, many felt that they had to cope alone.

There is no doubt that some nurse participants had experienced very serious incidents. As well as physical assault, two nurses had been taken hostage and locked in the nursing office for many hours. Attempts to have the police intervene were of no avail as the police saw it as a ‘mental health issue’. We asked the nurses about logging incidents and their response was that they logged only the most serious ones otherwise they would spend the whole day filling out incident reports.

Official documents from the Royal College of Psychiatrists and the Royal College of Nursing (RCN) in the UK emphasise that the initial response to an imminent incident should be one of de-escalation where the staff member tries to ‘talk down’ the patient.86 The reference density of de-escalation in our data was tiny with only two chunks of text coded. One person tried to explain why de-escalation was impossible owing to lack of staff; however, there was one instance where de-escalation was highlighted as best practice.

The perceptions of users, nurses and doctors on the issues of managing aggression and violence have been compared.87,88 Staff attributed some violent incidents to factors ‘internal’ to service users, such as their illness or demographic characteristics. Service users, on the other hand, found responses to ‘incidents’ to be unsympathetic, containing and controlling. This is consistent with other findings.89

Expert panels

VOTE

The expert panels (n = 13) confirmed that the measure allowed a good range of staff perceptions of acute inpatient wards to be expressed. A five-point Likert scale was reviewed at the expert panel stage, but those staff participants felt that a wider range of response options would provide more scope and reduce omissions. At this stage there were 26 items.

VOICE

The expert panels considered the measure to be an appropriate length and breadth and, following some minor changes in wording, concluded that the measure was appropriate for use by service users in hospital.

Final consultation

The service user reference group requested that the wording of several items be clarified, that all statements should be made positive, that staff responses to emergencies and personal crises should be differentiated and that the measure should not begin with items about the staff. There were no changes to the nurses’ measure as a result of the final consultation.

Psychometric assessment

VOICE: feasibility and acceptability

Feasibility took place in two waves (n = 40 and n = 106). In the first wave, 98% of participants found the measure both easy to understand and complete, and in the second wave, 82% of participants considered the measure to be an appropriate length. Two participants (2%) disliked completing the measure and six (6%) found some of the questions upsetting. VOICE took between 5 and 15 minutes to complete and was easy to administer. The measure was found to be suitable for completion by participants with a range of diagnoses and at levels of acute illness found on inpatient units. The Flesch reading ease score was 78.8 (reading ages 9 and 10 years) indicating the measure was easy to understand.90 Following the feasibility study, one item was removed as it was considered to be a duplicate. This left the measure with 21 items.

VOTE: feasibility and acceptability

The feasibility study (n = 40) revealed that 95% of staff agreed that VOTE was easy to complete and easy to understand. Generally, staff found that, with minimal explanation, the measure could be completed by self-report in approximately 15 minutes. Items identified as having confused phrasing were changed and those showing poor consistency (Cronbach’s alpha) or poor variability or poor reliability were dropped, leaving 20 items.

The acceptability study showed that 76% of staff thought that the length was about right and 91% that it was enjoyable and not upsetting. The Flesch reading ease score for the measure was 64%. The Flesch–Kincaid91 grade level score was 7.6, which means that the measure can be read by a 12-year-old who is of average ability.

Phase 4: psychometric assessment

VOICE scoring

Total scores were calculated by totalling all items with no missing data. For participants responding to at least 80% of the items, a pro-rated score was calculated. Less than an 80% response was considered as a missing total VOICE score. Negatively phrased items were reverse scored so that high scores indicated a more negative perception of the quality of care.

VOTE scoring

Total scores were calculated by totalling all items with no missing data. For participants responding to at least 80% of the items, a pro-rated score was calculated. Less than an 80% response was considered as a missing total VOTE score. Negatively phrased items were reverse scored so that higher scores indicated a more negative perception of the ward. When comparisons to other measures were made, the same rule was applied to their scoring, that is, high scores indicate poorer satisfaction.

VOICE reliability

The test–retest reliability (n = 40) was high [ρ 0.88, 95% confidence interval (CI) 0.81 to 0.95] and there was no difference in scores between the two assessments. A total of 192 participants had complete data on the VOICE scale and were used in assessing the internal consistency. After removing items with poor reliability, this left a 19-item scale with high internal consistency (α = 0.92).

VOTE reliability

Group 2 (n = 43) participated in the test–retest study using the final 20-item measure. As items tended to be skewed towards ‘agree’ or ‘disagree’, the kappa coefficient is expressed as a proportion of the maximum possible value. Six items showed substantial agreement (kappa maximum ranged from 0.60 to 0.73). Moderate agreement was shown in 14 items with kappa maximum ranging between 0.41 and 0.59. These results indicated moderate and substantial reliability. Concordance between the total scores (n = 34) was good (total score: ρ 0.77, 95% CI 0.65 to 0.89). The internal consistency of the measure, assessed using data from group 3 (n = 200), was good with the overall alpha at 0.82.

VOICE validity

The measure has high face validity. The wide range of items was determined by service users during the focus groups and the measure reflected the domains that they considered most important. The feasibility study participants felt that the measure was comprehensive and, therefore, had high content validity. Pearson’s correlation coefficient showed a significant association between the total scores on VOICE and the SSS-RES (r = 0.82; p < 0.001), indicating high criterion validity.

The data were suitable for factor analysis for construct validity [Kaiser–Meyer–Olkin measure was high (0.9), Bartlett’s Test of Sphericity was significant (p < 0.001)]. The exploratory factor analysis was conducted on 192 fully completed responses to the 19-item measure. A promax rotation, allowing the subscales to correlate, revealed two factors (Table 3) that accounted for 95% of the total variance. The item groupings indicated by the factors were used to determine subscales by summing the scores of the items. Items with a factor loading of > 0.4 were considered to load significantly. When items loaded onto both subscales, a decision was made to include them on the subscale with the closest conceptual fit. Subscale 1 was labelled security and subscale 2 was labelled care. Both factors were significantly correlated (r = 0.73; p < 0.001). One item (‘Staff give me medication instead of talking to me’) did not load highly onto either scale and so was not included in the subscales.

VOTE validity

A high level of staff involvement throughout the process of measure development ensured good face and content validity. This was achieved because staff participants provided feedback on the content of the themes arising from the qualitative data and on the language used in the item generation phase. Staff agreed that the results did capture what they had reported. The use of a flexible topic guide/interview schedule maximised exploration of the construct under study and minimised omissions in the data set. A t-test examining whether or not VOTE scores were associated with an a priori criterion of job satisfaction, as measured by the IWS, showed that those with negative perceptions of the daily pressures of working on an acute ward also had poor job satisfaction [t(157) = –10.34; p = 0.001 (n = 159)]. The mean VOTE score in the low job satisfaction group was 76.5 [standard deviation (SD) 10.4] and the mean VOTE score in high job satisfaction group was 60.4 (SD 10.1). The VOTE measure was strongly and significantly correlated with the IWS [r = 0.77; p < 0.001 (n = 159)]. Overall, the mean satisfaction of the group was 163.6 (SD 31.3).

An exploratory factor analysis for construct validity was carried out (n = 200). The Kaiser–Meyer–Olkin measure of sampling adequacy was good (0.8) and Bartlett’s test of sphericity was significant (p ≤ 0.001). The scree plot92 indicated three factors with an eigenvalue of > 1. The orthogonal rotation produced the most coherent solution because it had consistently higher factor loadings and this is presented in Table 4.

Factor 1 reflected staff perceptions of ‘workload intensity’ and appeared to be tapping issues relating to work pressures, administrative intensity and feeling unsupported (either through weak leadership or inadequate staffing). Factor 2 related to ‘team dynamics’ linking concepts such as staff confidence in their leadership and in the dynamics of their team. Factor 3 identified ‘interaction anxiety’, associated with powerlessness and a lack of confidence in the uncertain ward situation. Evaluations of the internal consistency of the subscales: factor 1 ‘workload intensity’ (Cronbach’s alpha 0.74), factor 2 ‘team dynamics’ (Cronbach’s alpha 0.75) and factor 3 ‘interaction anxiety’ (Cronbach’s alpha 0.60).

Exploratory analyses: how effective are the new measures in discriminating between groups?

VOICE: do demographic characteristics affect service user perceptions of the quality of care on an acute ward?

The ability of VOICE to discriminate between groups is indicated in Table 5. Bivariate analyses showed significant differences for legal status. Participants who had been compulsorily admitted had significantly worse perceptions (t = –3.82; p < 0.001). A multivariate analysis revealed that legal status remains significant even when adjusted for the other factors (p = 0.001).

We assessed any independent demographic predictors of VOTE using random-effects modelling and a forward selection procedure with the level of significance set at 0.05. The results showed that one variable was a significant predictor: country of birth (n = 180, groups = 17).

Staff who were born in countries outside the UK had more positive perceptions of the daily pressures of working on an acute ward than those who were UK born (Tables 6 and 7).

The final measures are available are in Appendices 7 and 8.

Discussion

Using a participatory methodology, we have developed both service user- and staff-generated, self-report measures of perceptions of acute care. VOICE encompasses the issues that service users consider most important, namely their care and treatment and feelings of security, and is suitable for use in research settings. VOTE identifies the daily hassles of staff that affect staff engagement with service users, colleagues and their professional identity, which are all areas that are important to stakeholders. Both measures have promising psychometric properties. In both measures, the internal consistency and criterion validity are high and test–retest data show stability over time. The full involvement of service users and staff throughout the development of each measure has ensured that VOICE and VOTE have good face and content validity and are accessible to the intended recipients.

Can VOICE and VOTE distinguish differences in views?

Detained participants were shown to hold more negative perceptions of inpatient services. This is similar to the satisfaction studies showing that service users who are admitted involuntarily are less satisfied with their care.93 More recently, research has identified that lower levels of satisfaction are linked with the accumulation of coercive events and perceived coercion.94,95 Our findings from the focus groups present a more complex picture and one that is worth further analysis.

We anticipated, but did not find, any differences on either VOICE or SSS-RES scores for ethnicity. Previous quantitative studies have shown differences for legal status but not ethnicity,96,97 whereas qualitative research has revealed that BME users hold relatively poor perceptions of acute care.16,98 Our study was set in areas of London with high proportions of people from BME groups.99,100 Staff demographics tended to mirror those of inpatients and it may be that services were better tailored towards BME groups. In addition, asking users for their perceptions while they were in hospital may have inhibited openness and honesty, particularly on sensitive issues.

In this study, staff had slightly more positive perceptions of job satisfaction than staff in other studies.101–103 Staff with negative perceptions of the daily pressures of working on an acute ward and barriers to change also had poor job satisfaction.

In previous studies, work stressors have differed between groups including occupational status and gender.43,48 Of all the demographic characteristics included in this study, only country of birth showed a significant effect on staff perceptions of the daily pressures of acute ward working, with staff born outside the UK showing more positive perceptions than those born within the UK. This may be a direct result of the inner London location of the trust under study, as a large percentage of the client group and the nursing workforce are not UK born.

However, the mean VOTE scores were most negative overall in the direct care staff groups (staff nurses/nursing assistants) compared with more positive scores in those who occupied more managerial roles (clinical charge nurses and team leaders). Although we did not address whether or not the stressors differed significantly between these groups, it is true that both nursing assistants and staff nurses spend the most time in direct client contact. Therefore, their more negative perceptions might be one of the inevitable aspects of delivering therapeutic acute care for patients with severe mental illness at their most distressed. Furthermore, staff in leadership roles are more likely to be involved in the planning stages of new changes and, therefore, have an increased sense of control and responsibility over them, while those in direct care roles are more likely to be involved in delivering changes. These important issues, revealed through stakeholder involvement, require further exploration to discover key drivers for these perceptions that might then be subjected to management and other interventions.

Are VOICE and VOTE different from other measures?

The benefits of the participatory model used to develop VOICE and VOTE are visible in both the breadth of the construct investigated and the rich content of the items. Although the VOICE total scores were correlated, there were also distinct differences in content between VOICE and the comparison satisfaction measure. We believe that this is because of the use of a participatory methodology. The factor analysis of VOICE discovered two main factors: security and care. In particular, the safety and security issues were given more weight in VOICE and items on diversity were included that did not appear in the conventionally generated measure. Although items regarding the physical environment and office procedures featured in the SSS-RES,78 these issues were not deemed as important by the service users in our study and, therefore, were not included in VOICE. However, other items referred to ‘care received’ overlapped with some items in SSS-RES.

It is often assumed that the only construct to measure is satisfaction with acute care. However, there are difficulties in encapsulating complex sets of beliefs, expectations and evaluations in satisfaction measures. Caution should be taken when making inferences from the results of such measures, as they may not accurately reflect the views of service users.104 VOICE is unique in that it captures service users’ perceptions and we anticipate that this will depict the inpatient experience more accurately.

The factor analysis conducted during the development of VOTE revealed three novel factors. Factor 1, ‘workload intensity’, seems consistent with previous work identifying the effects of poor resources such as inadequate staffing and too few beds on occupational stress for nurses.9 Factor 2, ‘teamwork’, is similar in content to the concept of ‘job satisfaction’ characterised by leadership, educational attainment, pay and stress. Factor 3 represents a new subscale that explains the lower correlations to the IWS and Maslach Burnout Inventory (MBI) subscales. It crystallises two ideas. First, it reflects some issues relating to burnout.46 Second, it draws together some of the feelings of fear and uncertainty that are associated with working with distressed clients in an unstable environment. This complex issue was revealed through stakeholder involvement and adds a new and unique aspect to the current library of measures that capture staff perceptions of stressors in acute inpatient settings. This issue clearly requires further exploration in the field, as one of the fundamental aspects of delivering therapeutic acute care is in the ability to work with clients with severe mental illness at their most distressed.

Further developments

Three further measures grew out of this WP; please refer to Appendices 1–3.

Strengths and limitations

This study was conducted in London boroughs with high levels of deprivation, ethnic minority populations and psychiatric morbidity99,100 and, therefore, may not be directly generalisable to other settings. In addition, our focus groups and expert panels included a high proportion of participants from BME communities. Although this is a strength, it may be that different items would have been produced by other groups. Furthermore, as recruitment for VOTE was limited to staff from acute wards it is not clear whether or not VOTE would be suitable for other settings such as forensic or rehabilitation wards.

It is impossible to accurately assess inpatient care without involving the people directly affected by that service. Developing outcome measures that are valued by service users and staff is essential in evaluating and developing inpatient services. The main strength of this piece of research is that it fully exploits a participatory methodology: staff and service users were involved in a collaborative way throughout the whole research process. VOICE and VOTE are the only robust measures of acute inpatient services designed in such a way. This method has resulted in measures that encompass the issues that staff and service users consider most important and that are acceptable, easy to understand and complete by a diverse staff group and service users with a range of diagnoses and severity of illness.

Conclusion

The study has demonstrated that a participatory methodology can generate items that are prioritised by staff and users, but not included in traditionally developed measures. VOICE is the first service user-generated and psychometrically robust measure of perceptions of acute care. It directly reflects the experiences and perceptions of service users in acute settings and, as such, is a valuable addition to the PROMs library.

VOTE can be used to promote staff involvement by identifying staff-valued work pressures that might affect burnout and interfere with quality of care, as well as being used to highlight the impact of service changes in acute wards. In future research, the effects of the ward environment on the workforce might be evaluated through an exploration of staff perceptions. Whether or not VOTE can also be linked to environmental factors such as incidents, staffing levels, and the amount of time that staff spend engaged in therapeutic interaction with clients should be explored. VOTE might be a useful tool for staff to assess the feasibility of delivering new interventions and exploring whether or not they can be sustained in the long term.

Clinical implications

VOICE and VOTE are measures of the experience of living and working on acute wards. This is consistent with recent National Institute for Health and Care Excellence (NICE) guidelines 2014.104,105 The factor structures demonstrated what was key in this experience for each group. Service users were concerned with the care provided and staff/patient interaction as well as security on the wards. Nurses were concerned with workload intensity, team dynamics and interaction anxiety. Although interaction anxiety could lead to an avoidance of service users and, therefore, compromised care, there was evidence from the post-intervention focus groups for WP3 that service users understood how pressured staff were and made allowances for it. These findings suggest that leaders in the NHS need to consider putting in place support structures for staff if care is to be optimised for service users. The 2013 Care Quality Commission (CQC) report106 paints a mostly bleak picture of inpatient care which corresponds to our findings and they call for similar measures. The RCN ran a consultation on control, restraint and rapid tranquilisation practices in 2013, which greatly exercised our focus group participants.

Research implications

It would be worth investigating more thoroughly the variation in VOICE and VOTE and what such variation might predict. Furthermore, a more detailed investigation could be undertaken for the WP4 data. We need to find out whether or not, within the triage system, the admission ward had different VOICE and VOTE configurations (total scores and individual factors) from the long-stay wards as well as looking at this over time. We also have test–retest data from a forensic ward and these could be further examined in terms of the VOICE and VOTE scores to investigate if there are differences between wards. Other ongoing research is examining whether or not VOICE scores are sensitive to a peer support intervention on acute wards.