Methodology

Box 6 provides a summary of Chapter 2 of this report.

Summary of key points

• This study was conducted across two stages. Stage 1 took the form of scoping work to better understand new and emerging digital technologies for social care and the challenges and lessons learnt from previous research and evaluation efforts. Stage 2 involved evaluation of the implementation of new and emerging digital technology, using the example of IndependencePlus, through qualitative data collection.
• The study has generated a number of lessons about the key things to consider when implementing new and emerging technology in social care.
• However, significant recruitment challenges were faced during stage 2 of the research, particularly related to the pressures of COVID-19 on social care, as well as a range of other factors. This had significant implications for our research, and is discussed in detail at the end of this chapter.

To address the research questions outlined in Chapters 1 and 3, we undertook two stages of data collection, each of which is outlined in more detail below:

• Stage 1: Scoping work to better understand new and emerging digital technologies for social care, with a focus on home sensors with AI technology, and the challenges and lessons learnt from previous research and evaluation efforts. This consisted of a rapid review of the literature, nine key informant interviews, three online project design groups and selection of potential case study sites.
• Stage 2: An evaluation of new and emerging digital technology, using the example of IndependencePlus, through qualitative data collection and analysis from three case study sites (20 interviews), supplemented by three interviews and focus groups with care technology providers/innovators and regulatory bodies. The work in case study sites consisted of selecting and recruiting local authorities and care providers who had been exploring the potential of new and emerging digital technology, interviewing key stakeholders (seeking to include decision-makers and operational leads, care staff, carers and people who draw on care and support), reviewing documents from case study sites and, finally, analysis and synthesis. This was supplemented by interviews with technology providers/developers, and interviews with national organisations involved in developing policy for and regulating AI in social care. We had originally intended to hold local workshops with case study sites to sense-check emerging findings, but due to recruitment and engagement challenges (described later), this was not possible during the initial life of the project. However, we will offer this to our sites later in 2022 in case circumstances have changed sufficiently for this to be a helpful contribution.

Each of these stages will be outlined in further detail below. However, we encountered significant challenges in recruiting case study sites and participants to interview (partly due to the impact of COVID, but also due to the nature of the pilots themselves, which were all deemed not to have worked and were subsequently abandoned). We reflect on this at the end of the chapter.

Although we were interested in new and emerging digital technology generally, we collected data from sites that had chosen to use a specific type of technology – home sensors with AI technology –provided by a single technology company. This form of AI technology had been identified via the NIHR-funded prioritisation exercise as worthy of rapid evaluation (see below for further discussion), with various examples of new and emerging technology cited. In this report, we have anonymised the technology provider and sites in order to maximise learning about the decision-making and implementation process more generally, rather than to explore the effectiveness of a single technology or product. We therefore refer to this technology and the company providing it using the pseudonym ‘IndependencePlus’. All case study sites were also anonymised to guard against reputational risk in situations where local authorities or care providers have experimented with new ways of working and not been able to deliver desired outcomes – but still generated important learning that could benefit others. During our scoping work, a range of stakeholders felt that this was a helpful safeguard which would maximise their ability to share genuine learning, whether outcomes were perceived to be ‘positive’ or ‘negative’.

Stage 1: Scoping work

To inform our study design, the research team undertook scoping work to better understand new and emerging digital technologies for social care, with a focus on home sensors with AI technology. We conducted a rapid scan of the literature, nine key informant interviews and three project design groups with stakeholders, and selected our potential case study sites. Scoping work included an exploration of which questions we should ask stakeholders, which themes to consider, and how best to collect the data from care providers and commissioners, people who draw on care and support, carers and care staff. It also enabled us to explore the challenges and lessons learnt from previous research and evaluation efforts and develop our research questions. This work guided our approach to stage 2 of the work. The approach to the scoping work is outlined below, and the results are described in the following chapter.

Rapid scan of relevant evidence

As part of our scoping work, we conducted a rapid scan of international literature relating to the implementation of new and emerging digital technology within social care, as well as evidence from research and evaluations of home sensors with AI technology specifically. Unlike systematic reviews, rapid evidence scans do not involve systematic methods of searching or selecting articles and do not aim to produce an exhaustive or reproducible summary of a topic. Instead, this was a pragmatic scan of articles for the purpose of informing possible research questions and identifying tools and analytical frameworks for the study (to be developed and tested with our key informants and project design groups). It was carried out through Google searches (including Google Scholar), recommendations of articles from experts, and ‘snowballing’, where sources emerged through looking at references and citations of relevant sources.52

Although the search was not intended to be comprehensive, we used search terms encompassing the following areas:

• ‘social care’
• ‘AI’ OR ‘artificial intelligence’ OR ‘preventative sensor technology’ or ‘machine learning’ or
• ‘sensors’ or ‘assistive systems’ or ‘telecare’.

Due to the informal nature of this rapid scan of the evidence, we are unable to provide exact search strings, the number of articles reviewed or reflections on the quality of the evidence, as would be required in a more formal or systematic review.

Key informant interviews

In total, nine interviews were conducted over several months in 2020 on the phone or online with research experts in the field and with decision-makers and operational leads (i.e. staff involved in the implementation of AI sensor technologies) from local authorities and care providers. Interviewees from local authority commissioners and care providers were selected on the basis that they had used IndependencePlus. Informants with research expertise in this area were also interviewed and were identified from relevant literature and through networks established within the research team. Interviews were semistructured, allowing a flexible approach in the topics covered. The topic guides can be found in the project documentation.

Online project design groups

The research team held three online project design groups with stakeholders in October 2020. Design group 1 involved four local authority commissioners and care providers who were decision-makers and operational leads with experience of using IndependencePlus. Design group 2 involved four people who draw on care and support, and carers. Design group 3 involved three people who draw on care and support, and carers. Attendees of the second and third groups were from the BRACE Health and Care Panel and the University of Birmingham Social Work Service User and Carer Contributor group. To avoid exclusion on the basis of an individual’s access to and proficiency or confidence with digital technology, individuals were asked if they would prefer to speak over the phone with a member of the research team, rather than attend the design meeting. Furthermore, one carer who was interested in contributing but was unable to attend the dates of the online project design groups was offered an individual interview, which took place over the telephone in October 2020. In these online project design groups for both groups of stakeholders, we asked participants about the type of questions and themes they thought we should ask in the evaluation and enquired about the practicalities of collecting data (particularly given the COVID-19 pandemic). This was to help us develop the topic guides for data collection and select appropriate and feasible methods in our evaluation.

Identification of case study sites

As part of the scoping work for this study, the research team created a list of potential case study sites. These were all of the local authorities and care providers in the UK known to be using IndependencePlus. We developed this list through e-mailing and then speaking on the phone or online with the organisations listed on the IndependencePlus website, which listed all their clients. We were aware of eight sites using IndependencePlus and had communication with five of these (three others did not respond), all of which provisionally stated an interest and willingness to participate in this study. These were three local authorities (i.e. councils with responsibility for adult social services, who assess people’s needs/eligibility and commission care and support services) and two care providers (public, private or voluntary organisations commissioned to provide care). Our process of obtaining case study sites was therefore pragmatic, with no selection based on any characteristics other than them using IndependencePlus and being interested in participating. These case study sites, however, cover both local authorities and care providers, a mix of urban and rural areas, and different groups of people who draw on care and support (e.g. older and younger ages, and a range of mental and physical health conditions). While we did not assess how similar or different these providers or councils are from other organisations providing social care, there is no reason to believe they are atypical in any significant way.

Stage 2: Evaluation of new and emerging digital technology

After conducting the stage 1 scoping work, stage 2 of the study focused on evaluating the implementation of new and emerging digital technology, using the example of IndependencePlus, through qualitative data collection and analysis from three case study sites, supplemented by interviews and focus groups with care technology providers/innovators. This stage of work consisted of recruiting case study sites, interviews with key stakeholders, and review of documents, each of which will be discussed here.

Recruiting case study sites

We formally invited each of the five sites identified from the stage 1 scoping work (three local authorities and two service providers) to take part in the evaluation. However, two sites were subsequently unable to take part due to a culmination of different reasons described later in the limitations section of this chapter. Therefore, we worked with three case study sites (one care provider and two local authorities). Table 1 provides an overview of each of the three case study sites.

Coordination and communication with case study sites

The team had already begun building relationships with the five potential case study sites that had expressed an interest in participating during the scoping work, through e-mail and telephone contact with decision-makers and operational leads, and through a scoping workshop. For the three case study sites that agreed to participate, these relationships were strengthened by the allocation of a member of the research team to each case study site and the identification of a lead contact at each case study site. We had regular and clear communication with case study leads and other local contacts to ensure that the relationship remained strong throughout the study – especially in such a difficult external policy and practice context.

Interviews with key stakeholders

Recruitment

Decision-makers and operational leads were recruited through the links made with case study sites during the scoping work. Interviews were undertaken between April and December 2021 (this is longer than initially intended due to the impact of COVID). Following each interview with decision-makers and operational leads, the local leads were asked to identify front-line staff, people who draw on care and support and carers who had experience of using the sensor technology (even if very limited) and who might be interested in taking part in a subsequent interview. We did not want this to enable local senior stakeholders to pick and choose who we approached, but it was designed to ensure that we did not invite someone to take part in inappropriate circumstances (e.g. if an older person had recently died). Inclusion criteria for people who draw on care and support, carers and care staff were:

• having had the sensor technology set up in their home and having attempted to use it (for people who draw on care and support or their carers)
• having had experience of implementation and delivering care through the use of sensor technology (for care staff)
• having had experience of using the sensor technology for a range of social care needs (i.e. no particular user groups were excluded).

There were no limitations as to the perceived level of ‘success’ of the sensor technology used. We were interested even if the technology was abandoned, as we were keen to learn from a wide range of experiences of the use of technology (and suspected that there may be even greater learning for others in situations where the technology was not taken forward than in pilots that were felt to have had positive outcomes – albeit it can be difficult to get access to sites that perceive their innovation has not worked as they intended; see below for further discussion). In addition, there were no exclusion criteria related to the length of time for which the technology had been used, so that we could obtain varied experiences. The criteria for sampling people who draw on care and support were deliberately broad, to include anyone needing support with their daily living, and may include older people, people with dementia or mental health problems, people with physical impairments or learning disabilities, and family carers.

The local lead(s) passed information about the study to people who draw on care and support, carers and care staff, seeking permission for the research team to contact people interested in taking part or finding out more. Where permission was granted, a member of the team e-mailed or rang potential participants to describe the study and its aims in detail. Each participant returned a consent form prior to commencing the interview and was given 1–2 weeks to decide whether they would like to participate. Prior to commencing the interview, participants also had the opportunity to ask questions about the study and/or wider Birmingham, RAND and Cambridge Evaluation Centre (BRACE)-related work. It was made clear that participants could withdraw from the study at any time, without having to give a reason, and they were also given information about how to find out more about the study, or to raise concerns about its conduct.

At each of the case study sites, we aimed to conduct qualitative semistructured interviews with 2–3 decision-makers and operational leads, five staff with experience of working with IndependencePlus, and five people who draw on care and support or carers. However, we faced significant challenges in recruiting individuals to interview (for reasons described later in the limitations and caveats section). Partly due to this, the stakeholder groups were widened to include interviews with a series of technology providers/innovators supporting the development of care-related digital technology (this included an interview with IndependencePlus itself) as well as with respondents from national bodies which regulate care. The total number of interviews was 23 (with 24 individuals) and we were unable to secure any interviews with people who draw on care and support (see below for further discussion). One carer provided feedback in written form, which is included in the count of 23 interviews below. The breakdown of the interviews conducted is provided in Table 2.

Within this report, interviewees are referred to using a code encompassing their case study site or role (e.g. technology provider) and a unique identifier. For example, the first interviewee from case study site 1 is referred to using the code ‘CS1 P01’.

Interview conduct

Topic guides were developed for the interviews: one for decision-makers and operational leads, another for staff, and a third for people who draw on care and support and carers (see the project documentation for topic guides). The themes and questions covered in the topic guides were informed by the stage 1 scoping work.

The interviews invited decision-makers and operational leads to reflect on how decisions were made about the adoption of IndependencePlus; the process of implementation; early experiences of its use (including potential costs and savings); and key learning points that could help others considering using these types of digital technologies.

Care staff and carers were asked about their practice/lived experiences of the implementation process; practical realities of the technology use; the extent to which the technology had enabled, improved or hindered daily activities; the extent to which the technology had increased or compromised feelings of choice and control; and how this type of technology (or the implementation of it) may be improved.

Interviews with stakeholders lasted 30–60 minutes and were conducted online using Microsoft Teams^® or Zoom, with the option of telephone interviews also offered. Interviews were audio-recorded (subject to consent being given) and detailed notes were taken by the interviewer.

Review of case study and technology provider documents

Once an interview with a decision-maker or operational lead was completed, we asked whether they could identify any documents (not containing sensitive information) that could be shared with the research team. This was to help build a picture of the decision-making process, the implementation process and any challenges/successes. The documents shared by case study sites are outlined in Table 3. We reviewed these as background and contextual information.

Data analysis

Once we had completed the interviews, the team held a data workshop to discuss our provisional findings and identify themes.

Given the limited number of interviews per case study, and the similarity of themes arising from the interviews, the data from each case study were analysed together. Our analysis explored the desired outcomes of new and emerging technologies for social care; how these outcomes are expected to be achieved; and what resources, approaches and activities are supporting the implementation in practice. Thematic analysis of interview transcripts and documents provided by interviewees was conducted using guidance from Braun and Clarke.53 We iteratively developed a coding frame (see the project documentation) using early interviews and discussion among the study team.54 NVivo 12 qualitative analysis software was used for interview coding. Pilot coding of a small number of interviews was conducted independently by two researchers (SP and IL) and the team then met to agree on refinements to the coding framework and to add any additional emerging codes. The remaining interviews were then coded independently by four members of the research team (SP, IL, LH and DT).

In seeking to place our subsequent findings in a broader context, we drew on the NASSS socio-technical framework developed by Greenhalgh et al. (2017). This framework describes considerations such as the service’s capacity for innovation, the expected input/adaptation from care staff and the context for widespread use of the technology,37 and proved a helpful mechanism for making sense of our key findings. We also compared insights from our study with those of the ‘rational model’ of implementation set out in much of the literature (see Chapter 10). By considering both frameworks, we hoped to acknowledge the reality of complexity and uncertainty when implementing new and emerging technology, while also exploring some of the key headings or ‘stages’ that others might consider if planning future implementation.

Ethical approval and consent

The University of Birmingham research governance team identified this study as a form of service evaluation which did not require approval by the Health Research Authority (HRA) or an NHS Research Ethics Committee. Ethical approval was obtained from the University of Birmingham Humanities and Social Sciences Ethical Review Committee to conduct the stakeholder interviews (ERN_13-1085AP41, ERN_21-0541 and ERN_21-0541A). We sought and received an amendment to our initial ethical approval when later including interviews/focus groups with IndependencePlus and national technology providers/innovators.

Information sheets and consent forms were shared with potential participants, which set out the study aims, design, risks, benefits and who to contact if they had further questions. The information sheet also made clear that participants have a right to withdraw from the study at any point, without needing to give a reason. Information sheets and consent forms were shared with potential participants via e-mail. Of the 24 interviewees, 23 completed the consent form electronically and e-mailed it to the research team, and the other interviewee provided verbal consent having expressed a desire to consent verbally rather than via the consent form. At the start of each interview, the researcher confirmed that the interviewee was happy to proceed and to be audio-recorded.

Strengths, limitations and caveats

There are a number of strengths, limitations and caveats to note for this study, which are described here. Most notable are the challenges of engaging and recruiting case study sites and interviewees, which are outlined in the later subsection.

The research team considered whether to include multiple types of digital technology for social care, provided by a number of companies. We decided to have just one ‘exemplar’ digital technology used consistently across our case studies because, through our scoping work, it was evident that by keeping this constant, we could better establish what factors influence local decision-making and early experiences of implementation of new and emerging digital technologies. This decision was taken in conjunction with potential case study sites on the basis that different companies are constantly developing new technology, and that a helpful focus for this study would be on the underlying decision-making and implementation process rather than a direct evaluation of a specific digital technology. The use of IndependencePlus was therefore our sampling frame, making sure that the type of new and emerging technology being implemented was consistent across all sites. Had we included different technologies in different case study sites, there would have been too many variables to draw broader conclusions.

Interviews were conducted online to reduce the spread of COVID-19 and in line with government public health guidance. Benefits of online or video data collection have been acknowledged, including reduced travel time for researchers and the ease of scheduling a time that is suitable for interviewees.55 There are, however, some drawbacks to conducting interviews online. For instance, it can be more difficult to build rapport or monitor how a participant is feeling.56 Since the start of the COVID-19 pandemic, however, the research team have been developing their skills in online and telephone interviewing with health and social care professionals, people who draw on care and support and carers. The team have in place some techniques and solutions for ensuring interviews are conducted appropriately and that participants feel comfortable, such as sending lay information and discussion topics ahead of time, spending more time on rapport-building prior to the interview, active listening, consideration of and planning for technical difficulties, and upholding good practice and regulations for security and confidentiality.

Some stakeholders may not have had a positive experience with the technology, and their dissatisfaction or frustration may make them less keen to participate, or they may have thought we were only interested in ‘success stories’. Therefore, we ensured that invitations to participate clearly outlined the aims and potential benefits of the study (e.g. learning from past experiences and providing local authorities and care providers with better information for future decisions – irrespective of the ‘success’ or otherwise of local pilots). We also spent significant time in our scoping phase working with key leads across our case study sites, reminding them that we were interested in lessons which would help other local authorities and care providers exploring new and emerging digital technologies in future – not just in whether or not the technology ‘worked’. We feel that this is a key feature of the current study, but we found that we needed to give ongoing reminders and provide significant reassurance to sites throughout, as they were not used to sharing learning from innovations which had been abandoned or not as successful as they had hoped. We believe that there is a significant methodological and ethical issue here. However, the sheer effort expended in helping sites to feel comfortable sharing lessons that did not feel positive to them (for the benefit of others) was substantial and needed to be sustained throughout the life of the project – and there are important practical lessons here, too.

Challenges in recruitment of sites and interviewees

As mentioned, negotiating access and subsequent data collection with the case study sites was very challenging during the pandemic. We attempted many different ways of supporting recruitment, such as multiple e-mail reminders/calls, flexibility in (re)scheduling interviews, and offering participants the opportunity to provide written feedback via e-mail (rather than taking time out for an interview). Throughout the study, we were in almost constant touch with sites, trying to form a judgement as to whether to persevere with data collection or to pause and recommence when service pressures reduced. During our scoping work, sites were adamant that they wanted to take part – both to share learning with others and to help them reflect on their own learning. However, the conduct of the study in the midst of a pandemic was extremely arduous for the research team and for sites, and we had to keep reflecting on whether we were persevering with the study in order to enable people to share their experiences or simply ‘making a nuisance of ourselves’ when people had more important priorities (and this was a constant source of discussion and reflection in our team meetings, and in interaction with local leads). The reality of conducting very topical research in adult social care during the pandemic was that many months could elapse without proposed interviews having taken place (with multiple slots scheduled, cancelled, rescheduled, paused, not taking place on the day, rebooked – and then eventually going ahead when pressures temporarily eased). We believe passionately that there is helpful learning from this study, but extracting the learning took much longer than planned, and required significant perseverance, commitment and flexibility on behalf of everyone involved. We remain incredibly grateful to participants for nonetheless making the time to share key lessons with others, at possibly the most difficult time in the modern history of adult social care.

In addition to those mentioned, other significant engagement challenges existed due to a culmination of multiple factors:

• The five initial sites we approached adopted the technology on a pilot basis, but experienced significant difficulties, and all stopped using it. We believe that there is important learning for others from these experiences, but it meant there was less incentive for sites to engage, given that they had often moved on in practice. Relatedly, care staff perceived several of the pilots to have failed and, in some senses, to have ‘wasted’ their time. While this is a significant finding in its own right, it also made recruiting additional participants difficult.
• On occasion, there were politically contentious or difficult topics to work through with staff (e.g. negative attitudes towards the technology provider) – which again makes the learning valuable, but also made it difficult to recruit participants.
• The timing of the prioritisation exercise and the subsequent commissioning/scoping of the project meant that these pilots were run some time ago, and things had often moved on significantly at a local level.
• The technology was designed to be unobtrusive and to exist largely in the background, so people drawing on care and support (many of whom were older people, including some with dementia) did not recall the pilot.
• It can be difficult to engage technology providers in such research, given that a number of the pilots were perceived to be unsuccessful, and given a number of issues around commercial confidentiality.
• As discussed above, the research took place when adult social care was under unprecedented strain due to the COVID-19 pandemic – with significant additional pressures falling on staff whose role is technology-related (who often had to drop all other work and focus solely on adapting to the pandemic). Participating sites have been very generous with their time and very passionate about sharing their experiences – but we spent significant time building relationships and had to change plans instantly if more important priorities suddenly emerged locally. The team gave potential participants multiple opportunities to participate, made participation as quick and unobtrusive as possible, and informally chased and prompted as much as was appropriate in the context – but any more would have placed undue pressure on sites in a very challenging context, and would not have felt ethical.

Despite this, the study has generated significant learning about the key things to consider when making decisions about and implementing new and emerging digital technology, and has produced some practical guidance and learning that future organisations could find helpful when considering the adoption of AI-based technology. The later inclusion of the perspectives of technology providers is also an additional strength, often absent from the literature (see Chapter 1). As anticipated, the study fills key gaps in the literature, provides useful learning and works well within the context of the broader NASSS framework. It remains very timely given the potential benefits of new digital technology and the rapid changes which have been brought about by the pandemic – but progress was often painstakingly slow/hard-fought and the study was by no means easy to conduct.