Impact and learning

Impact of the study to date

At the time of writing, our dissemination, engagement and follow-through work are ongoing. From the outset of the study, we have emphasised active and ongoing engagement of policy-makers. Senior policy-makers from NHS England and the Scottish Government sat on our EAG and had ongoing input to the RBD study as it unfolded. The 39 ‘elite’ interviews conducted at the outset of the RBD2 study served to engage a wider group of policy-makers with whom we have kept in touch. This ongoing engagement can be categorised as ‘soft power’, helping to influence ‘hearts and minds’ and place us strategically as available to be contacted if academic input is needed (see last paragraph in this section). The Chief Investigators have regular ongoing meetings with national policy-makers leading on primary care transformation.

Presentations and seminars to date on our overall findings include:

1. Parliamentary dissemination event, held on 24 April 2024. Co-chaired by two parliamentarians who are also medical doctors, James Davies MP (House of Commons) and Baroness Ilora Finlay (House of Lords). Attended by 50 people including members of the project team, representatives from practices, patients and lay people, as well as MPs, Special Advisers, Health Foundation, Kings Fund, NHS England (including senior policy-makers from digital innovation teams and inequalities teams, and those responsible for education and lifelong learning), equivalent teams for Scottish and Welsh governments, NHS Resolution, patient charities and advocacy groups (e.g. Healthwatch), Royal College of General Practitioners (RCGP), Patient Safety Commissioner, Care Quality Commission (CQC), Healthcare Safety Investigation Branch, postgraduate deans, primary care leads for England, Scotland, Wales and Northern Ireland. The event included presentations from the research team plus a presentation from a patient (see Patient and public involvement). The formal presentations were followed by a networking tea hosted by Baroness Finlay and Dr Davies. Various cross-sectoral contacts were made. Again, deliverables from this event are indirect and mainly take the form of enriched academic-policy networks. A summary of policy recommendations and advice to MPs about supporting their constituents in relation to digital and remote general practice was launched at the event and published online (www.nuffieldtrust.org.uk/sites/default/files/2024-04/RBD%202024%20event%20briefing_WEB%20%281%29.pdf). A blog was written by one of the research team (www.spcr.nihr.ac.uk/news/blog/spcr-trainee-blog-post-reflections-on-a-visit-to-parliament).
2. European Dissemination Event. On 25 June 2024, we held an interactive webinar, ‘Digital primary care: from research into policy and practice’. We collaborated with Aarhus University in Denmark and World Health Organization (WHO) Europe to co-host a pan-European dissemination event, aimed at bringing research directly to policy-makers and practitioners. The goal was to share the findings from RBD2 and place this alongside research and policy work from other European countries on remote and digital health care. Following an opening keynote from Professor Greenhalgh, there was another keynote from a representative from WHO Europe and then talks describing comparable studies in Denmark, Catalonia, Estonia and Greece. This 3-hour event was attended live by 80 senior policy-makers from 31 countries, including many senior UK stakeholders. It was recorded; a full video is in preparation for public dissemination and an interactive PDF summary of key messages will be produced and disseminated to attendees. The workshop was well received, and further joint work is planned with stakeholders in other European countries (see Plans for taking this work forward).
3. End of Project Dissemination and Networking Event. On 5 July 2024 (slightly delayed because of election purdah), we held a major event at Green Templeton College, Oxford, ‘The changing face of remote access to primary care in the UK: Moving from research findings into implementation’. Fifty-five people attended, including academics, policy-makers, clinical leaders from NHS Resolution, NHS England, NHS Wales, Healthcare Services Safety Investigations Branch, ICBs, clinicians and managers from participating RBD2 practices, senior leaders from British Medical Association General Practitioners Committee and RCGPs, people involved in the GP curriculum and in other senior training roles and representatives from government across three of the UK nations. The study findings were shared (along with those from related NIHR-funded studies ‘Mode of Consultation’ and ‘Video and Hybrid Group consultations’) as part of the broader evolving landscape of remote access to care. In the afternoon, attendees joined breakout groups to consider how the study findings could be implemented across the areas of (1) contracts and regulation, (2) education and training and (3) within the practice itself. Study findings and implementation actions were summarised through visual minutes (Figure 1) and to be disseminated through our networks. A networking event followed into the evening.
4. Welsh Assistant Medical Directors (AMDs) presentation. In Wales, the Primary Care AMDs play a key role in system leadership, including in developing the GP contract, influencing national primary care policy and developing local enhanced services. We presented the study findings to this group and had a follow-up meeting with the Senior Medical Officer for primary care in Welsh government. The AMDs have been vocal advocates of our work, as an example sharing the recommendations from the safety paper with all GPs and practices within the Aneurin Bevan Health Board area.

FIGURE 1 Visual minutes from the Dissemination Event in Oxford 5 July 2024. Information about the above event was reproduced from the Remote by Default 2 website (www.phc.ox.ac.uk/research/groups-and-centres/interdisciplinary-research-in-health-sciences/remote-by-default-2/project-resources), which was published under the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited (https://creativecommons.org/licenses/by/4.0/). (Figure not reproduced here; see the original article.)

Below, we summarise how we have taken (and continue to take) forward specific themes.

1. Quality and safety. Our findings on risks to quality of remote and digital services, especially in relation to long-term condition management,77 and on the features of safety incidents and how these are usually avoided30 led to a stream of impact-oriented work with quality and safety leads. These included:
   1. RBD2 workshop on quality and safety. This was held in September 2022 and attended by 61 stakeholders from clinical, academic, policy, third sector and lay sectors (see Methods).
   2. Webinar and follow-on work with NHS Resolution. In March 2024, we presented at a bespoke education/review meeting with an arm’s length body responsible for indemnity claims against the NHS, along with external attendees. We presented and invited discussion on the findings of our patient safety analysis which had used de-identified data from (among other sources) NHS Resolution’s closed indemnity claims. Two hundred and fifteen people signed up and 97 attended on the day (others planned to watch a recording). The letter of thanks from NHS Resolution reported the results of an evaluation, which was very positive. Free-text comments included:
      • Practical advice that was transferrable to professionals other than GP e.g. I’m a palliative care consultant in community and think there are situations where I could better integrate remote consultations into my practice in order to see more patients over a large geographical area.
      • [T]he seminar was delivered in real language discussing how real events occur.
      • [T]here was transferable learning and made me think about the way we have introduced remote consultation in services with little or no training – covid leading much of this response. National Health Service Resolution have taken forward our findings through sending our recommendations through to practices in their newsletter and encouraging their adoption.
   3. Editorials aimed at clinicians in safety roles. We published two short editorials, one in the British Medical Journal122 and one in the International Journal for Quality in Healthcare.123
2. Access and equity. Our findings on the tendency of remote and digital services to widen inequities of access, especially among the multiply disadvantaged,63,66 led to a stream of work to help mitigate these inequities. This included:
   1. RBD2 workshop on access and triage. This was held in April 2022 and attended by 50 stakeholders from clinical, academic, policy, third sector and lay sectors (see Methods).
   2. Direct link with leading policy-makers. The NHS England lead for Primary Care Digital Transformation has been a member of our Advisory Board throughout, gaining direct and timely access to our findings.
   3. Editorials aimed at clinicians. We were commissioned to write a summary editorial on how digitalisation can leave the most disadvantaged patients without adequate access.124
   4. Linguistic minorities. We worked with Deep End general practices serving disadvantaged racial minorities with limited English (e.g. refugees) to produce PDF and paper versions of instructions for accessing the practice. For example, in one practice we translated practice access leaflet into Tigrinya, Ukranian, Romanian, Polish, Arabic and Russian.
3. Workforce and training. Our findings on organisational suffering, technostress and challenges to relational co-ordination among practice teams, and linked findings on training needs and key competencies for general practice staff,82 led to a stream of impact work on reducing stress, improving team relations and building competence in the workplace. This has included:
   1. RBD2 workshop on workforce and training. This was held in January 2023 and attended by 51 stakeholders from clinical, academic, policy, third sector and lay sectors (see Methods).
   2. Ongoing work by Nuffield Trust on workforce. The RBD2 findings have informed a wider stream of work on the future role of the GP and how to preserve and strengthen medical generalist skills in a context where remote consulting is driving more transactional care. Nuffield Trust work on this subject has included a round table with national policy-makers which drew on findings from RBD2.
   3. Educational webinars for GPs and other primary care clinicians in all jurisdictions. Between April and July 2024, we delivered educational webinars for primary care clinicians as part of their regular postgraduate education series in England, Wales and Northern Ireland. These events consisted of presentations (focusing mainly on the clinical practice implications) and a long discussion slot where we answered clinicians’ questions. For example, in May 2024, the main study findings were shared in a webinar to the Northern Ireland national GP and practice team. Presentations were given by Trisha Greenhalgh, Francesca Dakin and Rebecca Payne. The event was attended by more than 70 GPs and GP leaders live and recorded; it has since been watched by many more who could not attend on the day. It was rated 4.8/5 with example positive feedback ‘practical take home points, for example what conditions should be offered face to face care rather than remote encounter’.
   4. Management in Medicine seminar in Oxford. In November 2023, Trisha Greenhalgh delivered a continuing professional development (CPD)-approved session ‘How can we organise remote consultation services that are safe, effective and equitable?’ on this highly regarded course run from Green Templeton College, Oxford, and supported by the Faculty of Medical Leadership and Management. An estimated 120 people, mostly clinicians in training, attended.
   5. Editorials and practical papers aimed at GP leaders. Rebecca Payne coauthored a short article ‘Teaching patient safety in remote consultations’ in Education for Primary Care.125 Francesca Dakin led on a paper in BMJ Leader ‘Supporting your support staff’, with practical recommendations on how to improve staff job satisfaction and fulfilment and strengthen team relations.40
      1. British Medical Journal webinar for doctors. On 3 July 2024, Trisha Greenhalgh and Emma Ladds led a 1-hour webinar ‘How to make the organisational changes for remote and digital consultations’ by invitation as part of a wider series by the British Medical Journal on digitalisation and innovation in health care. RBD2 findings were distilled into practical points for an estimated live audience of 100, with many more watching the recording afterwards.
      2. Conference presentations. We made several presentations (including the opening keynote) at the Organisational Behaviour in Healthcare conference in Oslo (April 2024). See Conference presentations and keynote lectures based on RBD2 findings.
4. Technologies and infrastructure. Our findings on technologies and infrastructure – particularly the challenges of identifying, procuring, implementing and continually reviewing a range of interlinked technologies that are ‘fit for purpose’ for individual practices, workflows and patients – led to a stream of impact work to improve infrastructure and the fitness for purpose of and configurability of technologies. Specifically:
   1. RBD2 workshop on technologies and infrastructure. This was held in April 2023 and attended by 48 stakeholders from clinical, academic, policy, industry, third sector and lay sectors (see Methods). The workshop provided opportunities for interaction in particular between the designers and vendors of technologies and the front-line users in our participating practices. Various links were made, and practices arranged follow-up meetings, as the following extract from an e-mail illustrates:
      • The Accurx team did visit on [date] 2023 with their Chief Executive [name] coming. Another visit followed on [date] 23 with two people from their product development team. In both meetings they took time to look at how we were using their product as part of our total triage model and clinical workflow. We fed back several points to them and suggested a few priorities for further product development. I’d like to think that as a result of these discussions, a couple of recent product updates have been made to Accurx Patient Triage. There is now a function for patients to choose a specific clinician when sent a self-book link. There is another update shortly to come where we are able to select specific clinicians as options when sending self-book appointment links to patients (so we can decide whose appointments to offer). Secondly a really big update has just landed to the Usage Dashboard. It allows us to dig into historical trends and play with the data. It is extremely similar in appearance and function to the excel dashboard that we had made for analysing the data and that I shared with you.
   2. Stakeholder interviews stretched across policy-makers, local and regional decision-makers and industry representatives, providing critical opportunities to discuss and reflect on emerging findings, and to build relationships. This informed nuanced discussion with practices about the need to consider and attend to infrastructure and interoperability, and about specific technological developments and adaptations; with industry and decision-makers about the context of remote care in general practice, the typically incremental, step-by-step engagement with digitalisation and infrastructural requirements to support it; and with decision-makers the challenges faced by practices (individually and collectively) in procuring technologies relevant for their local setting and population, and those faced by industry (particularly small and medium-sized enterprises) in entering the market, understanding and enacting requirements for public sector procurement, and appreciating the context in which practice were working to introduce and evolve digital technologies.
   3. Additional dissemination work on this final theme is currently being planned.

A note about diffuse influence and soft power

One of the important parts of impact – how ‘soft power’ can influence individuals and groups – is the hardest to capture. Soft power includes the patient who now feels more confident to push for a face-to-face appointment when a condition is not getting better via remote; the designer of undergraduate Objective Structured Clinical Examinations who now designs their assessments around evidence-based competencies for remote and digital care; the policy-maker who better understands the caveats of digital and remote health care and is able to make more balanced decisions; and practice teams who have a better understanding of which modality is most appropriate for which patients and which condition. A number of team members now receive regular sense-checking calls from those in positions of power and are thus able to bring our research findings to the heart of national decision-making.

Further dissemination activities (e.g. press coverage) are listed in Appendix 1, Table 5.

Anticipated longer-term impacts

As the RBD2 study reached completion, a new (Labour) government was about to take up office. Access to care and efficient use of a stretched clinical workforce became a central policy theme. The Nuffield Trust will draw on findings from RBD2 to inform their briefings to ministers, see for example, their briefing ahead of the general election (published online: www.nuffieldtrust.org.uk/sites/default/files/2024-06/Election%20briefing%203_general%20practice%20and%20dentistry_WEB_1.pdf).

At the Parliamentary briefing event in April 2024, we made a number of key contacts which we will be taking forward.

Lessons learnt for future research

In a previous study in UK general practice published in 2019, Newbould et al. covered 146 practices and demonstrated considerable variation in the impact of a ‘telephone first’ innovation, with some practices reporting overall increases in workload and efficiency while others reported a decrease.126 But the design of their study (larger, more quantitative and focusing more on before-and-after metrics than unfolding-over-time narratives) did not allow them to tease out the complex contextual influences and path dependencies that could fully explain between-practice variation. When we submitted our stage 1 bid for the RBD2 study, we had initially conceptualised a study with some similarities to the Newbould study, including a larger unit of analysis [the Primary Care Network (PCN), which would have become the Integrated Care System in England part-way through the data collection period]. On the basis of insightful feedback from anonymous reviewers, we changed the unit of analysis to the individual general practice – a shift which enabled us to compare rich case narratives of 12 very diverse practices. We think this shift worked well, in that we now have a good understanding of why a technological innovation that increases efficiency and patient satisfaction in one practice might have the opposite effect in another practice. We cautiously conclude that the trade-off between breadth and depth in RBD2 was worthwhile and that other researchers might consider a ‘small n’ study using the researcher-in-residence model to explore the complexities of contemporary general practice.

Related work

Three PhD projects were launched from the RBD2 platform and a fourth was linked to it.

Dr Emma Ladds, a GP, was funded by Wellcome Trust (via NIHR School for Primary Care Research) for a doctoral study entitled ‘What is the Role of the Therapeutic Relationship in Modern Day Primary Care?’ Inspired by the cross-cutting theme of continuity in RBD2 (which she led as an Academic Clinical Fellow), Dr Ladds designed a novel methodology to prospectively explore the unfolding therapeutic relationship between a small sample of patients and their GPs (and other practice members) over 2–3 years, using longitudinal ethnography. This ongoing study, which is undertkaen part-time alongside clinical work, will contextualise findings in relation to the historical evolution of the role of the GP and the opportunities and constraints of modern technology.

Laiba Husain, a psychologist and health services researcher, was funded on a THIS Institute Improvement Science Fellowship for a doctoral study entitled ‘The Shift to Remote Consultations and its Implications for Digital Health Disparities’. Ms (now Dr) Husain, a multilingual researcher from a minority ethnic group, recruited 23 participants from a telehealth service, all with multiple axes of disadvantage: elderly, first-generation immigrants, socioeconomically deprived, low digital literacy and multiple health conditions. Using in-depth interviews and home visits, she prepared individual case studies and used these to construct fictional personas. She is now working with providers and service users, drawing on these personas and her wider data set, to co-design services that are more accessible to this vulnerable group. She has now passed her PhD.

Francesca Dakin, an anthropologist and health services researcher, was funded by the NIHR School of Primary Care Research to look at the impact of the COVID-19 pandemic (and associated technological and structural changes in general practice) on staff in GP surgeries (including support staff such as receptionists and back-office workers, and clinical staff such as GPs, nurses, phlebotomists). She co-led the cross-cutting theme of ‘workforce’ in the RBD2 study, identified the subthemes of ‘technostress’ and organisational suffering which we describe in the findings section (findings to be published in British Journal of General Practice). Building on these findings, she developed a linked PhD project looking at team relations, psychological safety and relational co-ordination in the context of resource constraints, workforce shortages, advancing digitalisation and high demand (findings published in BMJ Leader). She also co-led the inequalities theme, developing a novel theory of digital candidacy and the concept of the digital facsimile (findings published in Social Science and Medicine). She has now passed her PhD.

Dr Amy Booth, a medical doctor, commenced a DPhil in Translational Health Sciences in 2021. During her time working as a clinician during the COVID pandemic, she became acutely aware of the impact the health system has on the environment. Her research explores the environmental impact of health systems and how a sustainability lens can be embedded into healthcare practice and policy. She has contributed to a major systematic literature review on the role of virtual consulting on developing environmentally sustainable health care.127 She worked with RBD2 researchers in residence to ensure that practices’ awareness of the green agenda (and its links to remote health care) was explored.

Dr Hassane Amani, a visiting scholar from the University of Montreal, joined us for a year and helped with data collection. He contributed to the stream of RBD2 work on infrastructure and the links between access and equity, and undertook a linked project on the system-level links between universal health care, human rights and the need for publicly funded (as opposed to private, for-profit) digital platforms.128

Our work on remote consultations surfaced novel research questions about what happens after such consultations in terms of patients obtaining the tests, medication and ongoing support they need. We have already published an Analysis article ‘After the teleconsultation’,129 and Rebecca Payne has successfully applied to study an aspect of this for a DPhil at the University of Oxford.

Collaborations and future funding

We have developed multiple new research collaborations. One example is that our work on the organisational and workforce aspects of remote and digital care led us to develop collaborations with academics in the Said Business School, University of Oxford (Associate Professor Michael Gill), and the Organizational Sociology Group at Aalborg University, Denmark’s Department of Sociology and Social Work (Professor Ninna Meier). We have begun to collaborate with researchers at Aarhus University, Denmark (with whom we co-hosted the European Dissemination event); a joint grant application is being planned.

The RBD2 study helped attract visiting professor Tiffany Veinot from the University of Michigan, USA, who has an international reputation for her work on digital inequity. Professor Veinot worked with the RBD2 team and supported and extended our digital inequity work. An additional book chapter is in preparation.

Within IRIHS, the RBD2 grant formed a strong platform from which to apply for further funding for digital health research. In addition to the original RBD2 funding (£893,417), three externally funded PhD places and three pre-PhD fellowships described in the previous section, we have secured the following additional grants related to remote and digital health (in addition to other grants on wider topics):

• Care Navigation Extension to RBD2 (Oxford and Plymouth) NIHR Health and Social Care Delivery Research: social care extension 2022–4. PI – Greenhalgh: £177,157.
• MoDCons: Mode of Consultation in general practice NIHR SPCR 2022–4. PI Shaw. £398,566.
• The Oxford-RAND Europe centre for rapid evaluation of technology-enabled remote monitoring: DECIDE (Digitally Enabled Care in Diverse Environments) (Oxford) NIHR Health and Social Care Delivery Research. 2023–6, PI – Shaw. £2,080,430.
• Social and Ethical Aspects of Remote and Hybrid Care in the Special Allocation Scheme in general practice (SEARCH): A mixed methods study. (Oxford) NIHR School for Primary Care Research. PI – Brenman, van Dael with Shaw as senior mentor. 2024–6. £231,514.
• Evaluation of Enhanced Reconnect services for prison leavers, NIHR Policy Research Programme, £400k, 2024–5. (Plymouth) Co-PIs: Byng and Rybcynska-Bunt.
• MocCons Goes Dutch (funding/funder). Cross-national perspectives on remote care and determining modes of consultation in Dutch general practice. GP. PI – Dakin. €25,000 from Amsterdam Huisartsen Alliantie (Amsterdam GP Alliance). Co-PIs: Wieringa, Shaw.

Plans for taking this work forward

A number of streams of work are ongoing to enable further impact from this study. In addition to the new research projects that have already begun (see above), these include:

1. Using our competency framework to inform training standards. We have begun to work with Health Education England and the RCGPs to map our competency framework for remote and digital services to the postgraduate training of primary care clinicians. This work is currently at an early stage. We have also begun to work with University of Oxford undergraduate teachers to inform medical student training. Again, this work is at an early stage at the time of writing.
2. Pursuing our insights on quality and safety with policy-makers. Rebecca Payne has informed CQC leads of the RBD2 findings on quality, and begun discussions as to whether these findings could inform CQC inspection methodology. Rebecca Payne, Trisha Greenhalgh and others have begun dialogue with policy-makers in the Welsh government, Scottish primary care leads, and NHS England as to which of our findings might be relevant to be included in developing metrics (e.g. within practice self-assessment tools). Some discussions with senior policy-makers are confidential at this stage and therefore not reported here.
3. International case study booklet. Four of the speakers at the European Dissemination Event (June 2024, see Impact of the study to date) are now working to produce a short, policy-oriented booklet covering learnings from their respective countries, aimed mainly at EU policy-makers.
4. Professional quality resources to convey key messages to different audiences. We continue to work with Design Science to produce accessible and engaging information and resource materials in PDF and paper format. Sara E Shaw is in discussion with policy-makers about aligning practice-focused resources and disseminating most effectively. Target audiences include:
   1. Resources to support care navigation. We are co-designing an infographic that will help care navigators (staff that provide personalised digital support during patients’ navigation of care) consider what they can do to better support patients navigating care services digitally (i.e. getting an appointment, using technology). This will include a ‘worksheet’ of digital candidacy archetypes that can be used to better understand how and why patients may have complex needs and how to identify them. National Voices are engaged in this work, and likely to support dissemination.
   2. Visual summaries of eight key papers. Findings on key cross-cutting thematic papers (e.g. quality, safety, access and equity) will be summarised in visually appealing one-page summaries.
   3. Patients. We are extending the existing guidance we produced for patients when they seek to access general practice care, highlighting the challenges that digitalised systems can create and how to navigate these.
5. Continuing speaking engagements. Several members of the RBD2 research team have been booked as speakers for postgraduate training events, seminars or conferences. For example, Trisha Greenhalgh has been invited to speak to the BMA General Practitioners Committee and the Urgent Health UK Annual Conference. RCGP Wales and Scotland are currently in the process of adapting our English parliamentary briefing for their devolved nation parliaments and administrations. This will be co-badged with RBD2. Release has been delayed due to the election but is scheduled for early autumn 2024.