Synopsis

Research summary

Increasingly, bacteria (germs) no longer respond to antibiotics. This can result in patients experiencing more severe or longer infections and may lead to an increased risk of dying from infections. This is due to too much – inappropriate – use of antibiotics.

The reasons for overuse of antibiotics include doctors not having relevant information when prescribing, concerns about missing possibly serious infections, and limited opportunities to review diagnoses and stop antibiotics if no longer needed. A detailed background is provided in Report Supplementary Material 1.

From our previous research, we found that hospital electronic prescribing (ePrescribing) systems can support new ways of working to safely reduce unnecessary antibiotic prescribing.

We built on our previous work to see if ePrescribing systems can safely reduce antibiotic prescribing.

We conducted interviews with clinicians and undertook observations in hospitals, and used these insights to develop new tools to support prescribers, training of prescribers and organisational changes. The development and go-live date of this complex intervention, which we called ePrescribing-based Antimicrobial Stewardship (ePAMS+), were delayed due to COVID-19.

Once pandemic restrictions were lifted, we were able to progress a trial of ePAMS+ across two hospitals, showing it was possible to implement ePAMS+ and to extract the information we needed to see if it was reducing inappropriate antibiotic usage. We found little appetite from clinicians to engage with the training and use ePAMS+, as hospitals were under pressure from pandemic backlogs and NHS strikes.

The delays to our main trial resulted in other already recruited hospitals having to innovate locally to address antibiotic overusage. These innovations overlapped with ePAMS+ such that it was no longer possible to progress the main trial as planned.

We decided not to progress the trial as we would have struggled to recruit enough hospitals to generate a clear answer.

Finally, we interviewed NHS staff to understand new ways hospitals were attempting to reduce inappropriate antibiotic use.

Work package 1: development and evaluation of a complex ePrescribing-based Antimicrobial Stewardship intervention for hospitals

Some text in this section has been reproduced with permission from Mureyi et al.1 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Phase 1: planning, developing, and optimising all elements of the intervention

Aim

We aimed to codevelop ePAMS+ with stakeholders, to facilitate optimal prescribing and timely review of antimicrobial therapies in hospitals in England, building on existing behavioural approaches embedded in antibiotic reduction and conservation (ARK).2

Methods

Approach to intervention development

In developing ePAMS+, we sought to build on ARK (https://antibioticreviewkit.org.uk/). The ARK intervention comprised of behavioural and organisational components aimed at helping healthcare practitioners review antibiotics prescriptions in hospital acute admission settings and safely stop unnecessary antibiotics. Our conceptualisation of ePAMS+ differed from ARK in three important ways: (1) ARK focused on stopping antibiotic prescribing only at review, whereas ePAMS+ aimed to improve the decision-making process at review for all viable options, including starting, stopping or modifying treatment; (2) ARK was designed for acute admission contexts, whereas ePAMS+ was designed to be implemented in all hospital contexts; (3) ARK is a behavioural and organisational intervention, while the ePAMS+ intervention included technological components embedded within the ePrescribing system. ePAMS+ was developed according to the person-based approach to intervention development3 – an approach widely used to successfully develop behaviour change health interventions, including ARK. The person-based approach is characterised by deep insights into psychosocial contexts, and perspectives of people whose behaviour is being targeted by the intervention.

These outputs were not necessarily produced in strict sequence.4 For example, modelling during ePAMS+ planning and development was a continuous and iterative process, stakeholder engagement for the eliciting of behavioural and contextual issues and challenges was also a dynamic, iterative and continuous process – as is required in complex health information technology projects.5 The activities pertaining to each person-based approach output listed in Report Supplementary Material 2, Box 1, were performed and the planning and execution of activities following the second structured stakeholder workshop (refinement of prototype and eLearning module, and qualitative interviews) took place over an extended period (2020–1) due to the COVID-19 pandemic. ePAMS+ intervention logical model (research pathway diagram) is listed in Figure 1.

FIGURE 1 ePrescribing-based Antimicrobial Stewardship intervention logic model (research pathway diagram). WP, work package. (Figure not reproduced here; see the original article.)

The three key steps taken, following the person-based approach to ePAMS+ planning and development are summarised below.

Identifying key behavioural issues, needs and challenges that the intervention must address

Reviews of literature, two face-to-face stakeholder workshops, meetings with the intervention development group (a smaller subset of expert stakeholders who helped review the contributions from stakeholder workshops and refine the intervention) and 18 online semistructured qualitative interviews with medical prescribers and pharmacists were conducted in Newcastle University Hospitals Trust.6

This helped to explore current prescribing and antimicrobial stewardship (AMS) practices, investigate potential areas for improvement as well as to solicit feedback on ePAMS+ prototype iterations. More specifically, this step included:

1. Review of relevant theory and evidence from past trials of complex interventions to improve AMS:
   • A systematic scoping review exploring the appropriate use of antimicrobials through hospital ePrescribing systems.7
   • Related conceptual work.8,9
   • A systematic review of the effect of digital AMS programmes on antimicrobial usage, length of stay (LOS), mortality and cost.10 Broader literature on the development of digital health technology tools and AMS tools, including ARK.11
   • Structured stakeholder workshop 1:

At this workshop, we introduced ePAMS+ with an oral presentation. This presentation covered: aims of the workshop and ePAMS+, the behavioural and technological components of the intervention (including what the proposed interface would look like to the user) and the planned project timeline from planning and development, to optimisation, implementation and evaluation. Appendix 1, Table 1 summarises details of the workshop (aim and participants) and lists the components of the ePAMS+ intervention that were initially proposed and presented at the workshop.

Participants were asked to consider what changes they thought should be made to the intervention components. A guide with prompting questions elicited feedback on each ePAMS+ component. Contributions included the advantages and disadvantages of proposed components, suggested additions or removals, comments on the timing of on-screen prompts and content of search menus and drop-down lists. A participant handout was also distributed, onto which participants wrote how they felt ePrescribing systems could better support them to adhere to antimicrobial prescribing best practice. The handout also specifically asked about what functionality in ePrescribing systems were perceived to facilitate or hinder the following tasks:

• starting antimicrobial therapy in a way that makes reviewing it easier
• writing an indication/diagnosis
• reviewing antimicrobial prescriptions
• stopping unnecessary antimicrobials
• changing from broad to narrower spectrum antimicrobials.

The issues were then discussed collectively. They mainly revolved around the compatibility of the proposed components within the context of hospital processes.

• Structured stakeholder workshop 2:

Stakeholder workshop 2 followed the same format. The components of the ePAMS+ prototype that had been refined based on feedback from stakeholder workshop 1 were presented and feedback elicited. Proposed changes to intervention features were evaluated against the MoSCoW prioritisation method ‘must-have, should-have, could-have, and won’t-have, or will not have right now’ (MoSCoW).12 This method provides a means of prioritising requirements by allowing these to be identified. It includes the following components:

• Must be implemented for the tool to meet the essential needs
• Should be implemented, if possible, but without which the tool can still be successful
• Could be implemented if the resources involved do not negatively impact the fitness of the tool to meet its objectives and
• Would be implemented if spare resources (time, expertise, funds, etc.) are left over at the end of an iteration.

Notes of stakeholder contributions from both workshops were taken and for ease of implementation, thematically coded as follows:

• Contributions to be included in the list of changes to be made to the user interface.
• Contributions to be included in the intervention planning table (i.e. contributions that mentioned a barrier to or facilitator of antimicrobial review and revision).

Qualitative interviews

The qualitative interviews are reported below in phase 2a.

Creating the intervention planning table

On the strength of insights from stakeholder workshops, qualitative interviews, experts and literature, an intervention planning table was developed to inform decisions about ePAMS’ content. Stakeholder feedback from the engagements already described were discussed by the intervention development group (comprising the research team and six members of stakeholder groups), 6 weeks after stakeholder workshop 2. Decisions about which interventional components to adopt were based on:

• Engineering/programming feasibility (e.g. changes that the Cerner ePrescribing system would not permit were not implemented. Complex suggestions were reserved for possible consideration during the optimisation stage).
• Potential usefulness to users (proposed features that stakeholder’s thought was unlikely to be used at all were dropped).
• Transferability to multiple settings or changeability in future iterations (changes that would necessitate re-engineering when clinical guidelines changed, or changes that would be at odds with clinical guidelines/templates at other hospitals, were not incorporated).

The aim was to create a pragmatic and usable intervention while addressing as many cited barriers to ‘review and revise’ as possible.

Theoretical modelling

A logic model was created that describes ePAMS’ key components, hypothesised mechanism of action and the intended effect on target outcomes. A behavioural analysis of the various components of ePAMS+ was also carried out, using three behavioural change frameworks: the Theoretical Domains Framework,13,14 the Behaviour Change Wheel [the Capability, Opportunity Motivation-Behaviour (COM-B) Model]2 and the Behaviour Change Taxonomy.15,16 The complete behavioural analysis outlining the identified barriers to and facilitators of AMS in hospitals, ePAMS+ feature addressing each barrier and/or supporting each facilitator and the behavioural change techniques embodied in each ePAMS+ feature, as well as ePAMS+’s description following the Template for Intervention Description and Replication (TIDieR), are attached in Report Supplementary Material 2. Additional details of work package (WP) 1 components are elaborated in Report Supplementary Material 2 (Figure 1, Box 2 and Table 2).

Key findings

Barriers to and facilitators of ePrescribing-based review of antimicrobial prescriptions

Stakeholder workshops, qualitative interviews and literature reviews identified 20 barriers to and 5 facilitators of antimicrobial prescription reviews and revisions that ePAMS+ aims to address (detailed in Report Supplementary Material 2).

The beliefs, lack of knowledge and conflicting clinical priorities among healthcare practitioners, which figured prominently among barriers identified, negatively impacted their confidence and motivation to review and revise antimicrobial prescriptions. For instance, healthcare providers may believe that stopping an antimicrobial course before completion may promote bacterial resistance, or they may lack the knowledge about what needs to be done, or who needs to be involved in the review and revision process. Additionally, patients may have beliefs about the efficacy of antimicrobials or certain routes of administration [intravenous (i.v.) vs. oral] and may be anxious about unexplained changes. Priorities that inadvertently compete with AMS render some prescribers more likely to increase antimicrobial usage rather than reduce it. Such competing priorities include the worldwide sepsis agenda, that is, the drive to prevent sepsis-linked mortality by promptly administering empirical antibiotic therapy.17

Limitations

The COVID-19 pandemic caused delays and interruptions, and research and development activities had a lower priority than front-line clinical work; therefore, the development work timescale may not be representative of the time it should take to develop an intervention of this scope. Furthermore, although there was considerable stakeholder input into the development of ePAMS+, this did not include the entire potential user population. However, engagement of a sample is superior to no engagement at all. In particular, nurses were severely under-represented in the stakeholder sample. Although they are not directly involved in prescribing, more engagement with nurses would have been useful because they are involved in medication management workflow.

Relationship with other work packages

This phase resulted in developing the prototype intervention that was then implemented and tested as part of the feasibility trial.

Phase 2: carrying out qualitative feasibility and process studies of the implementation

Phase 2a: qualitative research informing the intervention development

Some text in this section has been reproduced with permission from Cresswell et al.18 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Aim

We sought to understand what technology-based AMS practices were used by prescribers in their clinical practice, including existing workflows, and explore how these could be better supported.

Study design and setting

We conducted a qualitative semistructured interview study in an acute NHS hospital in England using one of the main ePrescribing systems in use, to obtain opinions and insights regarding AMS and prescribing practices.

Sampling and recruitment

We purposefully sampled pharmacists and doctors with a range of seniority/experience, who worked on wards that have a high use of antimicrobials in the hospital and that were to adopt the developed intervention. In doing so, we included participants who were involved in the prescribing and reviewing of medicines. We excluded those who were not involved in these processes. Participants were recruited with the help of gatekeepers, including the lead informatics pharmacist, the chief pharmacist, the lead consultant in infectious diseases and the AMS pharmacist. They were asked if they were interested in participating in an interview by gatekeepers. If participants expressed interest, their contact details were passed on to the lead researcher (Susan Hinder), who e-mailed them directly with the study information sheet and arranged an interview. We also snowball sampled further participants by asking interviewees to recommend others, actively searching for a range of different opinions on AMS practices and electronic prescribing. We stopped recruitment when no significant new themes were emerging during the concurrent analysis (saturation). Overall, we invited 34 participants via e-mail. Sixteen of these did not respond to our invitation (which included a reminder e-mail approximately a week later).

Data collection

One-to-one interviews were conducted remotely over Microsoft Teams (Microsoft Corporation, Redmond, WA, USA) by Susan Hinder using a semistructured interview guide between 9 February 2021 and 14 February 2022 (detailed in Report Supplementary Material 2).

Data analysis

The interview transcripts were coded using NVivo 12 (QSR International, Warrington, UK), applying a coding framework (added in Report Supplementary Material 2, Box 4) developed by the research team. This was based on the existing literature and included themes surrounding barriers and facilitators to AMS.

Key findings

We conducted 18 interviews.

Our analysis identified three themes, including: (1) uncertainty of prescribers and reviewers around prescribing decisions, (2) prescribing as a set of activities enacted by multiple actors and (3) hierarchies and relationships shaping prescribing and review practices.

Limitations

There is now a need to test these experiences across a wider range of settings with varying existing AMS practices, systems and infrastructures. The setting we studied was relatively advanced in terms of antimicrobial prescribing practices and stewardship. Studying sites that are less technologically mature might uncover additional issues. We were also limited by the fact that, due to the ongoing COVID-19 pandemic, we could not observe practices and dynamics. COVID-19 may also have altered the AMS landscape – there has been more remote working, less direct education around AMS and, in particular, the intricacies of the system (e.g. training regarding ‘possible’ and ‘probable’ infection and interpretation of this on review). This would have helped to uncover additional intricacies arising in interactions that may not have been recalled by individuals. We further could have studied the wider organisational environment in more detail. For instance, some participants mentioned a tension between perceived value of initiatives by management (which often centred around cost savings), and the difficulty of attaching value to improved practices that may in some instances result in increased costs.

Relationship with other work packages

This WP has provided a baseline assessment of existing work practices before the intervention was implemented in the feasibility trial.

Phase 2b: qualitative research in parallel with the feasibility trial

Aim

We sought to explore how ePAMS+ was implemented to extract lessons for the implementation and evaluation of complex digitalisation initiatives in healthcare settings.

Sampling and recruitment

The senior ePrescribing pharmacist in the hospital developed the technical aspects of ePAMS+ in collaboration with the research team, drawing upon wider stakeholder engagement (see Phase 1: planning, developing, and optimising all elements of the intervention). Their gatekeeper role helped us to establish contact with clinical leads and early users of the intervention. Fourteen wards altogether implemented ePAMS+ in the feasibility trial. We selected five acute wards with high antibiotic use for the feasibility trial to be first adopters of ePAMS+. These included Infectious Diseases, Elderly Care, two Oncology, and respiratory disease wards. We excluded wards that had high prophylactic use rates of antibiotics, for example, general surgical wards, as the indication for antimicrobial prescribing differed to medical wards.

Data collection

We explored prescribers’ experience of using ePAMS+ 2 and 7 weeks after ‘go live’, and the researcher collected qualitative data during 2 week-long field visits in October and December 2022. Interviews explored existing AMS practices, perceived impacts of different elements of ePAMS+, facilitators and barriers to effective use, and suggested improvements. Questions were tailored to emerging findings and the role of the interviewee, and we also fed emerging findings into concurrent data collection, which resulted in some modifications to the topic guide as the study progressed. They were digitally audio-recorded and professionally transcribed.

Where feasible, we also conducted non-participant observations of ePAMS+ use. This involved the researcher observing ward rounds and clinical discussions before and after these (including handovers). The researcher took notes during these meetings, including describing the set-up, issues discussed in relation to AMS, and emerging tensions. We did not record any patient-specific data. We stopped data collection when we reached thematic saturation, that is, when no new substantive themes were emerging from concurrent data collection in both interviews and observations.

Data analysis

Interview transcripts and observation notes were uploaded to NVivo v.12. Susan Hinder did the initial coding along the dimensions of the technology, people, organisations and macroenvironmental (TPOM) as the coding framework.19,20

Key findings

We interviewed 23 junior doctors, 2 consultants, a senior nurse, the ePrescribing pharmacist who developed the ePAMS+ intervention and a microbiologist over 2 site visits.

Overall, AMS was viewed as important with potential for improvement for prescribers. We found several factors affecting the delivery of the intervention, including evolving technological properties, social and organisational transformations that could not be planned, and external pressures on hospital operations. These will be explained in more detail in the paragraphs below.

Difficulty planning for edge cases that did fit within ‘typical’ prescribing processes in advance

ePrescribing-based Antimicrobial Stewardship was designed based on extended requirements elicitation with experienced healthcare professionals (HCPs). This had generated descriptions of ‘typical’ prescribing processes from the perspective of some specific individuals and contexts. The feasibility trial of the intervention in real-world settings identified a wider range of usage processes and contexts than had been identified by stakeholder engagement. Variations in prescribing workflows and in antibiotic regimen were not catered for with the developed generic intervention. This highlights the difficulties for stakeholders in anticipating the significant range of use cases, contingencies and needs in requirement elicitation exercises, which consequently tend to focus on generic rather than ‘edge’ cases.

What a focus group tells you and what a senior clinician tells you, and what a junior doctor wants is sometimes quite different, so we originally designed it with a link to the guidelines in but the senior clinician said there’s no point in having that because at that point you already know what antibiotic you want to prescribe so why would you need to go into the guideline but then […] I think what senior clinicians were thinking about that’s the initial decision […] but I think what they weren’t thinking about the purpose of the plan is to also allow review.

For example, ePAMS+ listed only commonly prescribed antibiotics and followed a generic process for administration and review. However, some antibiotics, for example, gentamicin, only had single doses, so the 48-hour prompt for review was not relevant.

The whole idea of a forty-eight-hour review is completely meaningless because it only ever is a one-off dose.

Similarly, for some antibiotics, the first dose was different from the following doses, which ePAMS+ had difficulty accounting for.

Doxycycline, you often will give like loading dose, and then give a lower dose the next few days, but ePAMS will only prescribe the loading dose.

ePrescribing-based Antimicrobial Stewardship also did not list all the antibiotics used in the hospital. This made it difficult to prescribe combinations of antibiotics, as ePAMS+ was not set up to support complex antibiotics plans with combinations of antibiotics.

So, it’s not got every antibiotic on. I was trying to use it on assessments for a couple of more challenging patients with complex antibiotic plans. And it’s a bit […] what I did find a little bit challenging is sometimes they were going for a combination of antibiotics. One antibiotic is on ePAMS and the other one isn’t. And I was less sure what to do in that situation. I just used it for one of them and not the other.

Limitations

Due to the pandemic and other health service pressures, the feasibility work was undertaken at a later stage of intervention development than originally intended. In line with the person-based approach,21 this work would ideally have been undertaken before the feasibility trial so that emerging problems could have been detected and ironed out at an earlier stage. In addition, due to intense competing pressures in the hospital, we did not have access to observations of direct ePAMS+ use, although we did have the opportunity to get insights into AMS practices during observations of ward rounds. In our data collection, we did not manage to speak to any user who had completed the entire review process using the ePAMS+ intervention. This was in part due to the limited scale (five wards) and period for which the intervention had been live at the time of data collection. The plan was that this would be redressed in the planned hospital-wide roll-out of the intervention where we hoped also to be able to observe how the intervention becomes embedded into routine practice. It was envisaged that the full implementation would also likely include a refined intervention based on these experiences. Nevertheless, our key contribution was a unique insight into how an intervention is developed and implemented in practice.

Relationship with other work packages

This work ran in parallel to the quantitative feasibility trial.

Work package 2: agreeing secondary outcome measures and establishing the feasibility of evaluating ePAMS+ through a hybrid cluster-randomised stepped-wedge trial

Aims

Work package 2a

The main aim of WP2a was to:

• Develop a nationally agreed set of secondary outcome measures for the definitive trial.

Work package 2b

Some text in this section has been reproduced with permission from Weir et al. 2023.22 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text. The qualitative and Fidelity Index aspects of the feasibility trial aimed to:

• Explore user acceptability of the content of the ePAMS+ technical component and identify any barriers to use.
• Assess whether ePAMS+ antibiotic order plans were used as intended in clinical practice and, if not, identify barriers.
• Assess acceptability to HCPs of the content of the ePAMS+ intervention plan (ePAMS+ organisational component) and training materials (ePAMS+ educational component).
• Understand how ePAMS+ may be best delivered across multiple care settings and site information systems.
• Determine whether the procedures for implementing ePAMS+ are acceptable and feasible.
• Develop a Fidelity Index to quantify the extent to which core principles of ePAMS+ are enacted in antibiotic prescribing practice and test its usability.
• Confirm hypothesised mechanisms of action, refine programme theory and identify appropriate process analysis measures of mechanisms of action for a future full-scale trial.

The feasibility trial and its quantitative analysis aimed to:

• assess whether ePAMS+ could be successfully integrated into hospital settings to enable changes in prescribing behaviour
• develop processes of collecting outcome data from ePrescribing systems prior to and following introduction of ePAMS+
• determine between-patient variability in total antibiotic consumption to enable planning of the full-scale cluster-randomised hybrid stepped-wedge clinical trial of WP3.

Methods

Work package 2a

Develop a nationally agreed set of secondary outcome measures for the definitive trial

In June 2021, 44 expert practitioners were invited to participate in an electronic consensus meeting (eDelphi). The remote modality replaced the originally planned face-to-face nominal group meeting due to the COVID-19 pandemic and had ethical approval via an ethics amendment. Out of the 44 invited, 12 people agreed to participate. The exploratory round contained 36 outcome measures for expert review – 33 from the scoping review of the literature, and a further 3 were included that had identified from a student project to scope digital AMS. In the exploratory round, a total of 45 were suggested by experts for inclusion.

In October 2021, the indicators and suggested indicators were sent back to participating experts to score each outcome/process measure using a seven-point Likert sale for appropriateness, reliability and feasibility. In total, there were 81 indicators sent to experts for scoring.

Although nine experts agreed to participate, only four people responded. In view of this, the research team assessed the outcomes to determine those to be taken forward.

Work package 2b

Design

The quantitative and Fidelity Index elements were as follows.

Quantitative analysis developed methods of deriving key summaries of antibiotic use and estimated the variability in these measures using routine administrative data extracted from the Cerner ePrescribing and Medicines Administration (EPMA) system at each site.

We developed a Fidelity Index prototype to assess the enactment of the ePAMS+ intervention. This involved examining how closely practice matched the underlying principles of ePAMS+. We planned to automate the Fidelity Index by developing rating scales to be applied automatically at critical decision points for ePrescribing in individual patients using data from the EPMA system.

The ePAMS+ intervention built on the work of ARK, which produced an Antibiotic Review Kit that increased the proportion of antibiotic prescriptions reviewed within 72 hours from 91% to 99% and the proportion of antibiotic prescriptions stopped within 72 hours from 9% to 35%.15 ePAMS+ extended the behavioural and organisational work of ARK in three important aspects:

1. While ARK focused on stopping antibiotic prescribing at review, ePAMS+ aimed to improve the decision-making process for all viable options, including starting and stopping treatment, optimising the dose regimen, switching the route of administration, changing the antibiotic and continuing treatment for an appropriate duration.
2. ARK was implemented only in acute admission contexts, whereas ePAMS+ was applied to all hospital inpatient antibiotic prescribing.
3. ARK was a behavioural and organisational intervention, while ePAMS+ also implemented a clinical decision support (CDS) tool that exploited existing ePrescribing system functionality to automate, sustain and integrate effective support for appropriate antibiotic prescribing into all hospital prescribing pathways.

The novel elements of ePAMS+ were informed through our extensive liaison with policy-makers, professional and patient representatives, vendors and international experts to create a novel intervention with potential to support HCPs and clinical teams at all key stages of antibiotic medicines management.13,16 ePAMS+ aligned with best clinical practice and the national ‘Start Smart–Then Focus’ guidelines,10 the main principles of which are that antibiotics should be started promptly if bacterial infection is suspected; reviewed regularly during the first 48–72 hours of initial prescription to determine if antibiotics are still required; and stopped, switched or optimised as appropriate, once all test findings to inform decision-making are available. The ePAMS+ intervention embedded the following tools in the Cerner EPMA system:

• Antibiotic order plans (to prescribe antibiotics and schedule a series of review points where changes in prescription may be required).
• Decision aid (to record the original prescriber’s level of certainty about the need for antibiotics – based on the ARK intervention classification15 of possible infection risk, probable infection risk or finalised diagnosis of infection – to inform a later decision to stop prescription as appropriate).
• Decision aid [recording proposed infection site (Body System) and working diagnosis (Indication)].
• Information pages (to help users benefit from ePrescribing tools).
• Antibiotic ward task list (to identify patients being prescribed antibiotics where a review may be required).
• Prompts (to promote antibiotic review to prescribers).
• Links (to microbiology and/or pathology results during a review).

In each study site, an ePAMS+ Champion formed a local Implementation Team (gives details and the approximate implementation timeline) to promote ePAMS+ within grand rounds, departmental/specialty team meetings, clinical governance meetings and training sessions for junior doctors/nurses/pharmacists. Prescribers, pharmacists and nurses working in study wards were encouraged to complete the ePAMS+ online eLearning training.

Fidelity Index

A Fidelity Index was developed to capture the extent to which prescribers applied ePAMS+ core principles. Assessing fidelity helps increase confidence that changes in outcomes are attributable to the intervention and that behavioural interventions are delivered as planned.23,24

Through Cerner EPMA system data extracts, the Fidelity Index aimed to address three areas:

1. It explored the critical decision-making points for prescribers: these are important for outcomes evaluation in the future confirmatory trial, as they would serve as intermediate outcomes to help explain the relationship between the outcome and the intervention.
2. It enabled understanding of the data structure to facilitate automated coding of fidelity coding.
3. It allowed development of individual and composite scales for capturing practice, to measure the actual implementation of the ePAMS+ intervention as compared to the intended. The scales will combine into a cumulative score for intervention fidelity.

After developing the Fidelity Index, its automation within Cerner EPMA systems and methods of deriving summary measures of antimicrobial use were pre-tested in the feasibility trial.

Quantitative component

We aimed to study at least 100 admissions per ward, to enable precise estimation of between-patient variability, by ward and overall, in antibiotic use and allow exploration of the feasibility of data extraction across a variety of clinical scenarios. Individuals eligible for inclusion were aged ≥ 16 years, were admitted to hospital as a medical inpatient and had an antibiotic order within the EPMA. All eligible admissions to participating wards in study sites were included. Although patient informed consent was not required, the implementation pack contained a patient information leaflet to assist clinical staff in explaining the process of antibiotic use and review.

Under the ePAMS+ intervention, prescription of antibiotics automatically flagged within the EPMA system to trigger decision aids and task lists for appropriate antibiotic management. Report Supplementary Material 3 included detailed information on how order plans, prescribing interventions and review processes were managed within the Cerner EPMA system using ePAMS+.

We extracted two types of data: outcomes for quantitative analysis purposes, such as data contributing to calculation of total antibiotic use; and process of care measures to build understanding of how the ePAMS+ system was used.

Deidentified data extracted from participating NHS Trusts were transferred via secure file transfer protocol (SolarWinds^® Serv-U^® FTP Server and SolarWindss^® Serv-U^® Managed File Transfer (MFT), Austin, Texas, USA) to the National Safe Haven maintained by Public Health Scotland. All data were held in a project-specific area in the National Safe Haven, with access limited to named project researchers via a unique username and multifactor authentication. The National Safe Haven reviewed all outputs to ensure these would not disclose the identity of any participant. ePAMS+ training information was recorded on the Learning Management System (professional discipline, date/time of module completion, time spent on learning, pre- and post-test scores).

The primary feasibility trial outcomes were: (1) to determine the number of defined daily doses (DDDs) per admission, and to estimate the variability in this outcome; (2) to assess the feasibility of standardised queries to capture data from local Cerner EPMA system configurations and completeness of the data extracted; and (3) to assess the feasibility of measuring defined outcomes such as mortality at 30 days post admission. From a consensus building workshop (see Work package 2a), we developed candidate secondary outcome measures to be used in the feasibility and large-scale trials.

The remaining quantitative outcomes were assessed according to: (1) whether it was possible to derive each outcome using the information available in the EPMA data extract and (2) descriptive summary data, overall and by site and by ward, with particular attention to the frequency of missing data. Additional details in association with WP2 are provided in Report Supplementary Material 3.

Key findings

Work package 2a

The proposed process and outcome measures from the consensus and researcher-led activities in this WP taken forward to the feasibility trial consisted of a subset of:

• Quality indicators [mortality at 30 days post admission; adherence to clinical guidelines; clinical failure rate; re-admission rate; resistance rates; acquisition of multidrug resistant organism; incidence of healthcare-associated infection (HAI); i.v. to oral switch; oral to i.v. switch; expert referral rate; documentation of indication; documentation of duration; documentation of stop/review; documentation of decision-making; time to administration; time to active therapy (first dose); disposition; incidence of adverse drug reaction; number patients discontinuing therapy; susceptibility].
• Quantity metrics (LOS; days of therapy; diagnostics; antibiotic expenditure; number of antibiotics prescribed; time to blood cultures; time to organism identification; time to positive culture; rate of diagnostic imaging utilised; rate of laboratory tests ordered).
• CDS (CDS ‘work around’; CDS alert frequency; CDS alert over-ride; use of CDS order set; number of alerts).
• Time spent prescribing.

An online learning tool was created to develop knowledge relating to AMS and to communicate information about the EPAMS. Some text in this section has been reproduced from Cresswell et al.18 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

• Online AMS and ePAMS+ training (approximately 30 minutes of learning):
  • Hosted online using an NHS-approved URL.
• The online module had the following learning outcomes:
  • List the factors to consider when initiating antibiotics, switching the route of administration, and stopping treatment.
  • Discuss the risks of staying on antibiotics for longer than is clinically needed.
  • List the tools provided by the ePAMS+ intervention in this hospital.
  • Explain how order plans work and how these have been set up to aid your decision-making.
  • Explain the components of the ARK decision aid and how these can encourage good stewardship within ePAMS+.
  • A pre-/post-test was developed so learners can assess baseline knowledge and knowledge acquisition.
• The module comprised videos of the ePAMS+ tool to explain to learners how the tool is accessed and utilised in practice.

Work package 2b

Quantitative evaluation

Cerner EPMA system data extraction took place between 1 September 2022 and 7 November 2022, during which period many admissions were recorded. For practical reasons, the ePAMS+ intervention was activated across study wards in a staggered manner from 5 October 2022 to 7 November 2022. A total of 1958 admissions (on 1768 patients) were studied, of which 1256 were completed prior to ePAMS+ and 587 after the ePAMS+ intervention had been introduced. About 1659 admissions had a full 30 days of follow-up to allow the 30-day mortality outcome to be recorded.

We recorded 24,884 antibiotic orders across the 14 study wards. Similar numbers of admissions in women and men were included: 706 females and 652 males before the ePAMS+ intervention and 321 and 279, respectively, after the ePAMS+ intervention. Median age in pre-intervention admissions was 71 years; post intervention, it was 72 years.

Fidelity Index

Development of the Fidelity Index (objective 6) was not fully supported by the data extract due to very few uses of ePAMS+ order sets. We did, however, identify aspects of intervention fidelity that are critical for prescriber decision-making when applying the key principles of ePAMS+. We were able to define proxy measures (often based on date and time) which link to other variables to determine intervention fidelity. Diagnostic confidence and other important fidelity features were not fully examined due to there being few records of the ePAMS+ review tool being used (28 review records from 600 admissions). Data extracts lacked information on clicks to external links for ePAMS+ guidelines or training that would have allowed us to determine if these were consulted during prescribing. As clicks to website links from Cerner were handled through generic user accounts, we were unable to map these to the prescribing practice of individuals. Similarly, it was impossible to extract data for the reasoning behind any action of by the prescriber following antibiotic review, as this information was held in a free-text field in the EPMA system.

Limitations

Work package 2a

While there was a lot of enthusiasm for supporting the development of secondary outcome measures for the trial, and the initial exploratory round of the eDelphi exercise generated many suggestions, subsequent engagement with assessing outcomes was limited. In view of this, the research team had to determine the outcomes to be taken forward from the potential 81 indicators identified to ensure that even for the feasibility trial, the number of process and outcome measures to be investigated were feasible.

Work package 2b

We showed in this feasibility trial that early evaluation of intervention implementation (in parallel with WP1, phase 2b) can highlight the degree to which a complex intervention has been implemented as planned.

A limitation of the trial was that we delivered it in a single NHS Trust, reducing our opportunities to learn about intervention implementation across a range of clinical contexts. Although we partially offset this weakness by studying as many wards of different types within the trust, nevertheless we only evaluated feasibility in two hospitals, and other care settings may differ.

Since we performed a feasibility trial, this means that we cannot draw conclusions from its data about the effectiveness of the ePAMS+ intervention. We also did not study a large enough number of sites to enable clustering effects in study prescribing outcomes to be determined. Had these been available, they would have provided useful information for the design of any future effectiveness evaluation of ePAMS+. The limited uptake of the intervention showed a clear area for ePAMS+ implementation improvement going forward.

Some process measures (such as use of CDS order set and documentation of duration) could only be recorded while the ePAMS+ intervention was switched on. The review process was rarely documented, with few records containing data on stopping of therapy, review of therapy or decision-making, indicating infrequent use of the ePAMS+ intervention review component in the trial.

Relationship with other work packages

• Work package 2a: the development of secondary outcome measures and development of the training materials were undertaken in parallel with WP1, phase 1 and phase 2a, reported in the relevant section above.
• Work package 2b: the feasibility trial was undertaken in parallel with the qualitative research of WP1, phase 2b, reported in the relevant section above.

Work package 3: evaluating the effectiveness of ePAMS+ in safely reducing inappropriate antibiotic use

Aims

• To explore, using an internal pilot study, the implementation of ePAMS+ and trial data collection processes.
• To determine the effectiveness of ePAMS+ in reducing total antibiotic consumption in adult medical inpatients compared to usual care (UC).
• To establish whether the ePAMS+ intervention showed non-inferiority to UC for mortality in the 30 days from admission.
• To quantify the impact of ePAMS+ on secondary clinical, safety and process outcome measures.
• To investigate the mechanisms by which ePAMS+ exerts any of its effects and to explore factors that may differentially influence the effectiveness of ePAMS+, drawing upon findings from the parallel longitudinal evaluation of the pilot and trial implementation of ePAMS+ (being undertaken in WP1).

Methods

Design

We planned a hybrid cluster-randomised stepped-wedge trial incorporating an internal pilot study, to be undertaken in 10 hospitals with established use of the Cerner EPMA system. The hybrid cluster-randomised stepped-wedge trial design evaluates whether the ePAMS+ intervention is superior to UC in reducing overall antibiotic prescribing and non-inferior to UC regarding mortality in the 30 days post admission. ePAMS+ would be considered successful only if it significantly reduced antibiotic prescribing without a deterioration in 30-day mortality.

The hybrid cluster-randomised stepped-wedge design means that the intervention would be implemented in one hospital from the start of the study (‘parallel treated’) and phased into eight hospitals at evenly spaced steps during the trial; one hospital will not receive the intervention (‘parallel control’).

Our proposed stepped-wedge trial had a continuous recruitment, short exposure design, classified as T4/D2/M4 according to the framework of Copas et al.25 We also included a hybrid element in the design (see Girling and Hemming26) to optimise efficiency and statistical power.

Study population

Adult medical inpatients were to be included in the trial.

Interventions

Usual antibiotic prescribing practice in study sites was to be the comparator group in the trial. The originally planned ePAMS+ intervention is described according to the headings of the TIDieR framework in Report Supplementary Material 4.

Outcome measures

• The planned coprimary outcome measures were:
  • Average total antibiotic consumption as measured by DDD per 1000 admissions.
  • 30-day mortality.
• Other secondary clinical, safety and process outcomes were specified through the consensus workshop (WP2a) and tested in the feasibility study (WP2b).

Sample size

Based on admissions data from NHS England, the expected sample size of a 24-month 10-site study would be 642,960 admissions in total. A 5% mean reduction in the antibiotic use coprimary outcome due to the ePAMS+ intervention was considered clinically important.

For the 30-day mortality coprimary outcome, the background 30-day mortality rate in the trial population was assumed to be 9%. A non-inferiority test for mortality (margin 0.45%) would have 85% power in the eight sites in the stepped-wedge component of the planned hybrid design.

Randomisation and blinding

Hospital sites would be randomised to parallel treated, parallel control or stepped-wedge in the ratio 1 : 1 : 8, and for sites randomised to the stepped-wedge part of the design, the step at which ePAMS+ was implemented in each hospital would be allocated at random.

Hospitals would not be blinded to the assigned group or time point.

Data collection

As in the WP2b feasibility trial, individual patient-level demographic data and study outcomes were planned to be extracted from the Cerner Millennium EPMA system via a general query and then filtered according to the study entry criteria.

Ethics

Consent would not be sought from individual patients for this trial but will be sought instead from the ‘gatekeeper’ of each site cluster. No identifying information on participants will be held within the study database; similarly, no participant will be identifiable from information in trial publications.

Reporting

The trial would be reported according to guidance in the cluster-randomised trials extension to the 2010 Consolidated Standards of Reporting Trials statement as well as published recommendations1 for reporting stepped-wedge trials. Additional details in association with WP3 are provided in Report Supplementary Material 4.

Key findings

Not applicable; WP3 not completed.

Limitations

Not applicable; WP3 not completed.

Relationship with other work packages

Delivery of WP3 depended on the outputs from the feasibility trial of WP2b, in particular the identification of the required number of sites in which to run the hybrid stepped-wedge cluster-randomised trial. Due to insufficient sites being identified and the challenges in ePAMS+ implementation found in WP1 (phase 2b) and WP2b, WP3 did not proceed. WP3 would also have contributed additional data to the cost-effectiveness evaluation of WP4.

Work package 4: economic analysis of Antimicrobial Stewardship Programmes

Some text in this section has been reproduced with commission from Nayyar et al.4 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Aims

The overall aim of WP4 was to build a generic economic model that estimates the costs and benefits of Antimicrobial Stewardship Programmes (ASPs) taking a health services perspective throughout. Specifically, we sought to:

• Conduct a scoping review of cost-effectiveness studies of ASPs.
• Develop a causal model to represent the relationships between AMS interventions and outcomes through mediating variables.
• Review the literature to retrieve estimates for the parameters needed to populate the model above.
• Populate the model using estimates (and plausible ranges) from the literature review, supplemented, where necessary, by expert opinion.
• Consider our findings and the limitations of our approach.

Methods

Scoping review of economic evaluation studies of Antimicrobial Stewardship Programme

Here we report on a scoping review of the literature of health economic studies of AMS programmes with the following objectives:

• determine the type of data that have been collected
• determine the types of economic model used
• examine the perspectives taken
• better understand methodological issues presented in this field
• assemble information on the estimates of cost-effectiveness
• provide guidance to future projects aiming to estimate the value for money of digital ASP.

We planned to include:

1. simple measures of cost per unit of clinical effectiveness
2. cost–utility
3. cost–benefit, where monetary values are ascribed to improved clinical outcomes.

We did not include studies based on therapeutic drug monitoring or complex drug monitoring regimes designed to avoid toxicity while maintaining effective antibiotic concentrations.27

Key findings

Types of study

We found six studies that fulfilled our criteria in the cost-effectiveness category and two in the cost–utility category. We found no cost–benefit studies as defined above [although we did find a cost-effectiveness study (Slayton et al.28) that was described as a cost–benefit study by the authors].

Cost-effectiveness studies

The six studies that included the intervention cost and at least one clinical effect are described in Report Supplementary Material 5, Table 9. The table includes the single randomised trial recorded in our review. This was a stepped-wedge cluster trial conducted in nine hospitals. There were four observational studies and one purely literature-based study.29 The intervention dominated conventional care in two of the six studies,28,30 that is, it reduced overall cost net of the intervention cost and improved health outcomes. Four studies used number of infections as an outcome measure, but not consistently: for example, Slayton et al.28 only considered Clostridioides difficile infection (CDI), while Lin et al.30 used all HAIs.

Cost–utility studies

The two cost–utility studies reported here are both associated with the use of ASP for treatment of bloodstream infections. Types of costs included in the analyses were hospital costs, blood culture costs, costs of implementing the computerised clinical decision system and staff time costs. The effectiveness parameter was measured as quality-adjusted life-years (QALYs) in both studies. The intervention dominates the control in Gebretekle et al.,31 with a saving of £1700 per QALY gained. The intervention reported by Scheetz et al.32 was highly likely to be cost-effective, with a cost per QALY gained of £1500.

Development of a causal model that can be used to evaluate the cost-effectiveness of different Antimicrobial Stewardship Programmes

Model development

Methods and model structure

The high-level model structure is shown and detailed in Report Supplementary Material 5. This model was built on the basis of:

1. the literature
2. team meetings in the ePAMS+ programme.

Model coproduction meetings held at the Royal College of General Practitioners on 8 March and 15 December 2023.

Cost

The cost–consequences of ASPs, as described in the literature, can be direct or indirect. By direct, we mean changes in costs of drugs, the costs contingent on i.v. – rather than oral – therapy and the cost of the intervention itself (fixed and recurrent). By indirect, we mean the costs that arise from changes in clinical outcomes – costs that are largely captured as those of LOS (bed-days).

Limitations

Intervention effects can ‘bypass’ each node in the causal pathway. To put this another way, there may be causal pathways that are not evident to the modeller. As with all models, ours does not encompass all possible causal factors, but we have consulted widely regarding the relationship between antibiotic stewardship and infection control. As stated, we do not include the original prescription as part of the ASP intervention. We thus conceptualise ‘stewardship’ as the measures that are taken once a patient has been started on antibiotics. We regard the quality of the original decision regarding making the correct diagnosis and prescribing the correct antibiotic for that diagnosis as a separate topic covered by a separate literature.

We have excluded externalities such as the carbon footprint associated with antibiotics, the need to develop new antibiotics as resistance grows and societal impacts such as productivity losses associated with antibiotic-related morbidity. Valuing all plausible externalities and wider costs and benefits of ASPs is an important next step, but one we cannot tackle at this stage.

Despite these limitations, we believe that the model provides a helpful representation of intervention effects and was ratified by the management group of the ePAMS+ study.

Discussion and key findings

Conceptualising the aims of Antimicrobial Stewardship Programme

The above CDC definition noted that an ASP has two aims:

1. reducing antimicrobial resistance (AMR)
2. reducing risk/improving safety for the individual patient who is exposed to a more appropriate profile of antibiotics than would otherwise have been the case.

The main aim of antibiotic stewardship is the first of the above aims – reducing antibiotic resistance in the environment. This includes the hospital environment and the broader environment. Reducing antibiotic use is associated with reduced ‘load’ of resistant organisms such that, for example, a person who acquires an infection in hospital is less likely to have a resistant infection if the overall use of antibiotics is curtailed. We classify the benefit that people (in the hospital and beyond) receive from a general lowering of antibiotics as the societal benefit of ASPs. It is not considered here since we do not know how to estimate parameters of societal benefit from reduced antibiotic prescribing, and for this reason, we specifically excluded consideration of this important issue. We did, however, promise to keep abreast of literature in this field and will return to this topic later.

This analysis has focused on the individual benefit that might accrue to an individual whose personal antibiotic exposure has been modified as a causal effect of an ASP. Such benefits can be achieved through three causal routes:

1. Reduction in ‘super-infection’ (opportunistic infection by an agent that is normally held in control by other micro-organisms that are inactivated by antibiotic agents). Clostridioides difficile is the most important of these opportunistic infections, but there are other candidates not considered, specifically Candida albicans or ‘thrush’, which is usually an infection that does not cause sufficient disutility or cost to impact on our model.
2. Changing from i.v. to oral therapy, which has numerous advantages in terms of cost (including staff time) and patient comfort/mobility. It also has some health benefits in terms of phlebitis at the cannula site, but the evidence is neutral (or balanced) as far as overall health benefit is concerned (as stated).
3. Switching to a more appropriate antibiotic in the case of bug–antibiotic mismatch.

It may be fairly pointed out that our classification of social and presumed benefits is not hermetic, since it has to be considered that a potentially infectious organism, residing in a person not infected by that organism, may nevertheless come to infect that person – say because their immunity has been lowered. We recognise this as a possibility but simply have not felt that we are in a position to address this theoretical possibility – again we will return to this issue. Additional details in association with WP4 are provided in Report Supplementary Material 5.

Limitations and future directions

The main limitation of our model is that it does not include the societal benefit of reduced contamination in the environment, nor does it include the wider economic impact of reduced exposure to antibiotics, resulting in reduced antibiotic resistance. As promised, we have reviewed the literature and consulted experts on this point. We find no clear evidence of the extent to which a given reduction in antibiotic use reduced the population prevalence of resistant organisms and the contingent effects on population health. We think that a model could be produced to explicate the causal pathway linking ASPs to population health, limiting it to the hospital environment, at least in the first instance.

Relationship with other work packages

The original purpose of the programme was to develop a complex intervention to promote AMS and then to evaluate this. As explained elsewhere, the pilot and definitive trials turned out not to be possible. Moreover, even if it had been possible, the findings would not be directly applicable to new contexts where costs might differ, and effectiveness may vary. Our model, on the other hand, can be used across different contexts to inform likely cost-effectiveness on a local basis. Our findings have another purpose. They show that computer-based ASPs are likely to be cost-effective at very low absolute values in terms of health gain – absolute values that could not be detected by empirical studies of any obtainable size – in a situation referred to as an ‘inconvenient truth scenario’.33

Work package 5: antimicrobial stewardship qualitative survey

Aim

To help contextualise our experiences of the ePAMS+ programme, we explored various experiences of implementing AMS interventions in English hospitals.

Specifically, we sought to:

• produce a framework of ePrescribing-based AMS functionality adopted in English hospitals, describing what works for whom and in what circumstances
• explore strategies used and their perceived effectiveness to promote AMS in English hospitals
• explore experiences and transferable lessons learnt in relation to development, implementation and adoption
• identify strategic priorities for the NHS to ensure ePrescribing systems are used to support AMS in NHS hospitals.

Methods

Our evaluation of the feasibility implementation found that ePAMS+ was potentially feasible to deliver, but its implementation would need to be carefully adapted to the local care setting. During the COVID-19 pandemic, however, many trusts had implemented ePrescribing-based AMS functionality in their systems, elements of which overlapped with ePAMS+. As a result, we struggled to recruit sites for the originally envisaged stepped-wedge trial to formally test the wider acceptability, effectiveness and cost-effectiveness of ePAMS+.

As there was no overall understanding of what functionality has been implemented in differing hospital settings, and what ePrescribing-based approaches to AMS work best for whom and under what circumstances. Although there was some work underway nationally to identify what functionality could deliver improvements,33 there was no systematic assessment of the range of systems, interventions and contexts in which this could be achieved.

Sampling

We built a purposive sample of hospitals encompassing various AMS strategies, ePrescribing and AMS functionalities to explore their experiences of developing, implementing and using the AMS systems. We sampled AMS leads at each of the hospitals. Sampling of participants was facilitated by our existing contacts and built on our previous ePAMS+ work. Our purposive sampling sought to ensure maximum variation in terms of demographics (geographical location, gender, ethnicity, experience with ePrescribing and AMS) and system type (integrated/stand-alone) and system vendor. We actively sampled a range of viewpoints to get a rounded understanding.

Data collection

We conducted semistructured online interviews. These were audio-recorded and professionally transcribed.

Analysis

We analysed the data thematically using the TPOM Factors Framework as a coding framework.33 We explored respondent experiences with AMS systems and strategies and what had and had not worked for them. Based on this, we extracted lessons for ePrescribing-based AMS interventions across a wider range of settings, documenting what is likely to work for whom and under what circumstances. We also sought to understand future strategic priorities and challenges and possible ways to tackle these.

Qualitative analysis was iterative, allowing emerging themes to be explored further and disconfirming evidence to be sought. Thematic analysis allowed exploring various perspectives and contexts. We identified overarching themes, similarities and differences, and potential implications for development, implementation, adoption and optimisation strategy. Data were analysed both deductively (using a codebook based on the TPOM framework) and inductively (to identify novel information and unanticipated themes). This allowed us to explore how strategies and experiences vary across systems, adoption contexts and organisations.

Key findings

We interviewed 38 participants from 23 hospitals in England.

Our work confirmed that multifaceted interventions, including behavioural, technological and organisational components, were perceived as having most potential in addressing issues surrounding AMS across organisations. One of the most important technological factors was the availability of real-time data of individual and group antibiotic prescribing. Interprofessional relationships and education were viewed as the most important factors in promoting AMS, independently of the ePrescribing system.

Interprofessional relationships and education to promote antimicrobial stewardship

Microbiologists worked closely with AMS pharmacists on antibiotic review conducting ward rounds with them either in person or remotely, or a combination of both. AMS ward rounds to discuss appropriate antibiotic prescribing were viewed as an opportunity to educate prescribers.

The ward presence and the ward rounds and what that brings with it is every patient is reviewed in a timely manner. You’ve got that presence. You’ve got that education and training that happens as part of those ward rounds. You know, because having those discussions, it builds on the prescriber’s knowledge as well and allows them to be able to take that knowledge and potentially use it again and again and again.

Depending on the size of the hospital, the clinical mix and the number of AMS pharmacists, the reviews were undertaken at different time intervals. Some wards, such as the intensive care unit, may be visited every day.

The culture in an organisation towards AMS, that is, beliefs about AMS and individual responsibilities concerning AMS, was viewed to underlie AMS practices. Education, both formal and informal, was regarded as an important part of not only imparting knowledge but also creating a culture which promotes AMS. The movement of doctors around the healthcare system, however, could make formal education difficult.

I think the problem is that at the moment, we do not have a culture of awareness, we don’t have a culture of antibiotic stewardship. Yes, the trust recognises the importance but really nobody else does. And people […] much like infection control, people see antibiotic stewardship as the problem that belongs solely to the antibiotic stewardship committee. And antibiotic resistance is also […] that’s their problem, not mine. Yeah, it’s their job to solve that problem and give me an antibiotic I can use. That culture […] before we’re going to get anywhere with this, that culture has to change.

Microbiologists usually restricted the reporting of all the antibiotics an infection was susceptible to. This inhibited clinicians from prescribing high-risk antibiotics where these were not necessary. Some limited reporting to one antibiotic and an alternative for penicillin-allergic patients; others provided several antibiotics the infection was susceptible to. The number and type of antibiotics reported were also determined by the drugs available at that time in the hospital formulary, local AMR rates and the patient’s condition. Clinicians knew they could contact the microbiology team or the AMS pharmacist if they sought further advice on the list they had been given.

We try to utilise as different of a variety as we can. So, obviously there are things that we would consider for example, people’s age, biochemical markers, comorbidities […] If it’s a chest infection […] we look at our antibiogram to say, okay, most of our chest infections are caused by this bug, so we want to use this narrow spectrum antibiotic […] so in [area] the resistance rates are the highest in the country.

Clinicians accessed microbiology results through systems that were either a separate digital system easily accessible through a tab on the ePrescribing system or through a separate system which they could not access through the ePrescribing system.

So, when we were using [EPMA], then our pathology results would come into [EPMA]. Very difficult to view it, not ideal, but I’ve got used to it […] I mean, you can see it in [EPMA] but there’s multiple places you have to look for various different things. It just irritated me.

All participants reported that a positive relationship between ward staff and Pharmacists and Microbiologists was crucial to promoting AMS. Visibility on the ward and informal ad hoc encounters were viewed as particularly important in this context.

So, just practically wandering around the hospital bumping into people is a good thing […]. So, people will know you – opportunity to just talk about, I don’t know, the weather or whatever or patients or, you know, just have just normal conversations […] I think that’s more valuable than actually talking about patients, because a lot of this is based on personalities, isn’t it? It’s more of a marketing […] stewardship to me is a game of marketing more than a game of medicine.

So having done it for 20 years, the thing that I think makes differences is the relationships that you build with a clinician and actually undertaking those ward rounds and being there and teams expecting you to be there to question what they’re prescribing. Also, to provide the support for what you’re prescribing. Because I think that’s a really important aspect, and I think AMS teams that don’t have that relationship […] will really struggle to effect change and build the confidence. It’s about building the confidence to clinicians that you’re not there just because of the C diff rates. you’re actually there to improve the outcomes for patients and support the team.

Pharmacists and microbiologists also often joined multidisciplinary teams (MDTs) or set them up when more intense and targeted education was required. In large hospitals, with many sites, pharmacists had to prioritise the specialties most in need of AMS recommendations. Embedding an AMS pharmacist in a MDT was viewed as having significant potential to improve antibiotic prescribing.

So, when we’re quite embedded into a team, so when we’re working with haematology, when we’re working on the acute medical units and we’re embedded in the team, they learn from us and are able to make those decisions, improve their prescribing. And we have seen improvements when we’ve given them quite intensive support.

Interprofessional relationships were also viewed as important in relation to nursing staff. Administering i.v. antibiotics takes considerable nursing time, and two nurses have to be present. Nurses, therefore, have a powerful motivation to encourage i.v. to oral switch, where appropriate, substantially improving AMS.

Who does see every patient every day are the nurses. It’s also in their interests to stop drugs that are not needed because it’s less time for them to administer it, it’s less harm, it’s less work, and so on and so forth. It’s also in their interest to de-escalate a patient from IV therapy to oral. So actually, part of stewardship is also a nursing responsibility.

One hospital had focused on training nurses to have the confidence to question doctors on the necessity of i.v. antibiotics.

This is how we get the nurses on board because we go look, […] that’s something like three, four, five, six, seven IVs a day for one patient. The patient doesn’t need them […] So, we’ve got nurses now that are actively on their daily ward rounds that they have with the MDT where they’re going ‘Does this patient need to be on IVs anymore?’, because they don’t want to give seven IVs or three IVs when they could give an oral and stuff. So, we’ve tackled the nurses as well, so they’re proactively asking that question. And obviously some are better at it or more confident than others at doing that.

Nurses could previously see antibiotic prescriptions on paper charts. However, on the ePrescribing system, they could only see the next dose of antibiotic, not when the course ends. If they could see the full prescription on the ePrescribing system, they could have an important role in alerting doctors to the ‘drop-off’ of antibiotics.

So, the nurses’ aspect is another way where I feel the system is working against us, in a way, that the nurses’ view doesn’t give enough information about the antibiotics.

Conclusions

We have found the 23 English hospitals studied had adopted a wide range of technological, educational and behavioural interventions to promote AMS. Many of these resembled elements of the proposed ePAMS+ intervention, in various combinations. These had often been developed locally and therefore lacked an evidence base. There is, therefore, an urgent need to focus on technological, social and organisational AMS interventions as a whole and creates an evidence base. However, commercial ePrescribing systems are different, and each system is amended to varying degrees to suit local requirements. The context into which AMS interventions may be rolled out is no longer a tabula rasa. This presents a potential barrier to uptake (as the outcome of the ePAMS+ project demonstrates) as well as a challenge to efforts to measure summative impacts.

Limitations

The short (6 months) timescale of the WP5 study made recruitment difficult due to the work pressures of relevant HCPs particularly acute in the post-pandemic restriction period. Invitations were specifically sent to AMS leads in the trusts and enabled discussions with a range of specialists in pharmacy and microbiology and beyond. Due to time pressures, it was not feasible, however, to include the views of prescribers on the wards.

This analysis did not include all hospitals, again, due to time constraints. However, the study provides many valuable insights into practices. It highlights the need for further research in particular to develop a stronger evidence base for different AMS practices and to assess the impacts of a comprehensive, complex intervention like ePAMS+. Additional details in association with WP5 are provided in Report Supplementary Material 6.

Relationship with other work packages

This was an additional piece of work to explore applicability of the ePAMS+ intervention. It builds upon the work done in WPs1–4 but was conducted after their completion.

What was successful and not successful in the programme

This was a very complex programme to deliver, for reasons almost entirely out of our control. A summary table with agreed key deliverables and its outcome is included in (Appendix 1, Table 1). During the first year of the programme (i.e. 2019), our research was progressing according to plan. This all, however, changed from 2020 onwards, chiefly as a result of the pandemic and the associated issues of:

• the principal investigator (PI) needing to be deployed into a number of UK and Scottish governmental COVID-19 committees
• the co-PI and other clinical members of the team returning to front-line clinical roles
• members of the team developing COVID-19 (and other health issues) and/or needing to care for family members
• some members of the team needing to take on daytime caring responsibilities while schools were shut
• suspension of non-COVID-19 research in NHS sites
• restrictions of access to NHS sites, lack of NHS staff capacity to participate in qualitative work, and social mobility more generally
• COVID-19 NHS-related backlogs.

These challenges were then compounded by repeated NHS strikes and the associated worsening NHS pressures, making it difficult for NHS colleagues to find time to engage with the programme.

That said, we were still able to achieve a great deal. Our key achievements include:

• recruiting and maintaining a genuinely interdisciplinary team of experts focused on one of the most pressing issues facing the NHS, namely AMS
• recruiting and maintaining our Independent Program Steering Committee (IPSC), which comprised of leading authorities in relation to sociotechnical approaches to promoting AMS
• recruiting and meaningfully engaging patient and public involvement (PPI) colleagues in all aspects of the work
• undertaking extensive qualitative work to understand the NHS inpatient AMS landscape (prior to the pandemic) and using these insights to inform the development of ePAMS+
• expert workshops with clinical, academic and technical colleagues to develop ePAMS+
• agreeing the primary and secondary outcomes of interest for the programme and demonstrating that it was possible to extract valid data on many of these
• undertaking a feasibility trial of ePAMS+ in two NHS hospitals with accompanying formative qualitative and process evaluations
• successfully undertaking secure transfer of feasibility trial data from the hospital sites to the University of Edinburgh’s safe haven
• undertaking a systematic review and organising expert meetings to develop a causal framework and economic model to assess the cost-effectiveness of ePAMS+
• contextualising our programme by undertaking a qualitative scoping assessment of current AMS strategies being pursued across a range of NHS hospitals in England
• published five papers in respected international peer-review journals
• ongoing open, honest discussion with our funders about the challenges we were encountering.

The key issues that we were not able to achieve were:

• engaging as extensively as we would have linked, in person, with key NHS staff
• mandating the use of ePAMS+ in the feasibility sites
• being able to extract valid data on all of our secondary outcomes of interest
• recruiting the additional hospital for the planned pilot trial
• formally recruiting sufficient number of hospitals into the randomised stepped-wedge evaluation to make this a viable proposition
• delivering the pilot and definitive trial of ePAMS+
• delivering a cost-effectiveness evaluation of ePAMS+.

Patient and public involvement

Our PPI strategy aimed to conduct research in the NHS with awareness of what is important from a lay perspective – whether what we plan to do makes sense, what is missing, and whether the way we present or disseminate what we have done is comprehensible and meaningful to health service users and members of the public. This might particularly relate to the patient/carer facing aspects of our work, but not exclusively.

Our PPI strategy was an important, cross-cutting component of the programme, with lay representation at every level of the research to contribute views, questions and priorities from the perspective of health service users and the wider public.

Our lead lay member, Antony Chuter, was a co-applicant on this programme grant, and he is supported by a second lay member, Jillian Beggs. Chuter is an experienced trustee and chair of Pain UK, and chair of the Royal College of General Practitioners Patient Partnership Group. As well as being lead PPI and co-applicant for this programme, Chuter had similar roles on a number of programme grants and studies at the University of Nottingham, University of Birmingham and Cardiff University. Beggs had supported PPI activities for over 12 years and supports additional PPI projects at the University of Nottingham, University of Dundee and the Royal College of General Practitioners. They have worked individually and collectively to contribute lay perspectives across all aspects of the programme.

The PPI members participated in the:

• kick-off face-to-face meeting in November 2018
• monthly Programme Management Group meetings since February 2019
• WP1 Intervention Workshops in January and May 2019
• stakeholder workshops in June and November 2019
• Intervention development group teleconferences
• IPSC.

A PPI representative was invited to shortlist and interview candidates for the WP1b Research Fellowship position at the University of Edinburgh in February 2019, and for the WP2a researcher position at University of Birmingham in early 2020. PPI members were actively involved in reviewing the documents for the WP1 qualitative studies and WP2b feasibility trial ethics application. They contributed to the intervention development plans. They were also actively involved in discussion around data (anonymisation, protection, transfer, etc.) in collaboration with all WPs. PPI members supported development of plain English summaries of ePAMS+ publications. PPI members contributed to the study design and advised on the lay summary of the feasibility trial protocol. Our PPI members were instrumental in recruiting two experienced lay representatives with an oversight role on the IPSC. We have a programme website (www.ed.ac.uk/usher/eprescribing-antimicrobial) that can be accessed by members of the public. It gives details of all aspects of our research work and has a section on PPI within the programme, including a named contact and details for anyone wishing to get in touch.

Equality, diversity and inclusion

We followed EDI policies and principles wherever possible. Considering the fact that COVID-19 pandemic has seriously affected our study plans, we were limited in many areas.

Recommendations for future research

We still need to find ways of safely reducing the considerable burden posed by AMR, and focusing on hospital inpatient settings and drawing on electronic systems is an important dimension to this. More specifically, work from this Programme Grants for Applied Research (PGfAR) has identified the following key research gaps:

• Thoughtful consideration on the most appropriate study designs to enable the effectiveness (and cost-effectiveness) of technology-based interventions to be established, particularly given the lengths of time inherent in doing the groundwork for and delivering randomised controlled trial evaluations.
• Developing national agreement on key outcome measures of interest and demonstrating that it is possible to extract valid data on these.
• Development of a typology of the range of approaches being used to promote AMS in hospitals and undertaking evaluations of their likely effectiveness and cost-effectiveness.
• Development of methods to extract indication data from free-text records in EPMA systems, automating the extraction of indication-based prescribing outcomes that are required in the evaluation of AMS interventions.
• Development and validation of a scoring scale to quantify, in relation to intervention fidelity, the series of critical decision-making points for prescribers identified during the feasibility trial of this PGfAR.
• Implementation of a Core Outcome Set to build consensus on the suite of outcomes relevant to patients, service providers and other stakeholders, that should be measured in AMS intervention evaluations.
• Developing an agreed approach to modelling the data to assess effectiveness and net costs.

Implications for practice

Antimicrobial stewardship remains a critically important issue for health systems – including the NHS – and society more generally. Medical hospital inpatient settings represent a particularly important target to reduce inappropriate antibiotic prescribing, particularly of broad-spectrum products. The now widespread availability of ePrescribing systems offers considerable opportunities for innovation and service redesign in relation to AMS, which was the focus of this PGfAR. While our work has found that it is possible to create an ePrescribing-based module focused on AMS and that elements of this may prove useful, we have not been able to demonstrate the successful implementation or the effectiveness or cost-effectiveness of ePAMS+. There are, as a result, no direct implications for practice other than to conclude that this initial version of ePAMS+ is not suitable for use in the current NHS landscape. Our modelling, however, shows that under most plausible scenarios, ePAMS+ is likely to be both more effective and less costly than a scenario where no antibiotic stewardship is implemented.