Main findings: study results
Parts of this text have been reproduced from Brierley et al.27 © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted. This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.
Feasibility of setting up a registry using hospital information system data
The outcome measures of the feasibility prospective cohort study were feasibility parameters to establish if (1) consent can be implemented at all four hospitals, (2) data linkage and extraction from multiple HISs can be achieved for > 90% of consented patients at all hospitals, (3) local data can be successfully linked with hospital episode data for > 90% of consented patients at all hospitals and (4) the proportion of patients activating the PPCI pathway who get a CMR scan is ≥ 10% in hospitals with dedicated CMR facilities.
The results suggest that the registry is feasible with respect to points 1, 3 and 4, albeit with some caveats, but not point 2. We found that recruitment and consenting of the patient population (using conventional consent), and linkage of local HIS data with hospital episode/vital status data are feasible. However, we experienced problems with identifying and approaching all eligible patients across all sites and we could not obtain all of the requested data from all hospitals. We had envisaged that it would be relatively straightforward to identify patients meeting the eligibility criteria (i.e. patients who activated the PPCI pathway, including those who had an emergency angiogram but did not receive PCI) consecutively, given the emergency nature of PPCI pathway activation. Indeed, we chose this population for a national CMR registry using routinely collected data because, at the outset, we expected that this population would be easy to identify from hospital databases. Although patients who received PPCI were easy to identify from catheter laboratory databases, not all eligible patients were identified in this way and many were identified manually by the research nurses.
The numbers of PPCI patients identified as eligible and approached at each hospital were lower than the numbers of PPCIs reported to BCIS by each hospital over the recruiting period (varying from 22% to 86% across the four hospitals; see Table 10). Some PPCI patients were missed because they were quickly repatriated to their local referring hospital. Other patients who arrived at the PPCI centre via A&E were not always classified as PPCI (e.g. patients who experienced out-of-hospital cardiac arrest), resulting in many of them being missed.
Patients who activated the PPCI pathway but had an emergency angiogram without PPCI (i.e. those with unobstructed coronary arteries) were difficult to identify from catheter laboratory databases. Contrary to expectation, there was no data field in any database capturing emergency admission to the catheter laboratory. These patients were also not described consistently in HIS databases and there was misunderstanding over eligibility (especially at the beginning of the study) by catheter laboratory staff and research nurses. Therefore, it is unclear what proportion of these patients was identified at each site during the recruiting period and how the proportion may have changed over time.
We met the feasibility objective of implementing conventional prospective consent at all hospitals, although consent rates were lower than anticipated for a study in which participants have no active involvement. The main reason for this was that many patients identified as eligible were missed (because of short hospital stay, no nurse cover at weekends, quick repatriation to referring hospitals, mobile inpatient population, etc.). There was also the issue of competing studies in the hospitals, including the perception that interventional studies should be given priority over observational studies. We relied on postal consent to invite patients identified as eligible and who were not approached in hospital, but return rates were low and we did not have the resources to send reminders or make telephone calls.
We explored alternative methods of obtaining consent as part of the feasibility study. We considered applying for approval to recruit patients without consent [‘presumed consent’, section 251 (of the NHS Act 200676) exemption] but concluded that this proposal would be unacceptable legally, given that, in principle, it is possible for patients to be asked to give consent for their data to be used for the study.73 Exemptions are granted only on a temporary basis (pending implementation of consent) or when consent is impossible, not just infeasible. We considered the option of requesting consent at the same time as requesting procedural consent (e.g. by adding a paragraph to the standard NHS form, either opt-in or opt-out consent), although the emergency nature of the procedure means that written procedural consent is not always obtained and verbal assent is often recorded by hand in the patient notes.27 Furthermore, we were unsuccessful previously when trying to introduce a research consent step into usual care; at the lead hospital, we could not get permission to alter the standard NHS consent form for one specialty and it was impossible to coordinate a change in the written procedural consent across all specialties. Although we were aware of at least one instance when a lead consultant had been able to integrate an opt-out consent model (to allow the use of routinely collected data for research) in the procedural consent form, this instance related to a diagnostic rather than an interventional context. The former setting has a more structured environment with fewer consultants and possibly less time pressure and we considered that an analogous process would be difficult to replicate in the PPCI environment, in which patients are critically ill and staff are under constant pressure to free beds for incoming patients.
Most prospective registries in the UK require explicit patient consent (e.g. National Joint Registry,74 Inflammatory Bowel Disease registry75), using stand-alone consent forms either with/without an approved ‘short text’ for obtaining consent that can be added to local consent forms. Participating hospitals are responsible for obtaining and recording patient consent with their uploads. However, the issue of missing information on consent is recognised and the National Joint Registry, for example, has dispensation [section 251 (of the NHS Act 200676) exemption] to record patient details when the indication for patient consent is ‘not recorded’, although this dispensation is granted on an annual review basis. Other registries, such as the cardiac audits under the umbrella of the National Institute for Cardiovascular Outcomes Research (NICOR) [which includes the Myocardial Ischaemia National Audit Project (MINAP) and the BCIS National Audit of Percutaneous Coronary Intervention (NAPCI)], have exemption under section 251 of the NHS Act 200676 to use patient information for medical research without consent. We cannot discern a rationale for the inconsistency in the consent arrangements for existing registries. Therefore, we are uncertain whether or not a future registry would be granted a section 251 (of the NHS Act 200676) exemption, as it is manifestly possible to implement prospective consent, as evidenced by some but not all existing registries.
A possible solution to the challenge of obtaining consent in the usual care setting would be to implement changes in NHS information technology (IT) administrative systems (ideally at the national level) to allow a record of verbal patient consent to be recorded in patients’ electronic notes (e.g. a ‘consent module’). The use of a system embedded within the NHS administrative systems could be applied more broadly to all registry or similar projects seeking to exploit routinely collected data, so that patients could be easily identified as having given or not given consent, having withdrawn consent, or not having been asked for consent. It could also capture whether or not the patient consents to all data being used and if not, which data are specifically excluded from the consent. However, we are uncertain whether or not verbal patient consent recorded in this way would comply with NHS Digital’s requirement for ‘explicit’ patient consent (the legal basis for accessing HES/ONS data).27,77
Although patients and the public are broadly supportive of the use of electronic patient data for research,78–81 there are concerns about potential breaches of privacy and the misuse of health data.82 The public response to the care.data public consulation83 highlighted important challenges in terms of patient confidence and trust in how electronic health records are used in medical research.84 Newer consent models have been proposed, including a dynamic consent model, which would allow patients to control consent electronically over time and receive information about the uses of their data and up-to-date lay summaries of research projects that have used their data.85 However, such a model would require investment in and maintenance of an electronic infrastructure, which would be difficult in the current economic climate of the NHS. Others have called for more adaptive governance models depending on the particular circumstances of the proposed research.86 It was beyond the scope of this study to explore these possibilities for obtaining consent.27 The difficulties we experienced in identifying consecutive eligible patients and consenting those who were identified in the emergency setting of activating the PPCI pathway highlight that the conventional consent model that we successfully implemented would not be feasible for a national registry.
We did not meet the feasibility criterion for data linkage and extraction (the target was that this should be achieved for > 90% of patients at all four hospitals) from HIS for all the requested data sets. Reasons for missing data included the following: difficulties in extracting data from local stand-alone databases, some of which were old and did not have an interface for data extraction; some data (e.g. medications on discharge, bedside ECHO) were not recorded electronically; imaging data were available electronically but only as free-text reports, making individual parameters of interest difficult to extract. The main impacts for a registry of failing to obtain these HISs is the inability to characterise the registry population in detail, to adjust any comparisons for patients’ characteristics and to describe changes in these data over time (potential indicators of the consequences of important changes in management).
We had initially envisaged establishing a data linkage and extraction model in one hospital and using this to guide data extraction at the other hospitals. We also envisaged that this model would be automated, with minimal IT support beyond the set-up phase. However, this was not possible because HIS for each data set differed across hospitals and different hospitals provided different levels of start-up IT support for the research project.27 We budgeted for only a small amount of funding (£5000) to each hospital to help with this aspect of the study. Data linkage requires expertise in several areas, including knowledge of the databases to be linked and skills in creating linkage programmes. This expertise was variable across hospitals and in-depth knowledge was sometimes lacking; this was perhaps not surprising given the diversity of the databases involved, and the fact that some of these were old and were being phased out.27 We relied on each hospital identifying key individuals to help with this aspect of the study. In addition, even in hospital A, the highest priority for the IT team was to maintain the integrity and functionality of the clinical care systems; in this context, it was difficult for them to prioritise our request to support this research project, even with the funding that was available.
The complexity of data retrieval from multiple HISs in secondary care in the NHS has been highlighted previously.61 The government’s vision for the NHS, outlined in the National Information Board’s complex Personalised Health and Care 2020 framework,87 is that it will be an entirely digital organisation by 2020. Although such a system, if properly implemented, should provide benefits for research as well as patient care, it is an ambitious goal given the current financial and operational pressures within NHS organisations. The earlier (and equally ambitious) National Programme for IT, which aimed to create a single electronic care record for patients, connect primary and secondary care IT systems and provide a single IT platform for health professionals, was dismantled, mainly because IT companies could not provide solutions to match the ambition.27,88 Despite such programmes, our experience from this study has highlighted the poor-quality electronic infrastructure in hospitals and the multitude of clinical databases that are not amenable to ‘interoperability’ (i.e. not needing extensive customisation or permission from the database vendor to provide data flow). Full interoperability for HISs across the majority of hospitals, therefore, still appears to be a distant goal.89
We met the feasibility criterion of having > 10% of our consented population being referred for CMR in hospitals with a dedicated CMR facility. However, many of these referrals were for research (patient participating in a research study) rather than clinical purposes. For example, in the lead hospital, 33% of all referrals were for research. It is not clear whether or not this reflects practice in other hospitals with dedicated CMR facilities. Referral for non-clinical reasons (unless the findings are used to change clinical management) would dilute any observed effect of CMR on outcomes in an extended registry. Another challenge was that imaging reports were largely in free-text format, so individual CMR parameters would have to be extracted manually, which would not be feasible for an extended registry. A registry assessing the prognostic value of CMR, including individual CMR parameters, such as contractile function, myocardial oedema, MVO, intracardiac thrombus, myocardial haemorrhage and myocardial scar, would allow for an assessment of the value added of each parameter on patient outcome. Similarly, ECHO reports were also in free-text format, and bedside ECHO reports were missing entirely from our data sets because these are not routinely uploaded to electronic databases.
We made the decision to obtain data about follow-up from hospital episode data rather than from the source data submitted by participating hospitals, which are used to compile the hospital episode data. Using hospital episode data has two advantages: (1) follow-up episodes of care can be linked to index admissions, wherever they occur (not just in the participating hospitals) and (2) the organisations compiling hospital episode data apply established data management processes to improve data quality. We met the feasibility objective with respect to linkage of index admission data with hospital episode data. Linkage was achieved for 99% of patients. We failed to achieve linkage for the remainder mainly because of missing identifiers (NHS numbers) in the data sets submitted by participating hospitals. We had originally envisaged collecting information from local HISs on outcomes in the 12 months following the index procedure. However, we would have been unable to capture these data on every patient included in the study because many patients were discharged to, and followed up in, different hospitals, and it would have been difficult to liaise with and obtain data from multiple sites.
Is it feasible to set up a national registry from hospital episode data?
We explored the feasibility of setting up the registry retrospectively entirely by using linked, routinely collected data sets [e.g. hospital episode data, NHS England’s Diagnostic Imaging Dataset (DID) and national audits, such as the BCIS’ NAPCI90]. Setting up the registry in this way would avoid the need to obtain patient consent. This would be deemed appropriate on the basis that all of the information made available would be anonymised.
We were able to identify patients who activate the PPCI pathway in hospital episode data using the relevant procedure codes. We identified 98% of patients in our cohort who underwent PPCI and 85% of patients who underwent emergency angiography without PPCI. We could not, however, identify the CMR exposure in hospital episode data for the majority of study participants who had a CMR scan. This is because CMR is carried out mainly as an outpatient investigation and the hospital episode outpatient data sets do not contain details of the exact nature of the outpatient visit (only the specialty), so the reason for outpatient visits coded under the cardiology specialty was unclear. One option to consider would be the use of NHS England’s DID to identify CMR exposure. DID is a central collection of detailed information about diagnostic imaging tests carried out on NHS patients, extracted from local radiology information systems and submitted monthly to NHS England.91 The main driver for the creation of the DID was to assess how the use of diagnostic imaging could contribute to the early diagnosis of cancer (in particular, GP direct access to these tests). As such, DID has already been linked to cancer registry data, but not other national audits, and there could be substantial difficulties/delays in applying for linkage permissions with, for example, the BCIS’ NAPCI data set.90 Although DID captures information about referral source and patient type, details of the test (e.g. type of test and body site), demographic information (e.g. GP-registered practice, patient postcode, ethnicity, gender and date of birth), plus items about waiting times for each diagnostic imaging event (it does not store the images themselves, the individual parameters measured by the test or the outcomes/diagnoses related to the test). DID could not, therefore, be used to assess the prognostic value of individual components of a CMR scan, or the extent to which a CMR scan altered patient management. Nevertheless, it should allow ascertainment of CMR exposure.
Economic evaluation
We developed a series of economic decision models to identify the key drivers of cost-effectiveness. To our knowledge, this is the first time the key drivers of the cost-effectiveness of introducing CMR have been considered. We chose two patient subgroups for our models: patients with multivessel disease and patients with unobstructed coronary arteries. Both of these subgroups were identified as potentially benefiting from CMR in the formal consensus study.
Our results showed that, for each of the base-case models, the differences in QALYs between strategies were very small and, therefore, the results were largely driven by the differences in costs, although mean differences in cost for different strategies were modest. Sensitivity analyses around the two models identified the diagnostic accuracy of the ischaemia tests as the key driver of cost-effectiveness. In the model for patients with multivessel disease, the costs of ischaemia testing and the QALYs associated with MACEs and no MACEs exerted an influence on cost-effectiveness results, but not to the same extent as the diagnostic accuracy of CMR and pressure wire. Similarly, in the model for patients with unobstructed coronary arteries, the cost of CMR, the difference between QALYs associated with MACEs and no MACEs, and the proportion of patients who have MACEs who are assumed to die exerted an influence on the results; however, the better diagnostic accuracy of CMR and standard ECHO combined (the reference standard in the base case) compared with standard ECHO only was more important.
The key drivers of cost-effectiveness highlighted by each model are similar, and, therefore, it is likely that if CMR was introduced for our entire eligible population (patients who activate the PPCI pathway), it would be these parameters that would need to be measured accurately in any future study. This work highlights the importance of having accurate information about the diagnostic accuracy of different tests, as estimates for these parameters are such key determinants of cost-effectiveness. We have identified some of the key unknowns in terms of costs and outcomes in the population of patients who activate the PPCI pathway, which can inform future studies to ensure that important variables are captured in sufficient detail.
Identifying important changes in management resulting from cardiovascular magnetic resonance and the subgroups of patients to which these refer
Parts of this text have been reproduced from Pufulete et al.4 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.
A key objective of our feasibility study was to define a primary composite outcome, acceptable to cardiologists and other stakeholders (e.g. clinical commissioners) as representing a clinically important change in management (e.g. expected to prevent future MACEs) as a result of an eligible patient having had CMR, that could be used for the registry in the medium term. Although such a primary outcome is not typical, it would allow evidence to be obtained more quickly to inform decisions about implementing CMR than would be possible for an evaluation based on MACEs. In the long term, we envisaged that the primary outcome for the registry would be MACEs, but addressing relevant research questions using the registry would need data for a large number of patients. For example, in order to have 90% power to detect a 1.2% difference in MACEs (4.8% vs. 6.0%) between those having CMR and those not, at the 5% level of statistical significance, and assuming 10% of patients who activate the PPCI pathway receive CMR and 90% do not, the registry would require > 37,000 patients. This represents 20 hospitals, with an average of 500 patients giving consent per year per site for > 3.5 years. We expected that important changes in management would be much more frequent than MACEs, allowing the effectiveness of CMR with respect to this outcome to be evaluated more quickly with data from fewer patients.
We used a formal consensus approach because there were no studies in the literature that reported the impact of CMR in our study population, reflecting the fact that changes in clinical management are difficult to use as end points in trials. This is because some important changes in clinical management are not easily observed92 and the causal relationship between changes in management and improved outcomes is not well defined. Imaging has been identified as one of the key drivers of increased health-care costs; therefore, the adoption of an imaging modality in a new clinical context should be supported by evidence of its benefit to specific patient groups or clinical scenarios. Such evidence is difficult to obtain in practice.
We identified five subgroups of ACS patients who activate the PPCI pathway for whom there was consensus that CMR changes patient management in a clinically important way: (1) patients who have an OHCA (≈7% of those who activate the PPCI pathway),69 (2) patients who have a ‘normal’ (unobstructed) coronary angiogram (≈10% of those who activate the PPCI pathway),15,16 (3) patients who develop LV thrombus (3% overall and 9% in patients with anterior STEMI),70 (4) patients who have multivessel disease (between 40% and 65% of those who undergo PPCI)7,10,71 and (5) patients in whom CMR markers indicate a poor prognosis (up to 60% of patients after STEMI).2,72 There was consensus about the first three subgroups in both rounds of the survey, whereas consensus was reached about the other two subgroups in the modified survey. These results suggest that CMR benefits a large proportion of patients who activate the PPCI pathway.
Cardiologists who participated in our research agreed that CMR is superior to ECHO in establishing a diagnosis in patients who survive an OHCA, which has implications for treatment and prognosis. Despite this view, there are few studies that have reported the role of CMR in managing these patients. A retrospective case series of 54 OHCA survivors showed that CMR diagnosed the cause (ischaemic or non-ischaemic cardiomyopathy) in 40 (74%) of the survivors.93
The cardiologists also agreed that CMR can provide a definitive diagnosis (e.g. acute MI, acute myocarditis and cardiomyopathy, especially Takotsubo cardiomyopathy) in patients with ACS and unobstructed coronary arteries. Evidence for the role of CMR in patients with ACS and unobstructed coronary arteries also comes from small retrospective case series. These suggest that CMR provides a definitive diagnosis in 80–90% of these patients.20,23,94 Without access to CMR, the management of these patients is variable in clinical practice, which may have long-term implications. A 2015 systematic review50 showed that the overall all-cause mortality in patients presenting with suspected MI and unobstructed coronary arteries was ≈5% at 12 months.
The cardiologists agreed that CMR identifies LV thrombus better than ECHO, which allows more patients to be identified and treated appropriately. Accurate identification of a LV thrombus is important because it often directs subsequent anticoagulation therapy to prevent embolic events. A 2015 systematic review,95 published after we conducted the formal consensus study, showed that late gadolinium enhancement (LGE) CMR is the most accurate modality for detecting LV thrombus, with 88% sensitivity and 99% specificity (compared with standard ECHO, which had 24–33% sensitivity and 94–95% specificity, and contrast ECHO, which had 23–61% sensitivity and 96–99% specificity), although most of the included studies in this review did not use a pathological or surgical gold standard for the detection of LV thrombus.
There was agreement among cardiologists that CMR-based testing of ischaemia after PPCI would be likely to optimise the revascularisation strategy for patients with multivessel coronary disease, although they acknowledged that there is no evidence to support the view that a CMR-based revascularisation strategy would improve outcomes. Patients with multivessel disease have a two-fold increase in MACEs compared with patients with single-vessel disease.96 There is continuing debate about the benefits of complete versus single lesion revascularisation during the index PPCI admission.97 Despite recent evidence from randomised controlled trials (RCTs) showing improved clinical outcomes when complete revascularisation is undertaken at PPCI,98 current American99 and European100 revascularisation guidelines for acute MI recommend revascularisation only of the infarct-related artery at PPCI in patients with multivessel disease.
There was considerable debate about using CMR to identify patients with a poor prognosis after PPCI. Although there was agreement that CMR parameters of cardiac function (e.g. impaired LV function, large infarct size, MVO) are useful for risk stratification after STEMI, there was disagreement about whether or not this would lead to a management change. Some cardiologists felt that there would be no changes to prescribing for secondary prevention because all patients should receive aggressive secondary prevention in accordance with guidelines. It was acknowledged, however, that CMR markers are prognostic for outcome;4 two recent meta-analyses of prognostic studies (conducted after this formal consensus study) showed an increased risk of MACE, by 13–15% for every 10% decrease in EF assessed by CMR,2 and in patients with MVO assessed by CMR [odds ratio (OR) 2.60, 95% CI 1.68 to 4.02 and OR 4.30, 95% CI 2.19 to 8.43, depending on the method of MVO assessment].3
We could not identify all patient subgroups from HIS data sets. Patients with LV thrombus and OHCA were identifiable only in hospital episode data, from ICD-10 codes for these diagnoses assigned to the index admission. Patients with unobstructed arteries could not be positively identified from HIS; we identified them by their exclusion from the hospital BCIS’ NAPCI data sets (as only patients who have a stent inserted, i.e. have PPCI, or have had an attempt to have a stent inserted, are included in BCIS’ NAPCI). Given the uncertainty we faced when trying to identify this population prospectively (see Feasibility of setting up a registry using hospital information system data) and the initial confusion over eligibility at the beginning of our recruitment period, we decided to review the medical notes (representing the reference standard) for a subset of these patients to determine the proportion classified correctly according to our definition of ‘unobstructed coronary arteries’ (exclusion from BCIS’ NAPCI). Although we classified 81% of patients correctly, some patients we identified as having unobstructed arteries were identified as having a culprit lesion in the manual review of patients’ notes. Furthermore, there was also a degree of disagreement between medical notes and the primary diagnosis recorded in hospital episode data for the index admission of these patients, with 11% of patients who were identified as having unobstructed coronary arteries in their medical notes having a primary diagnosis of MI in HES. Our investigation using medical notes highlights a degree of misclassification of this subgroup of patients, although this is not surprising given the uncertainty in diagnosis during the index admission.
Implementation of a primary outcome based on the changes in management for which consensus was achieved using hospital episode data
Parts of this text have been reproduced with permission from Pufulete et al.68 This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
There was consensus that CMR changes patient management in a clinically important way for five subgroups of ACS patients. For each group, we listed ‘activities’ (e.g. new diagnoses or procedures, additional diagnostic tests, changes in medication, outpatient appointments) that reflected a change in management. Activities were identified from the relevant consensus statements and by looking at care pathways, when available, to confirm that we captured all aspects of patient management in these activities. We then compiled a list of the relevant ICD-10 diagnosis codes and OPCS procedure codes (with help from clinicians on the study team and a dedicated HES coder). We identified these codes in hospital episode data for participants up to 12 months following the index procedure.
We were able to identify all of the activities that we identified as representing changes in management in the hospital episode data. Therefore, we conclude that it would be feasible to look for differences between patients who have and those who do not have CMR within ACS subgroups in terms of frequency and time to first event for some of the activities. We recognise that some events that were listed as representing a change in management (e.g. cardiology outpatient appointment, changes in medication) are not immediately amenable to inclusion in a composite outcome for a time-to-event model.
This approach to constructing a proxy outcome reflects our original concept and appears to be feasible when the important changes in management are hypothesised to cause a change in the same direction in all events/activities included in a composite outcome for a subgroup. However, through the consensus process, it became apparent that some of the hypothesised changes in management arising from CMR will not necessarily result in a unidirectional change in the frequency or rate of an activity. In several instances, an important change in management was more appropriate targeting of a treatment activity to the patients who were likely to benefit most, without necessarily any change in the overall frequency of the activity. For example, in patients with multivessel disease, CMR may result in better (i.e. more cost-effective) targeting of additional revascularisation. In these situations, a simple evaluation of the frequency or rate of a proxy outcome would not detect any value of CMR to patients and the NHS.
We were unable to identify some important changes in management that could have arisen from CMR. There are no medication records in hospital episode data, so changes in drug therapy could not be identified. Initiation, withdrawal or changes in medication represented an important change in management for all of our patient subgroups and, in particular, patients with LV thrombus, for whom anticoagulation therapy is the main change in management. A potential solution to this problem is the linkage of HIS and hospital episode data with data from the Clinical Practice Research Datalink (CPRD), a primary care data set that has detailed prescription information. However, because not all general practices contribute data to CPRD, medications data would be available for only a proportion of patients in a registry.
We were also unable to identify additional tests or procedures carried out in the outpatient department because most outpatient episodes in the HES outpatient data set do not have diagnostic/procedure codes assigned to them. We are uncertain about the extent to which DID, recently added to the HES portfolio of data sets, may improve this situation. Finally, hospital episode data alone may not capture all activity related to changes in management as some patients will be seen multiple times by their GP or in private facilities. However, as the extent of after care in private facilities in the UK is minimal, it is unlikely to invalidate a future comparison in a registry.
Patient and public involvement and engagement
The chairperson of the Study Steering Committee (SSC) was a patient representative. We also included on the SSC a representative from the British Heart Foundation (the Area Development Manager). We felt that it was particularly important to have members of the public and a previous patient on the SSC because one of the main objectives of the feasibility study was to investigate the feasibility of consenting patients into a registry.
We sought the views of both of our lay members about various aspects of the study at all stages. For example, at the first meeting, we explored the issue of consent. Both our lay members suggested exploring the possibility of not having consent for the study and confirmed that they did not think using ‘assumed consent’ would be a problem if it was implemented in routine practice. We did this (as described in Feasibility of setting up a registry using hospital information system data) but concluded that this viewpoint, although prevalent among patients and the public, is not currently consistent with the data protection law.101 When implementing our conventional consent method, we also asked patients who had been in a similar position to review the patient information leaflet and consent form to ensure that the information was not excessive but covered all the points that patients would want to know. This input was very useful and helped to guide some changes during the finalisation of these documents for the Research Ethics Committee submission.
We had planned to hold a focus group with patients and the public to engage with them about developing the primary outcome as part of the formal consensus process. The lay representatives on the SSC were consulted at length about this, including reviewing questionnaires that had been drafted for the consensus meetings. They concluded that, to form an opinion about the importance of the changes in management for which consensus was obtained, lay people would require an understanding in terms of technical knowledge and recent experience and, therefore, it was not reasonable to expect lay people to comment on or refine the outputs from the consensus process. Throughout the project, the lay representatives on the SSC (and, in particular, the SSC chairperson) expressed dismay about the lack of availability of HIS and hospital episode data and the delays this caused to the study. Accepting that our application for these data was held up by the one-off event of the moratorium in processing applications at NHS Digital (see Chapter 2, Major delays with the study), lay representatives were surprised that the plan for implementing new governance processes did not appear to have considered how applications in progress should be handled. The ruling from NHS Digital and ONS that the wording of the patient information leaflet was not compliant with the new guidance, after we had completed recruitment, seemed Kafkaesque102 from the perspective of the researcher team.
Strengths and limitations of the study
Feasibility prospective cohort study
We established a successful collaboration among four hospitals and exceeded the recruitment target for the study of 1600 patients. We also identified the main challenges to establishing a prospective registry in the way we had envisaged: identifying the eligible population is difficult to do from local HISs; the conventional consent model fails to capture a sizeable proportion of the eligible population and would be costly to implement; and, most importantly, the required local data from HISs are not uniformly available/exportable.27
We did not identify or test a more efficient method to obtain consent. The methods used by hospitals for the study are almost certainly not feasible to implement in the long term for a registry, and are likely to result in varying completeness across hospitals. However, the barriers to any form of ‘presumed consent’ or ‘opt out’ seemed insurmountable without a long-term plan for implementing consent.
We are aware that CMR may benefit other groups of ACS patients (e.g. those with NSTEMI). However, we considered at the outset that it would be impossible to identify consecutive patients presenting with a broader diagnosis of ACS but not requiring emergency coronary angiography; being able to define and ascertain the entire eligible population is a key requirement of a high-quality database. This was a particularly important consideration given that the proposed registry was based on the premise of linking information collected in the course of usual care, avoiding the time and cost burden of primary data collection.27 Even so, we still encountered difficulties in identifying from electronic records the group of patients with unobstructed coronary arteries in our eligible population.
We concluded that we could identify our eligible population in hospital episode data, based on the fact that we identified the index admission for most patients recruited into our study. However, because we had only hospital episode data for patients in the cohort, we cannot comment on or quantify the risk that an algorithm designed to identify such patients from hospital episode data could also identify some patients who would have been ineligible for the registry. Our impression from exploring codes assigned to hospital episode data for index admissions is that this risk is low and is likely to reduce as the quality of hospital episode data improves. The only way to quantify the risk would have been to request anonymised linkage of all HISs and hospital episode data for all cardiology admissions during the study period, with identification of the study cohort participants as a subgroup.
A major limitation is the length of time it took to complete the study. The main reason for this was because our application for HES data coincided with the moratorium on data requests by NHS Digital (formerly known as the Health and Social Care Information Centre), which resulted in a delay of 23 months in processing our application. This meant that the study took > 4.5 years to complete, whereas we had projected 1.5 years for completion. During this time, it is likely that data availability and the quality of the data will have changed/improved, given the rapidly evolving NHS IT systems/platforms. During this time, new diagnostic tests for detecting ischaemia (e.g. pressure wire), directly competing with CMR, were introduced and rapidly adopted by some cardiologists. These changes highlight the need for new research questions; therefore, it is likely that the aims and objectives for a national registry now would be different from those we proposed when we set up this study. For example, it may be more relevant to compare ischaemia testing using pressure wire with CMR, rather than to compare CMR with no CMR, which we proposed for our registry. We have tried to adapt the study to the current circumstances, for example with respect to the cost-effectiveness models, but have been unable to do anything about the changes in practice or the historic nature of the data collected.
Economic evaluation
There is considerable uncertainty around the majority of parameter estimates in both models. In the model for patients with unobstructed coronary arteries, the probability of patients truly having a MI was obtained from the results of a systematic review,50 but all other probability estimates in both models are based on single papers, often on small or modest sample sizes. There was only sparse evidence found in the literature for many of the probability estimates required. This was particularly true for the probabilities of MACEs for patients who truly did and those who did not have a MI (and had appropriate treatment or not), in the model for patients with unobstructed coronary arteries. Members of the study team were concerned that the estimated ratio of MACEs in MI patients compared with non-MI patients (in patients with unobstructed coronary arteries) found in Pathik et al.52 was very high; this is why we investigated this parameter in the sensitivity analyses.
Even when papers reported on the correct patient group and outcomes, studies may have been conducted in other countries, where practices may differ from those in the UK. For example, Smits et al.38 specifically reported numbers that could be used to estimate the probability of a MACE by 1 year for those truly with ischaemia separately for those who were and those who were not revascularised (see Figures 1 and 2, and Chapter 5, Cost-effectiveness models); however, this study was conducted in Europe and Asia, with no UK centres included. Occasionally, estimates from a different patient group were the only estimates available. For example, in the model for patients with multivessel disease, the diagnostic accuracy of stress ECHO (its sensitivity and specificity) was taken from a paper considering the diagnosis of significant coronary artery disease, rather than a PPCI population, and estimates are based on the number of vessels rather than the number of patients.39
In the base-case analyses for multivessel disease, CMR and pressure wire were treated as reference standards, and both were assumed to have 100% sensitivity and specificity. There are suggestions in the literature that this is reasonable; for example, Emrich et al.103 found that CMR correctly identified all 20 out of 125 patients with MI in their study. The assumption of equal diagnostic accuracy is critical, because the sensitivity analyses showed how diagnostic accuracy of the ischaemia tests was the key driver of cost-effectiveness. Similarly for the unobstructed coronary arteries model, reducing the diagnostic accuracy of CMR plus standard ECHO compared with standard ECHO alone (sensitivity analysis 1) doubled the ICER.
Individual patient data on resource use were not available from hospital episode data in time to be used in these analyses; resource use associated with each of the patient pathways was, therefore, estimated based on discussions with the PIPA study team and costs were largely obtained from NHS Reference Costs.40 These methods have allowed standard care pathways to be costed but will have underestimated the variability between individual patients and their actual patient pathways.
As well as the costs of initial ischaemia testing (if applicable), and any revascularisation, we included the costs of follow-up outpatient appointments, medications and rehabilitation over the first year. We also included additional health-care costs to 1 year based on the Office of Health Economics’ estimate of the annual cost per person of NHS care.43 It is recognised that there will be an element of double counting here, as the costs of ischaemia testing, revascularisation and subsequent costs will overlap with these annual costs, but this will be the same for all strategies and should not affect the comparison of costs between strategies. These annual costs were included to account for differences in costs for patients who died; these costs were halved for patients with a MACE who died by 1 year.
Utility estimates for a PPCI population were found in the literature, but there was uncertainty around the utility decrement for patients with MACEs compared with those without MACEs. A decrement of 0.05 was assumed for patients with a MACE compared with those without a MACE.48 Although this decrement has been used by others, the original reference for this decrement implies that it was a sensitivity analysis, and was not based on any primary data.104 Utility estimates for patients with unobstructed coronary arteries were not found in the Tufts Cost-effectiveness Analysis Registry,46 and a recent paper measuring quality of life in patients with MI with unobstructed coronary arteries reported that no studies measuring quality of life in these patients have previously been conducted.105 This study measured quality of life using the SF-36 at 3 months post acute event and found similar findings in patients with unobstructed coronary arteries and a control group of MI patients with coronary heart disease, suggesting that the estimates used in our model for patients with unobstructed coronary arteries are reasonable.
Formal consensus study
Parts of this text have been reproduced from Pufulete et al.4 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.
A key strength of the study was the use of a formal consensus approach, based on a systematic search of the literature as well as expert opinion. The nominal group technique is one of the four well-established methodologies for formal consensus [the other three are (1) the Delphi method, (2) the Research ANd Development Corporation (RAND)/University of California, Los Angeles Appropriateness Method (RAM) and (3) the National Institutes’ of Health consensus development conference methodology].4,106,107 Our literature review included all study designs to determine the size and nature of the evidence base and highlight research gaps. The search was systematic and extensive, including five databases and published abstracts. Four cardiologists contributed expert opinion. Two are UK/European CMR experts. The process provided evidence for the content validity of each statement. All cardiologists who drafted the statements proposed the same patient subgroups as potentially benefiting from CMR. There was generally good agreement between draft statements generated by cardiologists and the non-clinical reviewer.
There were no disagreements in the group meeting regarding the potential management change in each of the patient subgroups identified. Similarly, the studies identified in our literature review (both prognostic and diagnostic) were consistent in their findings. The majority of prognostic studies reported that CMR markers such as EF, MVO and haemorrhage were associated with MACEs, although most did not evaluate the prognostic value of CMR indices in addition to indices available from alternative imaging modalities. The evidence supporting many of the statements came from small, retrospective case series. There were no prospective studies in the literature that reported the impact of CMR on clinical management in our population or whether or not any changes in management affected patient outcomes.
The main features of a formal consensus method are anonymity (i.e. statements were rated free from peer-group pressure) and iterative feedback (i.e. participants could adjust their initial rating based on the feedback of the group rating). We described the supporting evidence for each statement, identified via the literature review, and fed back qualitative (i.e. panel members’ comments) and quantitative (i.e. group median ratings of each statement) information at the face-to-face meeting. We also included a varied group of cardiologists from different specialties in order to encompass diverse perspectives.4
A potential limitation of this study is that we did not perform any formal quality assessment of primary studies. We made this decision for two reasons. First, the purpose of the review was to identify the evidence base in order to inform the consensus process, not to use the literature to quantify the effectiveness of CMR or its diagnostic or prognostic value. Second, no single assessment tool would have been appropriate for the diverse studies that were identified, hence assessments would not have been comparable. Therefore, we have not referred to our overall review as being ‘systematic’ (only the search), although its conduct closely reflected established guidelines for systematic reviews and we believe that other researchers following the same steps would replicate its findings.
The number of panel members in our study was lower than the recommended 8–12 members for a consensus panel.107 Nevertheless, smaller groups are preferable to larger ones because, although having more group members increases the reliability of group judgement, large groups reduce the ability to elicit potentially important contributions from every member of the panel.107 To overcome any criticism of the number of panel members in the study and to prevent the possibility of introducing bias (given the self-selected nature of the panel), we extended the survey to UK cardiologists who did not participate in the formal consensus process. Apart from one statement (statement 1, relating to the ability of CMR to identify patients with a poor prognosis after PPCI), the same statements were in consensus when the survey was completed by cardiologists external to the consensus process. This indicates that the process was robust and the survey statements were clear and unambiguous.
A further limitation of the study was that we did not include other stakeholders (e.g. GPs and patient representatives) in the consensus process.108 Although we had originally planned to do so (see Patient and public involvement and engagement), we decided that the specialised nature of the statements would not be easily understood by those without cardiology expertise.4 We did, however, hold a separate meeting (outside the consensus process) with a GP commissioner and the director of the Southwest Cardiac and Stroke Network. The conclusion of the meeting was that, given that the cohort is an emergency population and the consensus process identified that a large proportion of the population would potentially benefit from scanning, it would be better to offer CMR to the whole population rather than complicating the patient management strategy by scanning only subsets of patients.
Lessons for the future
Future studies attempting to answer research questions through linkage of routinely collected data sets (at the local and national level) should consider the following:
- Timeliness – a study must be reported within a reasonable time frame given the rapidly changing environment (e.g. NHS IT systems, introduction of new diagnostic tests).
- Data availability in hospitals – the HIS available at each hospital and the level of IT support available to provide the data need to be investigated thoroughly; funding is likely to be required to support data extraction.
- Data sources in hospitals – the relevant data sources at each hospital need to be identified before starting a study. For example, we decided to use BCIS/MINAP submission data sets instead of catheter laboratory databases after the study started; this caused considerable confusion and difficulties over patient eligibility, given that BCIS/MINAP could provide data only for patients who had STEMI/received PPCI and not patients with unobstructed coronary arteries.
- Hospital episode data – these data are made available (after ‘cleaning’) by NHS Digital (or Information Services Division Scotland, PEDW or Northern Ireland Department of Health) for whole financial years. Our failure to appreciate this meant that only ≈67% of patients had a whole 1 year of follow-up.
Implications for practice
Our experience highlights the need to assess ‘feasibility’ before attempting to set up a new registry. Although some of the pitfalls we experienced may, in hindsight, be viewed as resulting from a lack of ‘preparedness’, our experience is not unique. Registries (e.g. national audits) that have managed to overcome issues with data collection and quality have achieved this through a lengthy process, necessitating many iterations (and a feedback loop) before attaining any quality targets. This process requires considerable investment, infrastructure and commitment at the national level, which would be difficult to achieve in a research setting.4 The process also has drawbacks; for example, iterations add to the complexity of creating a data set that is consistent over time and that takes into account changes in the methods of collecting data, which may result in concomitant inadvertent changes in definitions of the data.
We consider the following as desirable developments to make it more feasible to construct a registry from routinely collected data:
- More uniform implementation of IT across acute NHS trusts (e.g. a shortlist of ‘approved’ database suppliers with compatible features, including features to facilitate data extraction for secondary purposes).
- Lighter touch governance when anonymisation is done centrally. Access to patient data is likely to remain a challenge, particularly with the introduction of the General Data Protection Regulation (GDPR).109,110 The lack of clarity over the safe and ethical secondary use of routinely collected data is currently a major barrier preventing access to routine data for research.
- A method for obtaining consent that is practicable to implement in the busy setting of the NHS. For example, this could involve an opt-out model of consent or presumed consent, combined with a national publicity campaign promoting the benefits of using routine data in research. Opt-out models of consent are currently very difficult to implement because of the specific, and changing, nature of the instructions for informing affected people. The apparent failure to make provision for analyses of historic and ongoing data sets, when data collection started before changes in law, is particularly wasteful and frustrating for researchers.
- Efficient provision of routine data by data guardians, for example NHS Digital.
Future research recommendations
The present study highlighted the difficulties of implementing a conventional consent process for a registry that recruits patients who undergo an emergency procedure and have a short hospital stay. An important area of future research should, therefore, be to identify and test a more efficient method of obtaining consent for similar studies in similar settings that need to link data sources. This issue should be approached at a national rather than a local level, and involve detailed discussions with all relevant stakeholders. Critically, clarification is required about whether or not it is possible, given the recent debates about data sharing and the changes to information governance policies at NHS Digital,111 to obtain approval (under section 251 of the NHS Act 200676) to use patient information for medical research and link national data sets without consent. Currently, there is discrepancy between UK national registries, with many implementing prospective consent,74,75 whereas others, such as the national cardiac audits managed by NICOR, operate on the basis of section 25176 exemptions.
Alternatively, we need to test the feasibility of conducting the study using linked, anonymised national data sets (e.g. hospital episode data, BCIS NAPCI, DID, CPRD for medications). This would avoid the need for obtaining individual patient consent or section 25176 exemption, as data are anonymised when made available to researchers. Such a linked data set would be able to capture all the eligible population and could use MACEs as an outcome, although there may still be regulatory hurdles in obtaining permission to link these data sets and CPRD captures data for only a small proportion of the UK population.
There has been a rapid adoption of pressure-wire testing for detecting ischaemia during this study despite a lack of evidence of benefit over CMR. Our cost-effectiveness models have highlighted that the diagnostic accuracies of the CMR versus pressure-wire ischaemia tests are key drivers of the relative cost-effectiveness of management strategies based on these tests, compared with stress ECHO or each other. Therefore, future research is needed to quantify the relative diagnostic accuracy of CMR and pressure wire versus stress ECHO, and versus each other, for ischaemia testing. The fact that both CMR and pressure-wire testing are regarded as reference standards in clinical practice and no superior standard is recognised may mean that the cost-effectiveness of CMR versus pressure wire can be tested with respect to effectiveness only in a large RCT.