This section reflects on the successes, and failures, of the project – particularly within the context of evaluating a major government programme while it is in flight. It covers the complexities of working with a government project, and also a reflection on the nature of the data access and linkage that was planned in the initial project planning, compared to the nature of the data access and linkage we were actually able to achieve.
Involvement with the delivery team
Evaluating a product which is still in flight in terms of delivery presents challenges to an evaluation team, which we clearly felt in this project. There is a classic evaluator’s dilemma – how close to get to the delivery team?89 If you are closer, then you become more trusted, may have better access and insights, and become able to better influence and impact the programme being evaluated. On the other hand, becoming too close presents a risk of becoming too closely engaged with the team, and a risk of alignment and bias.
This evaluation took place at a complex time in terms of wider events, health system reconfiguration and digital health reorganisation. First and foremost, this project took place during and shortly after the COVID-19 pandemic. As a result, many organisations – both at the national level and in terms of NHS delivery at the local level – were working in an emergency response mode and an atypical way. At the start of this project, the NHS App was commissioned by NHSX and delivered by NHS Digital. Over the course of the programme, the governance and organisation of digital work in the NHS changed, and the two organisations were merged (along with others) into NHS England and its new transformation directorate.
As with many collaborations with delivery organisations, there was often a changeover of staff within teams, which presents complexity for continuity of relationship and understanding of the progress. We had strong high-level support from the evaluation from leadership figures in both NHSX and NHS Digital at the start of the progress. Some of the senior sponsors and supporters left, particularly in the time around COVID-19 where large numbers of people moved roles or were redeployed. This meant we had to build new relationships with the senior leadership team on more than one occasion, including rearticulating the value of the evaluation, and the independent nature of its findings.
We were able to establish working relationships with the delivery team for the app. We were grateful for the largely highly continuous nature of the core delivery team – as this meant we were able, over time, to build up a trusted relationship with the team delivering and deploying the app. As often occurs, the relationships were slower to start with, and then grew in terms of strength and effectiveness and the evaluation team got to know the delivery team. Towards the latter stages of the project, we had high functioning monthly meetings with the delivery team, in which we would share our latest results and thinking, and the delivery team would share back both their plans and their take on our results and interpretation.
Useful specific aspects that emerged from this programme of regular meetings included: (1) collaborative identification of potential case study sites; (2) detailed discussion and reflection on the nature of the findings; and (3) detailed functional discussion on the potential for data linkage, and what was and was not possible given the existing privacy notices and terms associated with the app.
These regular meetings also led to opportunities for our findings to be considered at a national level, for example, (1) our data on inequalities of access were shared with the NHS App delivery team, who went on to share the data with other interested parties, including the NHS inequalities board and (2) we were invited to share our findings with the parliamentary and social care select committee (described in the next chapter).
NHS App data linkage: a reflection
This project was by nature a multimethod study – using both qualitative and quantitative approaches. As part of the quantitative component, we had intended to do some data linkage studies. We always anticipated data linkage would be difficult – it is with all projects that use NHS data, often for very appropriate reasons of governance and the maintenance of trust. However, in this project, what we planned to achieve proved not to be possible – at least during the study period – and there are a number of lessons learnt, and suggestions to take forward that might allow data linkage in the future.
Practice-level data linkage
In the approved protocol that was agreed by the NIHR funding committee, we had hoped to link data about patient outcomes and patient flow into hospital at the practice level, so we could see whether higher usage of the app was associated with better (or worse) outcomes and more (or less) healthcare service utilisation.
We had intended to conduct this analysis using a bespoke Clinical Practice Research Datalink (CPRD) linkage at the practice level. We were unfortunately not able to conduct this analysis in the funded time frame, as due to the COVID-19 pandemic, the CPRD bespoke linkage service was turned off. As a result, we had to limit our practice-level analysis to those where we were able to obtain data, including from the GPPS. We should note that this change was discussed with and agreed by NIHR.
While these results were calculable and are interesting – they are limited in their interpretation. The results remain fundamentally at the ecological level – providing a useful suggestion of the nature of relationships but limiting causal explanation due to risk of ecological fallacy. As a result of the limitations of our quantitative approach, we reprioritised some of the time and effort towards the qualitative work – and in particular to look in more detail at inequalities with the inclusion of an additional, unplanned case study site designed to focus on inequalities in uptake – as this was a core finding of the initial quantitative work.
Person-level data linkage
What we would have ideally liked to have achieved with this project – if we could find a legitimate mechanism – was to link data at the person level, allowing us to investigate NHS App usage with various health outcomes and healthcare utilisation patterns at the individual level. This had been included in the first submission to the NIHR funding committee for this project – but the committee cautioned against this as they thought it would be too complex to deliver. Even though we had – through close working with the NHS App programme team and their management at NHSX and NHS Digital (as they were) – succeeded in securing support for a direction under section 254 of the Health and Social Care Act to allow this to happen. This would have provided a legal basis for the linkage to take place through an established route (section 254 directions are used around a dozen times a year by NHS England to achieve data flows for important purposes for the system).
In reality, the wisdom of the NIHR funding committee was entirely correct – and due to the events of COVID it would not have been possible to obtain this direction, due to the other more urgent prioritised data activity at the time. However, the ability to link NHS App use to other data remains both a substantial challenge, and a real opportunity for future research.
If, as it is planned, more services are added to the NHS App – including possible use of digital therapeutics, preventative services and more detailed patient information and choice – then being able to track the impact of these interventions – at the individual level, will become increasingly important. At present, NHS England have within their systems both person-level app-usage data and person-level primary care and often secondary care usage – via their secure data environment (SDE). The potential for the systematic and data-driven evaluation of new services as they come online – in a cost-effective way – is real, but only if appropriate data linkage can be achieved.
Potential options for data linkage in the future
During this work, we also explored other potential routes for data linkage for the NHS App, as we were not able to use the route suggested in our original protocol. Despite our best efforts, we were not able to find one that was operational during the study period, but they may become feasible soon. These include:
- OpenSAFELY. We discussed whether it would be possible to use the OpenSAFELY platform to measure outcomes for those who used the NHS App, compared with those who did not. This could potentially be feasible but would require a usable indicator of who is using the NHS App in the patient record. Most GP IT systems report the Interface Mechanism 1 Application Programming Interface usage – a marker of NHS App usage, but this is unfortunately not captured by databases that are accessible in the same format as the other captured information.
- Using the NHS SDE. The NHS SDE is a developing resource allowing data from a number of NHS data sets to be linked – predominantly using the NHS number as the common linkage point. As NHS England also holds the NHS login data for each NHS user, they are able to identify the NHS number for each app user – therefore potentially allowing for a detailed link between NHS App usage (including frequency of use, and use of specific functions) to a number of other outcome and healthcare utilisation data sets.
While both of these linkages are technically possible, at the moment the NHS App terms and conditions do not provide a specific consent for individuals data to be linked, nor is there a wider legal basis for the linkage. Linkage would therefore require either: (1) person-based consent, (2) a Section 254 direction or (3) a control of patient information notice extension allowing linkage of the data.
Data linkage for the NHS App is complex – and made more so by the events that led to its widespread adoption, described elsewhere in this monograph. While it is now used by more than 75% of adults in England, its rapid increase in use was driven in large part by the use of the COVID Pass feature. This was required for people to perform many normal life activities, including attending large events, booking holidays abroad or ordering COVID tests. As such, while never mandated, there was often a high level of requirement for people to download it if they wanted to live a relatively normal life. Consequently, sharing data on users of the app has a further level of caution that must be considered beyond a routine digital health intervention.
We suggest that being able to link these data is of substantial interest to both the NHS and to researchers studying and seeking to learn lessons from large digital patient portals around the world. This would enable fundamental questions to be answered with a more causal evidential basis, including whether people who used the app had different patterns of healthcare utilisation, healthcare cost and, ultimately, patient outcomes. These options should be explored but should be done with a keen desire to retain public trust and confidence in how the data are being used – and would likely benefit from specific public engagement and consultation exercises.