Methodology

This chapter will present the design and methods for the realist review and evaluation aspects of the project. Excerpts of the chapter have been reproduced from Aunger et al.137,138 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Taking a realist perspective

A variety of theoretical contributions have sought to improve understanding of IOCs in health care.14,48,110,126,127 These authors have generated valuable insights as to ‘what’ leads to successful partnering, but less so to how and why particular features enable collaborations to be successful. In this sense, IOCs can be seen as interventions that frequently fall into the trap of what Dixon-Woods and Martin128 term ‘magical thinking’, that is, the assumption that ‘doing X’ will lead to outcome Y without any articulation of how and why this change will occur, and this means that, often, the assumptions underlying how collaboration is intended to work are left implicit.

Any collaborative effort is likely to have a long and complex process of implementation, from initial discussions between stakeholders to the realisation of the intended benefits and/or failure. However, until now, few have used a realist lens to shed light on this phenomenon. Using a realist methodology to identify when, how and in what circumstances the causal links during implementation break or hold, as well as why collaboration may lead to better performance. A realist methodology also enables synthesis of all literature types in acknowledging the complexity of the interventions that constitute an area such as IOC.

Realist methods are built on the epistemological approach of critical realism, which is based on the concept of generative causation, where mechanisms generate outcomes that are context sensitive.129,130 In realist terms, contexts refer to the situations into which interventions are introduced that affect the operation of the intervention mechanisms.129 An intervention may work through one mechanism in one set of contextual features, but work through a different mechanism, producing a different outcome, in another. As a result, context and mechanism are keenly interlinked and cannot be separated.129 Mechanisms, in realist terms, are the interactions between programme resources and the changes in reasoning by programme actors that occur as a result.131 Mostly, these mechanisms are not directly observable but, nonetheless, can be explanations of why particular outcomes come to be.129

Those who have used a realist perspective to understand IOCs have focused on particular subtypes and contexts of collaboration.127,132,133 However, to the best of our knowledge, none have yet attempted to address the wider topic of IOCs between health-care providers. In this project, we have drawn on realist methodology in both synthesis and evaluation types to test and refine a robust theory of how IOCs in health-care work, to what extent, why and in what circumstances.

A starting point of realist evaluation is identifying the ideas and assumptions underlying how programmes or interventions work, known as programme theories. Realists also work with the premise that programmes are never universally successful, rather how they work (i.e. their mechanisms) to produce outcomes is shaped by contextual features. The goal of realist evaluation is to explain how contextual features shape the mechanisms through which a programme works, and this is achieved through testing and refining programme theories, expressed as context–mechanism–outcome configurations (CMOCs). Initially, CMOCs are tentative ideas, and as the project progresses these ideas are brought into conversation with evidence (i.e. tested in relation to the evidence) and are then refined to produce a more detailed explanation of how context shapes mechanisms.129,130 A refined theory can support the process of adapting the intervention to local circumstances. The aim of this project is to produce a fully refined and actionable theory for practitioners and those implementing such arrangements.

Methods: realist synthesis

Details of search strategies

Theory gleaning

Searching processes in realist reviews tend to be evolutionary in nature, and that was the case here.134 Initially, systematic searches were conducted to gather evidence about how IOC works and the contextual factors that shape across a range of entities, such as alliances, buddying, mergers, acquisitions and hospital groups. Searches were run between 20 February 2020 and 4 March 2020 on databases including the Healthcare Management Information Consortium (HMIC), MEDLINE, Social Policy and Practice and PsycINFO (see Appendix 3 for search strategies). The HMIC commentary search (see Appendix 3) was run on 12 January 2021. The searches were limited to 1990 onwards to provide the most up-to-date literature. In addition, a Google Scholar (Google Inc., Mountain View, CA, USA) search was conducted on 11 March 2020 to identify any grey literature or papers missed. The Google Scholar search used the terms ‘theory organisational collaboration’ to identify theoretical papers and ‘interorganisational collaboration healthcare’. Reference-scanning and citation-tracking was also employed to ensure as many papers were identified as possible (see Appendix 3 for the full systematic search strategy).

After data synthesis, we realised that we lacked elucidation on some of the mechanisms underlying how leadership, among other elements, may be key to understanding the process of collaboration. Therefore, a non-systematic purposive search was used to identify middle-range theories (MRTs), which would allow us to gain further insight into mechanisms uncovered through our analysis of papers identified in our initial searches. MRTs were identified using terms and combinations of terms such as ‘inter-organisational conflict’, ‘inter-organisational communication’, ‘inter-organizational trust’, ‘organisational capacity’, ‘collaborative leadership’, ‘organizational flexibility and effectiveness’, ‘collaborative accountability and governance’ and ‘collaborative regulatory environment’. The searches were conducted in Google Scholar in May 2020. Finally, we conducted an additional Google Scholar search in December 2020 using the terms ‘confidence’ and ‘trust’, ‘formalisation’, ‘contract’ and ‘contractualization’, combined with ‘inter-organisational collaboration’ or ‘partnership’ or ‘network’, for further MRT papers.

Theory refinement

Literature for this refinement stage of our realist synthesis was identified through a combination of existing literature from prior stages (systematically searched), novel (to this stage) systematic searches intended to locate case study literature explicitly and grey literature sources (for identifying organisational reports and evaluations), as is typical of a realist synthesis.134 Case studies from the existing search were included here, with these case studies being brought over into this refinement stage of the synthesis. In addition, a novel systematic search was conducted on 10 June 2020 on the Social Policy and Practice database to identify additional case studies. Further searches for grey literature were conducted on 7 October 2020 and 8 October 2020 on UK-specific websites for evaluations of collaboration types, including The King’s Fund (London, UK), the National Institute for Health and Care Excellence (NICE) (London, UK), the Nuffield Trust (London, UK), The Health Foundation (London, UK) and NHS Employers (London, UK). The searches were on the publication sections of each website, with a focus on identifying evaluative reports. The searches used the terms ‘collaboration’, ‘partnership’ and ‘integration’ and were limited to 2012 onwards to maximise relevance to contemporary developments in collaborative arrangements. Figure 8 depicts the full methodology of this realist project.

FIGURE 8 Depiction of phases in developing realist theory. Adapted with permission from Aunger et al.135 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text. (Figure not reproduced here; see the original article.)

Inclusion criteria

Theory gleaning

Selection of documents was performed on the basis of relevance to the realist synthesis, as is typical of a realist review.130 The systematic review used the following inclusion criteria for the title and abstract stage: ‘the paper clearly relates to collaborations between one or more public sector organisations on either a structural or individual level’ and ‘the paper is a case study, evaluation, opinion, or review’. In the full-text screening, as well as that for relevance, the paper had to include ‘propositions about the success or failure of collaboration in the public sector, mechanisms underlying how collaboration works, or include information about “entry points” (i.e. drivers of collaboration)’. Exclusion criteria for all stages included papers that ‘relate to collaborations or partnerships between staff and patients rather than between organisations’. Titles and abstracts were screened by Justin Avery Aunger, with a subset of 10% screened by Ross Millar in accordance with other systematic reviews.12 Agreement was reached for all selected papers.

Theory refinement

For the refinement stage, we included only papers that (1) were case studies or evaluations (defined as reporting results of arrangements using descriptive methods), (2) report on an IOC between health care-providing organisations and (3) were in English (because of resource limitations of the study). Some papers had both literature review and case study portions, and these papers were also included, but data extraction was performed on the case study parts only. Selected studies were then subject to rigour and relevance checks in line with realist synthesis methodology.

Rigour and relevance screening

In line with guidance from Wong,136 the screening for rigour was ongoing during the analysis process and aimed primarily to increase the trustworthiness of the findings. This process involved including a CMOC only when supported by (1) clear data in included studies and (2) multiple sources.136 For theoretical sources of evidence, only theories that had seen significant use in the literature since publication were used in the building of our MRTs and CMOCs. If documents were excluded on the basis of trustworthiness, then the reasons for doing so were to be recorded; however, no studies or extracts were excluded on this basis.

Data extraction

Data extraction was carried out by one reviewer (JAA), which involved combing the included papers for information relating to mechanisms underlying collaboration, programme theories and contextual factors (often termed ‘success factors’ or barriers). As is typical of a realist review,134 identified passages in the documents were highlighted for relevance, before being extracted into separate documents according to realist logic and how they aided in understanding the intervention. This was performed using custom data extraction forms (available on request).

For the refinement stage, another custom data extraction form was created, which recorded the study, collaboration type, primary driver (as best deduced from the study), CMOCs that fit into prior theory and novel CMOCs (which could be novel in context, mechanism or outcome) that did not fit wholesale into the prior theory. This type of custom form is typical in a realist synthesis and is available in Report Supplementary Material 1. In addition, we attempted to extract information on whether studies were reporting on externally mandated forms of partnering or voluntary forms, but it was not always possible to determine this information, unfortunately, because of inconsistent reporting by authors.

Realist synthesis methods

Theory gleaning

The highlighted passages from the included documents were coded according to whether the passage sheds light on entry points into collaboration, contextual factors, mechanisms or other elements relating to collaborations that helped elucidate the underlying ideas and assumptions regarding how collaboration was intended to work and the sorts of contextual features that might shape the different mechanisms underpinning these. Most successful factors and barriers were typically identified to be the inverse of one another, and so these factors/barriers were amalgamated into becoming contextual factors at a later stage of the synthesis. As more papers were extracted, categories that were found to be thematically similar were merged to result in the final categories seen in this review. Contextual factors, mechanisms, outcomes and entry points into collaboration were coded separately, but contextual factors had their posited underlying mechanisms recorded alongside them, as well as any potential outcomes. The sources that supported the existence of these contextual factors were also recorded. Synthesis results were regularly discussed by Justin Avery Aunger and Ross Millar to maintain validity and consistency.

In some cases, mechanisms were explicit in papers identified in the systematic review and in other cases the evidence was missing. Therefore, in cases where analysis was completed and mechanisms were missing, a purposive search was used to locate MRTs that could elucidate mechanisms that were triggered by these contextual features inherent to collaborations. Contextual factors were then clustered according to their underlying mechanisms and the case study and review literature, and MRT evidence synthesised. The theoretical clarity of mechanisms and the evidence underpinning them were discussed by two authors (JAA and RM), and CMOCs were then formed. Included documents then underwent a second pass, using specific search terms relating to mechanisms, and identified contextual factors to ensure all sources of relevant information were included.

Theory refinement

Using the existing realist programme theories from step 2 of this realist synthesis process as a base, we aimed to test our existing CMOCs against case studies and improve our understanding of how CMOCs are situated temporally and causally to improve our theory of collaboration in health care, and this constituted phase 3 of our overall analysis (see Figure 9). Literature was identified through systematic searches of databases and searches of organisational websites. The literature was then categorised by collaboration type, as well as whether the collaboration was a mandated or voluntary arrangement (as could best be identified), and the literature was then rigorously searched to identify CMOCs. Testing of existing CMOCs then occurred against the newly identified literature from this stage’s searches, and this comprised identifying whether CMOCs were identical to the existing CMOCs from the theory gleaning phase, or could be considered novel in terms of context, mechanism or outcome content or novel in terms of the relationship of one CMOC to another. Although we did not intend for the focus here to be on theory gleaning, novel CMOCs were still included when identified with sufficient evidence to support them. Both CMOCs from the existing theory that had support as well as novel CMOCs not present in the existing theory were recorded. Any conflicting information about the configuration of existing CMOCs was also recorded.

FIGURE 9 Evolution of literature synthesis by phase of review.61,138 Adapted with permission from Aunger et al.135 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text. (Figure not reproduced here; see the original article.)

The process resulted in significantly more overall CMOCs than were present in our prior realist phase, and this allowed us to gain a greater understanding of how the outcomes of certain CMOCs can become a context for another further down the chain. To identify these relationships, these CMOCs were deductively coded in NVivo 12 into categories according to their mechanism to better investigate the literature for presence of demiregularities, which, in realist terms, are patterns of how outcomes generally come to occur.137 The data were then used to refine the MRT and programme theories to provide a better understanding of the links between these elements. This chapter was written according to the RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) II reporting standards.134 All data requests can be submitted to the corresponding author for consideration.

Methods: realist evaluation

Objectives

We conducted a realist evaluation to further test our refined programme theory by exploring the experiences of a range of stakeholders across several examples of IOCs in England. Our primary objective was to test the mechanisms of collaborative functioning and CMOC derived from literature against the stakeholders’ views to explore whether previously identified mechanisms and CMOCs, and causal links between them, were affirmed, refuted or revised, and to refine our understanding of how IOCs work, in which circumstances and why. In addition, we sought to elucidate the part of the causal chain that links collaborative behaviour to collaborative performance to identify how and why performance benefits may arise from the process of collaborating. We intended to produce a refined realist programme theory rooted in both literature and practice, with a view towards practical use in the future.

In the realist evaluation, for collaborative functioning, our prior programme theory135 was adopted as a MRT, against which interview data were ‘tested’, based on the realist synthesis, and this means that we explored if and how interview findings affirmed prior CMOCs, proposed refinements to existing CMOCs or, and to what degree, identified CMOCs that were novel. This theory, which we adopted as the MRT, will be outlined in the following realist synthesis chapter.

For understanding collaborative performance, Dickinson and Sullivan’s48 framework (adapted from McKenzie216) was adopted as a MRT to inform our analysis. Dickinson and Sullivan’s48 framework was selected as a MRT for three main reasons: (1) the framework focuses on performance rather than the functioning aspect of the causal chain; (2) the framework provides three criteria against which realist mechanisms can be categorised and (3) the framework captures cultural efficacy, which neatly complements the concept of programme mechanisms as changes in participant reasoning within realist theory. Thereby, we assume that frameworks relying purely on the ‘techno-bureaucratic’ aspects of performance would likely neglect key social components of the causal chain. Enabling a greater understanding of collaborative performance forms an essential part of the overall programme theory.

Therefore, the final aim of our realist evaluation was to produce a refined MRT, seeking to answer ‘what works in IOCs, for whom, under what circumstances, why and how?’, as well as ‘how do performance improvements in IOCs in health care arise, why, and what underpins them?’. This chapter was written in accordance with the RAMESES II reporting standards for realist evaluations.139

Details on programmes evaluated

The ‘programmes’ evaluated comprised a range of IOCs, as outlined in our initial realist theory paper.61 The programmes included arrangements of relatively low integration (i.e. buddying) through to highly integrative types, such as mergers. Our interviewees had direct experience of four different types of IOC and comprised five examples of hospital groups, two alliances, three ICSs and two mergers. The IOCs reflect a range of drivers and contextual differences. Although mergers result in formation of a singular organisation, we consider mergers to be collaborative entities during the merger process until the fully merged organisation begins operating38 (see Table 2 for full details on the types of IOCs and regulatory organisations included in this evaluation).

Data collection methods

The realist evaluation drew on interviews conducted with an ‘issue network’140 of stakeholders, comprising the leaders or architects of collaborative programmes, regulators, policy-makers, professional bodies, front-line staff and patient representatives. We defined these stakeholders as ‘a broad collection of individuals possessing knowledge about the issue in question with some influence on policy outcomes’.

Recruitment process and sampling strategy

Participants were identified through contacts via our Study Advisory Group and from direct contact with potential individual and organisations identified through scoping work. Phase 1 interviews were conducted, which had a particular focus on ‘theory gleaning’ and explored current programme theories being used to develop partnering approaches across NHS providers. Recruitment to phase 1 used a purposive sampling approach, with further snowball sampling employed to identify relevant stakeholders within this ‘issue network’. Phase 2 participants were also recruited, drawing on a selection of key informants from NHS provider organisations currently engaged in a range of partnering activities. Here, particular attention was paid to testing and refining programme theories identified in from the literature and the theory gleaning interviews to produce refined theory.

Participants were chosen based on their likelihood of being able to provide rich information about various aspects of the programme theory, from being engaged in implementing such arrangements themselves to delivering the policy and regulatory agendas.

Participants were approached to participate via e-mail. Patient representatives were recruited from patient representative bodies and were intended to be greater experts on outcomes rather than ‘mechanisms’.141

Sample

The final sample comprised 37 interviews with 34 participants and one focus group with 8 patient representatives. The interviews and focus group were conducted across England between January 2020 and May 2021. Table 5 outlines the characteristics of the participants.

Interviews and setting

Semistructured interviews were conducted by experienced qualitative interviewers (JAA, RM, AMR and DF). The interviews drew on realist interview methodology, with the focus on gleaning and refining theory.141 Questions were posed that would both directly and indirectly work towards these aims. Earlier phase 1 interviews were more focused on theory gleaning and were conducted alongside construction of the programme theory in our earlier realist review.138 Later phase 2 interviews incorporated direct questions, in which the programme theory was explained to the participant and the participant was asked about particular elements, with the intention of refining and consolidating theory directly in a manner in line with the teacher–leaner cycle.141 This strategy was used particularly with the participants with whom we were able to conduct follow-up interviews. The interview guide was refined and developed as our interviews progressed. Owing to the emergence of the COVID-19 pandemic shortly after the start of the research, interviews were conducted virtually over Zoom (Zoom Video Communications, San Jose, CA, USA) or Microsoft Teams (Microsoft Corporation, Redmond, WA, USA) and recorded on an external dedicated encrypted audio-recorder. Interviews lasted between 30 and 90 minutes, but were typically closer to 60 minutes in length. Files were sent for verbatim transcription at a third-party transcription service.

Data analysis

Analysis was performed in NVivo 12 software by one coder (JAA). Coding logic was independently verified by a second coder (RM). Coding was performed retroductively.142 Retroduction uses both inductive and deductive logic.143 Retroduction involves thinking through what causal powers might be at work in producing observed patterns or changes in patterns. Retroduction is underpinned by a belief that an understanding of causation cannot be achieved using only observable evidence. Use of retroduction meant that analysis was carried out in a deductive manner for features relating to our existing realist theory; however, for better sorting and understanding of themes, codes within higher-order codes were created inductively where commonalities were identified.144

For the collaborative performance analysis, the framework of cultural efficacy, technological effectiveness and organisational efficiency was used as broad codes in the interpretation of the results and the mechanisms identified. Therefore, higher-order codes for excerpts relating to cultural efficacy, technological effectiveness and organisational efficiency were created prior to the start of coding. However, within these higher-order categories, we inductively created subcodes for excerpts that supported various concepts, such as ‘communication’ and ‘improved reputation’, as demiregularities (or semi-predictable CMOC patterns) were identified in the transcripts. Within the codes for these concepts, we then analysed the excerpts for identifiable connections and outcomes to formulate the explicit CMOCs. Transcripts themselves were not anonymised, as the coders were also the interviewers, rather any information that would identify participants was withheld from this final report.