Discussion

Shared learning

In total, 14 scoping considerations were identified, each evident in two or more of the eight case studies. (The development of this framework of 14 scoping considerations is described in Chapter 2, Thematic analysis.) These were then checked and (where necessary) refined by case study authors at each centre, and so should not be viewed as a formal or rigorous qualitative thematic analysis. Table 11 shows the final 14 scoping considerations together with the case studies that illustrate them.

Consultative/external issues in scoping and question formulation

Consultative/external issues relate to liaison with either the commissioner (i.e. HSDR), the direct client (e.g. NHS England) or the stakeholders of a review. Typically, these issues involved communication, engagement, negotiations, managing expectations, conceptualisation and definitions.

Managing and deciding priorities (consultative issue 1)

This theme captured how the review team manages and negotiates with NIHR, policy stakeholders and other customers to ensure that priorities are addressed within resource constraints. In most review scoping processes, the choices made represent trade-offs between various competing goals, especially between the duration/timeliness and comprehensiveness of a review (i.e. resources) and its rigour and relevance. Where a policy or health service customer of a review is identifiable, it is important to ensure that they are aware of these trade-offs and the scoping decisions that affect them.

A clear example of this need to manage trade-offs comes from the rapid review to inform the independent review of the MHA16 (i.e. case study 2). The initial question was expressed as a policy question rather than a research question, and would conventionally be answered by finding comparative quantitative effectiveness studies. Our preliminary searches revealed few, if any, such studies of this kind. To both reduce and share the risk with the policy customer, the review team expanded their focus and question to patient, carer and professional experiences of the ‘nearest relative’ provisions and, at the same time, made some aspects of the review scope and protocol conditional on how much evidence we found. This process was achieved through close and regular consultation with the policy customer.

These discussions and negotiations also taught the review team about particular aspects of reviewing evidence about legislation (as opposed to health interventions). Legislation, by definition, is jurisdiction specific and so this reduces the value of research evidence about similar legislation in other countries. Similarly, sometimes the legal context changes so significantly at a particular time point that any evidence from before then becomes far less relevant. In this rapid review, this gave us a defensible rationale to exclude evidence from before 1998, which was when the UK’s Human Rights Act22 came into law (and, we were advised, this significantly altered the operation of the MHA16).

The key point, perhaps magnified in the context of such a rapid review, is that trade-offs between rigour and relevance, timeliness and deliverability of the review, and all of these with closeness to the perceived original evidence need, have to be carefully negotiated with and fully involve the policy customer. Where the main policy customer is identifiable, the review team needs to present these choices, and their likely implications for these trade-offs, as far as they can be anticipated in the early review stages. Outlining the available choices is also helpful when a network or group of key stakeholders are the main users or audience of the review, as in the review of the regulation and inspection of health and social care, although it makes the process for agreeing the review’s aims more complicated. This consultative issue, therefore, links to all three interface issues and the two-way discussions between stakeholders and the review team that often take place to identify the important gap in synthesised evidence to balance rigour and relevance and to maximise applicability (for our teams, to the UK health and social care context)

Reconciling different priorities/perspectives (consultative issue 2)

This theme captured how the review team manages potentially competing tensions between what different groups or stakeholders may want to achieve within the overall project remit. The need to reconcile priorities was perhaps most evident in the review of the regulation and inspection of health social care (i.e. case study 3), where the diverse stakeholders included both national service regulatory bodies (such as the Care Quality Commission and Health Inspectorate Wales) and the regulatory bodies of different types of care professionals. The review team noted that ‘the stakeholders emphasised slightly different areas of interest, deviating to a greater or lesser extent from the original questions’.

In the review of hospital-led intervention to reduce length of stay for planned admissions (i.e. case study 6), individual (rather than institutional) stakeholders had different reasons for agreeing on the overall importance of the review topic for older people. One stakeholder noted the overall importance of the goal of reducing the length of hospital stays for older people to reduce the risks of hospital-acquired harms, like falls or infections. Another stakeholder felt that older people may often be excluded from enhanced recovery programmes, perhaps because of perceptions that evidence of benefit was lacking in this patient group. However, a clinical stakeholder was aware of the opposite risk (i.e. older people might be subject to enhanced recovery protocols without evidence that it is safe for them or the protocols may not take account of more complex needs). Therefore, in this case study, the differing stakeholder views did not pull the review scope in different directions, but rather reflected the underlying uncertainty of the initial topic, specifically in relation to older hospital patients (in this case, helping to confirm a narrower focus of shared interest).

Achieving buy-in and engagement (consultative issue 3)

This theme captured how a review team secures input from stakeholders into the scoping and topic prioritisation process. Although the focus is on the initial review stages and on developing a detailed review scope and protocol, we acknowledge that these processes have importance beyond scoping and question formulation.

Although often implicit within our case studies, an important incentive for a stakeholder to engage in question refinement and topic scoping is when their organisation has proposed the need for the review. For example, in the review of the regulation and inspection of integrated care (i.e. case study 3), the topic had originated from Health Inspectorate Wales, and this organisation and Care Inspectorate Wales were strongly engaged with the review throughout. This case study suggests another potential incentive – for buy-in and engagement from equivalent agencies in the different devolved nations of the UK – and so, ultimately, the scoping of this project also involved the Care Quality Commission (for England) and the Professional Standards Authority [for the UK (health professionals) or England (social workers)]. We further speculate that securing the input of lead agencies in relevant jurisdictions may incentivise equivalent agencies in other jurisdictions or subsidiary agencies to ‘not be left out’ and to represent their constituencies.

A review team needs to reconcile different priorities and perspectives held by wider groups of relevant stakeholders, often with equivalent or overlapping remits [see Reconciling different priorities/perspectives (consultative issue)]. Although this could present a challenge to scoping, it may also secure greater engagement, overall, by intentionally using a transparent and collective/group engagement process. Such engagement could be achieved by meeting as a stakeholder group or, as in the review of regulation and inspection of integrated care, by sharing individual stakeholder discussions with the wider group of stakeholders. Engagement could also be achieved by ensuring that stakeholders understand that the initially specified questions may not be the ones that best serve user needs.

In contrast, the realist review of access to social care for ethnic minority and LGBT+ populations (i.e. case study 4) had a single identified stakeholder (the DHSC) and was conducted within the context of a scope that was evolving throughout the review process. Here, early and ongoing engagement seemed to be aided by the DHSC’s close involvement in shaping the core concepts that would provide the ‘anchors’ or ‘compass’ of the review. Similarly, as the review progressed, the DHSC were closely involved in discussing and shaping the emerging programme theories. However, it is unclear from this case study whether the promise of involvement throughout a review process acts as an incentive or a disincentive to close involvement in scoping and question formulation. Conceivably, there may be some policy or service stakeholders who invest time in the early stages of scoping and question specification in the hope that this will mean a need for less subsequent engagement and effort. Overall, it seems that factors that determine the strength of stakeholder engagement will be varied, complex and context specific, and may exist at an organisational level or be steered by the interest and commitment of individual policy-makers or commissioners.

Educating the end-user about synthesis process and products (consultative issue 4)

A related consultative issue in many review scoping processes is how the review team explains to stakeholders about review methodology and the choice of different potential synthesis outputs produced (e.g. given different prioritised questions or different available evidence). This closely relates to the technical scoping issues of choosing the different methods of synthesis (technical issue 1) and balancing the goals of mapping, scoping or reviewing previous research (technical issue 3). This dialogue also strongly influences the consultative issue of managing stakeholder expectations [see Managing stakeholder expectations (consultative issue 5)].

Most service, clinical professional or policy stakeholders are most aware of systematic reviews that use quantitative research, such as clinical trials, to produce integrative syntheses of evidence, for instance overall summaries of effectiveness of well-defined interventions. The review of access to social care for ethnic minorities and LGBT+ people (i.e. case study 4) involved working with the DHSC to show how a realist review might use diverse sources of research evidence and provide richer explanations of how ‘access’ worked or did not work for different groups. More generally, it involved conveying to the end-user that not all types of review have to be prespecified and that the concepts guiding them can evolve during the review, including studies that can expand to enable relevant explanations (in this case, by including refugees and asylum seekers).

Another recurrent feature of HSDR reviews experienced across the teams, and one the stakeholders needed to be aware of, was the two- or three-stage nature of many review projects. They would begin with searches and a type of rapid scoping review, and then work with stakeholders to reassess what kind of synthesis, or perhaps just mapping or description, would be feasible and useful on the basis of the number and types of studies known to exist [this common sequencing is further explored as Mapping versus scoping versus reviewing (technical issue 4)].

Managing stakeholder expectations (consultative issue 5)

This theme captured how the review team communicates what the review project will and will not be able to achieve within the available resources and time frame, particularly when the deliverable is likely to ‘fall short’ of the conventional systematic review goals (e.g. of providing a summary answer to effectiveness or cost-effectiveness questions).

A key expectation is often around the fixed or fluid nature of the review question itself [see Balancing fixed versus fluid questions/components/definitions (technical issue 2)]. In the realist review of access to social care for ethnic minority and LGBT+ groups (i.e. case study 4), uncertainty related to the definition of key terms and concepts, and in the recognition that the review would have to refocus at a later stage (in this case, in a realist review, when a smaller set of programme theories are chosen as the basis for the rest of the review). In addition, with realist reviews, the synthesis and end product comprises a set of refined and empirically grounded programme theories.

It is important to have early conversations with stakeholders about the variety of possible synthesis methods, their relative suitability for answering different types of review question and the expected content and format of their findings [see Choosing the methods of synthesis (technical issue 1)]. At the same time, the review team needs as much clarity as possible about the other PICOS elements that define the boundaries of a review’s scope [i.e. population, intervention (phenomenon), comparator, outcomes and study type]. Perhaps the basic strategy for managing expectations about a review is to develop the review protocol iteratively and collaboratively with the main stakeholders or user.

Interface issues in scoping and question formulation

Interface issues concern how scoping and question formulation occur in transactions and negotiations between the review team and other interested parties. Negotiations may require translation (as between the languages of policy and research), interpretation or the realignment of the time and quality and/or resource implications of the review. As such interactions involve identifying and agreeing the niche/gap that meets users’ needs while seeking to complement previous reviews, and balancing rigour, relevance and applicability, these considerations were evident, to some degree, in all our case studies. Only in situations with no identifiable review user and little stakeholder engagement would interface issues not be prominent.

Identifying the niche/gap and optimising added value (interface issue 1)

This theme captured how the review team positions the intended synthesis product within previous literature or reviews and in addressing users’ specific needs. This builds on the principle that a new systematic review adds most value in situations where:

1. the potential review question matches or is very similar to a current and important policy or health service question
2. a sufficient number of primary research studies have directly or indirectly generated evidence to answer the potential review question
3. no recent, high-quality systematic review has sought to answer the same or a very similar review question.

Where any one of these three conditions is not met, the value of a new, high-quality systematic review is significantly diminished. Many scoping processes can be viewed, in retrospect, as an iterative process of trying to establish these conditions. Nevertheless, the central interaction between the review team and the review users is based around condition (a), whereas conditions (b) and (c) are technical and data driven and do not require much input from review users.

The review of access to health care for adults with IDs (i.e. case study 8) illustrated the complexity of building on or filling gaps between previous systematic reviews. Although, ostensibly, the earlier systematic review had addressed a similar question and ‘simply’ needed updating, concepts and terminology relating to ID and to accessibility had changed considerably. This meant that search strategies had to be redesigned and the scopes of the earlier and the new review carefully reconciled. In addition, the methods and, by current standards, the quality of much older systematic reviews may mean that a review update that replicates the methods of the earlier review would be neither useful to stakeholders nor appealing to academic review teams.

In situations where a recent high-quality systematic review exists, the implied gap or opportunity for justifying a new systematic review may lie in asking different questions within the same topic (intervention, population and outcome) boundaries inhabited by a documented body of research evidence. For example, in the review of interventions to reduce preventable hospital admissions (i.e. case study 7), the earlier (2012) systematic review of effectiveness by Purdy et al.45 showed where and in which specific patient groups there was sufficient evidence of effectiveness for a realist review to add value. Apparent similarities between previous and currently planned reviews, or apparent gaps/differences between high-quality systematic reviews, requires careful exploration to confirm the existence of genuine gaps and a clear need for an updated or new systematic review.

The review of digital innovations in primary care (i.e. case study 1) illustrated challenges related to identifying the gap/niche and meeting user needs. In this case, the user (i.e. NHS England) had identified numerous needs (nearly 20 questions within four themes), which led to an iterative, multistage approach with interim evidence synthesis products to inform potential next stages. This included an initial report, presenting a scoping review, which helped to focus the final stage synthesis on a narrower and more manageable set of prioritised questions.

Finally, both review teams and review users should be open to the possibility that no clear niche or gap might usefully be filled by a new evidence synthesis within a given topic. Such a situation exists when no primary research addresses the main questions and uncertainties of the policy customer, or where such primary research does exist and has already been summarised and synthesised in a high-quality systematic review. This issue links to earlier consultative issues on managing stakeholder expectations (consultative issue 5) and educating the end-user about synthesis processes and products (consultative issue 4).

Balancing rigour/reliability (reproducibility)/relevance (interface issue 2)

This theme captured how the review team manages potentially competing tensions of scientific quality, confidence in the review output and utility to the intended users within the constraints of remit and resources.

The review of digital innovations in primary care (i.e. case study 1) showed how meeting user’s needs might involve departing from conventional standards of academic rigour and reporting. They chose to meet their user’s needs not by conducting a ‘full systematic review’, but, instead, by adopting systematic review methodology wherever feasible. This decision ultimately meant that their evidence synthesis products were also less conventional, going beyond the summary and synthesis of previously published research within a written report. For example, they produced an interactive resource (i.e. a type of evidence map) for identifying relevant evidence, alongside information on ongoing relevant research and links to academic researchers active in the fast-changing area of service innovation.

Similarly, the evidence map about access to health care for adults with IDs (i.e. case study 8) traded-off the quantity of literature covered against data validity. The team constructed their evidence map using evidence from study titles and abstracts, without verifying the abstracts against the full text of the included sources. Such trade-offs are common when seeking to achieve timely and deliverable review products that still have credible quality.

Transferability/applicability of study evidence to the policy/service/service user context (interface issue 3)

This theme captured how the review team manages the need to provide UK-specific interpretation from evidence bases drawn from other countries and contexts. A key and commonly negotiated choice, evident in several of the case studies, relates to whether to include international evidence or restrict to studies conducted in UK health-care and social care settings.

Case studies 2 and 5 both identified specific policy customers and a phenomenon of interest that was explicitly based in legislation or the application and use of legislation (respectively, the MHA’s provisions for involving trusted relatives or friends in compulsory detention16 and the strengths-based working by social workers as promoted by the Care Act 201436). This meant that applicable evidence was likely to be jurisdiction specific, and studies from outside the UK were likely to be of limited relevance. Both of these reviews were commissioned within tight, externally imposed timelines (2 and 6 months, respectively), adding a further pragmatic reason to consider restricting to UK studies. However, in both cases, this presumption had to be discussed and confirmed with policy stakeholders. In case study 2, to inform reform of the MHA,16 interest in looking at international evidence was balanced by existing variation in both relevant legislation and in related evidence between the UK’s four devolved nations.

Other contexts where international evidence may have less applicability might include accessibility and patient experience, especially when linked to cultural, ethnic or demographic factors (as in case study 4, about access to social care for ethnic minorities or LGBT+ people). In contrast, it is often presumed that hospital-based care systems are similar across high-income countries and, therefore, evidence is applicable between them. For example, in case study 7 (a review of the implementation of interventions to reduce hospital admissions), evidence was sought from the major anglophone high-income countries: the UK, USA, Australia, Canada and New Zealand. However, both case studies (4 and 7) involved realist approaches to synthesis, in which the transferability of evidence between studies, and from studies to the intended review user’s setting, is achieved through identifying shared theoretical mechanisms across different contexts. Therefore, applicability and transferability of research evidence is not a static, inherent property of the evidence itself and where it has come from. Rather, it is judged in relation to the models of causality on which the particular synthesis methods are based. Consequently, some ‘configurative’ methods – like realist synthesis – require studies from diverse contexts to develop their outputs, which is in contrast to conventional ‘aggregative’ methods (like quantitative meta-analysis) that work best when between-study heterogeneity is minimal.

Technical/internal (review team) issues in scoping and question formulation

This broad category of scoping considerations relates to how issues of method and reporting, which are typically outlined conceptually in textbooks and handbooks on systematic review methodology, work out in practice. Technical issues relate to choice of methods, depth of analysis and format of presentation of review findings. These considerations seek to inform methodological choices that are both rigorous and appropriate to the review question, audience and types of data/study available. These issues are technical in the sense that they rely more, but not exclusively, on the skills and experience of reviewers and information specialists in matching the detailed review methods to the review questions, and to the quantity and quality of available evidence. These considerations, therefore, include what might be called the ‘science’ of scoping, drawing on shared and established principles and research methods.

Choosing the methods of synthesis (technical issue 1)

This theme captured how the review team explores different options and makes an informed decision about which type of synthesis product will best meet the needs of the intended users. The steps to arriving at a particular method of synthesis varied considerably between the reviews in our case studies, as did the final type of evidence synthesis delivered. These eight case studies comprised effectiveness reviews (typically, narrative syntheses), realist reviews, scoping reviews, evidence maps and reviews of qualitative evidence. The outcomes of interest spanned effectiveness, cost-effectiveness, patient and carer experiences, accessibility and implementation.

In general, the starting point for choosing the method of synthesis was the review question, based on translating the policy customer’s evidence need into a specific outcome of interest (e.g. ‘what works?’ becomes ‘what is the effectiveness of . . . ?’). Within the review question, the explicit or implied outcome of interest was a key determinant of whether quantitative or qualitative evidence was sought and synthesised. For example, in case study 6, the study of hospital-led approaches to reduce length of stay for planned admissions, the review questions were framed around evaluating the clinical effectiveness and cost-effectiveness of alternative approaches. These questions can be validly and reliably answered using only quantitative comparative evaluations, like RCTs, and evidence synthesis methods, like narrative synthesis alongside meta-analysis, typically using an integrative approach to synthesis (i.e. ideally, to produce a generalisable summary answer to the question).

In case study 1, to synthesise evidence on Digital First Primary Care, the questions were so broad and the potentially relevant research so diverse, that a thematic synthesis approach was adopted, largely to accommodate multiple questions and types of evidence. The York team described this as not being a full systematic review, but a process of transparently identifying, critically appraising and summarising the research evidence relating to these questions.

The second main consideration that drove the choice of synthesis method was the quantity and design of available primary research studies. For example, in case study 2, the focus was on patient and carer experiences (of the ‘nearest relative’ provisions) and the scoping searches mainly identified qualitative research studies. Therefore, some type of rapid/pragmatic qualitative synthesis was required. That the synthesis method cannot always be rigidly specified at the protocol stage is further underlined by case study 4. The Sheffield team reflected that ‘The choice of review synthesis approach is not a decision to be made once the scope has been finalised – it represents an integral part of the scoping process (PICOSS)’ and this underlines a wider blurring of the conventional separation of scoping and protocol agreement and commencement of the systematic review [see Scoping as a continuous process through to shaping the end product (technical issue 5)].

Conversely, certain approaches to evidence synthesis are seen as too labour-intensive and time-consuming to do well in the context of rapid and policy-responsive review teams, notably meta-ethnography for reviewing qualitative evidence (although rapid meta-ethnographies have been conducted). In addition, of the eight case studies presented here, only one (i.e. case study 6, on strategies for reducing hospital length of stay in older people) addressed an effectiveness question where a sufficient number of high-quality randomised studies existed for meta-analysis to be feasible. However, even in this systematic review, the meta-analyses included only a smaller subset of all studies comparing the relevant alternatives.

Balancing fixed versus fluid questions/components/definitions (technical issue 2)

This theme captured the extent to which the question as a whole and/or its individual PICOS components are predefined or if they emerge during exploration of the literature. This balancing act seems to be a dominant trade-off for responsive reviews in the health and social care services and policy field, as exemplified in seven of our eight case studies. This technical issue is linked to the consultative issue of managing stakeholder expectations (consultative issue 5) and the technical issue of determining which review activity, of mapping, scoping and reviewing, will take place. Responsive, rapid reviews for policy-makers are multistage and, unlike RCTs, offer valuable opportunities for review and interim revisions of aims and methods.

A key example of this balancing act is case study 6, in which the plans for evidence synthesis had to be significantly adapted when > 100 effectiveness studies were identified (according to the original inclusion criteria). This led to a revised plan for evidence synthesis that sought to balance rigour and applicability. The review included two evidence syntheses: one of comparative effectiveness studies from the UK only (but of all high-quality comparative designs) and the other of studies from any high-income country, but RCTs only. This hybrid approach meant that review users interested in applicability to the UK could use the first synthesis and those interested in the most scientifically rigorous (internally valid) evidence across diverse contexts could use the second synthesis. As a result, the review avoided the effort of having to data extract, quality assess and synthesise the less rigorous evidence from beyond the UK. The realist review of access to social care for ethnic minority and LGBT+ populations (i.e. case study 4) was also conducted within the context of a scope that evolved throughout the review process.

In short, although review protocols are drafted and initial inclusion criteria and synthesis plans are prespecified (and even published, e.g. in PROSPERO), it is acknowledged that they will likely change. Moreover, even where the question remains fixed, learning and clarity about key concepts and terms within it are expected to emerge throughout the review. These technical aspects, therefore, go hand in hand with interface and consultative issues as the evolving protocol for the review is transparently revised, especially in consultation with the ultimate policy end-user.

Taking stock of (and building on) what studies exist (technical issue 3)

This theme captured how the review team explores the quantity, quality and characteristics of primary studies and/or reviews in determining which output will be both feasible and useful.

Sometimes, as in case studies 7 and 8, a key previous review offers a starting point for the new review. In the review of interventions to reduce preventable hospital admissions, the effectiveness review by Purdy et al.45 provided a useful sampling frame of medical conditions on which the realist review might focus in depth. In contrast, in their review of access to care for people with IDs, the much older review by Alborz et al.,49 ostensibly on the same topic, did provide some reusable resources (such as an interview schedule). However, it largely seemed to serve as a marker of how concepts of disability, notions of access and frameworks for understanding the performance of health services have all advanced significantly since 2003, and that the focus and intended value of the new review had to be carefully articulated.

In other reviews, taking stock is directly a process of assessing the diversity and quantity of available evidence and tailoring the planned synthesis to make the best of what evidence is available, in relation to the review’s questions. However, these decisions necessarily occur once searching and screening is mostly complete and so may not strictly constitute the topic-scoping phase.

Mapping versus scoping versus reviewing (technical issue 4)

This theme captured how the review team manages the relationship between exploring the characteristics of the evidence base (mapping), determining the parameters of the specific planned synthesis (scoping) and conducting the synthesis (reviewing), and the extent to which these processes transform into discrete project deliverables. This common three-stage sequence of responsive reviews is partly a response to the risks of overspecifying the planned final evidence synthesis at the beginning (and also relates to technical issue 2 of balancing fixed vs. fluid questions).11 Although they are conceptually useful, we recognise that, in practice, scoping, mapping and reviewing are processes that necessarily interact and often overlap, as well as being labels applied to review outputs that may signal a formal end or transition of a stage in a continuous process.

Scoping as a continuous process through to shaping the end product (technical issue 5)

This theme captured the extent to which the scoping process is used as an opportunity to precondition the users to the content and form of the final synthesis product. Through their potential engagement, the scoping process offers an opportunity for commissioners and stakeholders to have an early influence on tailoring the final product to their requirements

Therefore, in case study 7, the focus on cardiovascular and respiratory conditions was determined not only by evidence on effectiveness, but also from the commissioner perspective, as articulated by the NIHR programme staff. For the review team, it is also part of a risk-averse strategy that minimises the chances of producing a product that is neither useful nor appropriate. In case study 4, the review team shared a previous similar realist review of access to care for a different marginalised group (i.e. older people in rural areas) to show the end-users the kind of findings that a realist review can produce [similar to the earlier consultative issue of ‘educating the end-user about synthesis process and products’ (consultative issue 4)].

Choosing between general and specific and broad and deep focus (technical issue 6)

This theme captured how the review team decides whether to cover an entire topic or to select one or more subtopics as exemplars of the whole, and the extent which they optimise coverage compared with detail. This includes decisions about whether to primarily describe and not to synthesise studies (as in an evidence map), or to also critically appraise and analyse included studies within a formal synthesis (e.g. a framework synthesis or narrative synthesis).

In the review of experiences of strengths-based approaches to social work (i.e. case study 5), this decision involved a choice between reviewing evidence explicitly about strengths-based practice and working itself, although it is a nebulous and loosely bounded set of principles and concepts, and ‘deconstructing’ or ‘representing’ strengths-based working as a collection of specific programmes and initiatives that are widely seen as fostering or exemplifying strengths-based working. Through tabulating documentary sources and consulting with diverse stakeholders, we ultimately identified 17 subsidiary approaches and programmes that were seen as fostering strengths-based working. Although this made searching for named programmes more straightforward, it did, however, risk treating the 17 approaches/programmes as equally strengths based in approach (i.e. not appreciating the substantial differences between them).

The review of interventions to prevent avoidable hospital admissions (i.e. case study 7) also showed the challenges to review teams of selecting a representative and manageable body of evidence to synthesise from a larger whole. The challenge in this case lay in the plentiful ACSCs and the solution was to choose a smaller selection to be reviewed in detail as useful exemplars for understanding others (e.g. cardiovascular and respiratory conditions). These two conditions had been sufficiently explored, and adequately supported as effective, to make these an appropriate focus for a realist inquiry.

Summary of the findings

This report has presented the scoping stages of diverse review projects commissioned or conducted in direct response to the expressed or perceived needs of health policy-makers or health and social care service commissioners. Despite the diversity of topics and goals, and their being conducted by three independent university-based research teams, they reveal similar challenges and draw on common principles and processes developed to scope topics and formulate review questions.

Iteration and perseverance

Several case studies highlight that effective topic scoping and question formulation takes time and effort. Reviewers should not expect the ‘optimal’ review question to emerge from a single cycle of scoping searches and stakeholder consultation. Related to this, it is important to involve as many members of the review team as possible in the discussions (or shared notes) from stakeholder meetings, reading key sources and the screening and discussion of scoping searches. This maximises the chances of identifying recurrent needs or uncertainties in previous reports or stakeholder opinions about where clearer evidence is needed. It also creates a shared understanding of the key terms, service/policy contexts and complexities relevant to a particular review.

Another time-consuming, but ultimately productive, element of many of the scoping processes described is the time taken to identify, approach and build relationships with the key stakeholders. This often includes following false leads and the use of multiple strategies (e.g. e-mail, telephone) to contact people in relevant organisations who are able and willing to share their experience and knowledge.

Navigating between ‘the two fears’

One way of understanding scoping processes is as a process of navigating between ‘two fears’ or averting two risks: (1) the fear of finding and having to summarise and synthesise too much research (posing a risk to deliverability) and (2) the fear of finding too little or no research (posing a risk to the usefulness of the review findings). Many decisions made during scoping try to anticipate and avoid these two extremes to produce a review that is deliverable, on time and within resources, but that is also as useful as possible in addressing policy-makers’, service commissioners’ or clinicians’ evidence needs. Nevertheless, ‘too much’ evidence is also defined in relation to the (typically fixed) size of the review team, and the general expectations that a review and report will be produced in a certain time frame (e.g. 6, 8 or 12 months). These trade-offs were typically shared and discussed with the policy customers as the ultimate arbiters of what combination of scope (i.e. size), timeliness and relevance of review would be most useful to them. This key tension between finding ‘too much’ evidence or finding ‘too little’ has also been noted in relation to systematic reviews as student projects.2

Although finding sufficient high-quality studies to create a useful answer is the ideal, there are circumstances where producing an ‘empty review’ (i.e. finding no good-quality studies to answer a particular question) remains useful to policy-makers. Authoritatively concluding an absence of reliable scientific evidence on currently implemented service designs or policies may legitimise innovative service and policy change, or fully justify the need for funding rigorous research and evaluation of current and new services. Given the considerable time and other resources involved in producing an empty review, it is worth establishing in advance, from the main policy customer(s), that such an outcome would still be valuable. The review team will also inevitably be aware that an empty review is less publishable.

Negotiation or not negotiation?

Team members were divided over the extent to which the process of determining and agreeing the scope should be considered ‘negotiation’. Some resisted the business or commercial implications of ‘negotiation’ and drew attention to mutual interests, of research team and research commissioners, in seeking to deliver a rapid review that meets a genuine health service need in a way that minimises inefficiency and duplication and optimises the use of public funding. Certainly, the teams considered it helpful that negotiations did not start from an adversarial standpoint. In this sense, resolution of the negotiations comes not necessarily in agreeing a review protocol, but in a necessary prerequisite of reaching a common understanding. Our teams found that the protocol only loosely served as a contractual safety net, but more often served as a snapshot of an intended direction, and perhaps had most value through the process of sharing and negotiating it to develop a shared understanding.

From this shared platform, both review team and review commissioner can then adopt a position from which to take forward detailed plans for the review. This understanding did not simply relate to the topic under review, but extended to acknowledgement of what might reasonably be achieved within the resources available. Others saw the process from a public health perspective of negotiation, in recognising that negotiating and influencing are particularly important when these skills offer the only route by which to deal with external organisations and partners.

Managing stakeholder expectations (consultative issue 5) was seen as particularly important and the tension between underselling the potential value of the review and overpromising on what might be delivered needs to be handled within each individual project and each year’s programme as a whole. A key challenge is helping the commissioners of a review to gain clarity about what a review might be expected to address. The teams reflected on classic models of information needs and negotiation, whereby such a lack of clarity might be accompanied by a need for ‘sense-making’ by which to resolve ‘questions, confusions, muddles, riddles and angst’.54 This explains why careful defining of terms and of the research problem constitutes an essential first step.

Another resonance with negotiation occurs where the NIHR HSDR programme staff act as intermediaries, or even mediators, between the review teams and diverse stakeholders, in managing stakeholder expectations (consultative issue 5). End-users may not realise the resource implications of different review methods or their suitability for their questions. In contrast, the funders, as more experienced users of reviews, could help other agencies to revise their expectations more realistically.

The teams shared recognition that the process typically began at an earlier stage than negotiation (i.e. a more open-ended stage of ‘pre negotiation’ that some preferred to describe as ‘information sharing’). At this point, neither review commissioner nor review team knows fully what they might realistically achieve in connection with the underlying information need. Sharing of priorities and perspectives is a necessary prequel to reconciling those different priorities/perspectives (consultative issue 2). Some team members felt that the review team held particular influence because they were able to propose both review methodology and the associated resources to accompany each topic. At the same time, the NIHR programme team are themselves able to build up experience and knowledge on what might be achieved and expected, and are able to use this in agreeing the final ‘sign-off’.

Review teams have previously seen themselves as having an ‘honest broker’ role when interpreting topics with which they have not specifically engaged. The reality is that reviewers will also seek to optimise the benefits of the chosen review for themselves and their academic organisations, alongside meeting the needs of the review commissioners and of the health and care services. For example, review teams may privilege a given review method, from multiple options, that might offer the possibility of methodological innovation or experimentation. Alternatively, they may seek to make the review output as publishable as possible. Viewed in this micro-context, the model of negotiation – the classic ‘win–win’ situation – is seen to exert an influence on outcomes from the scoping process.

Co-production? To what extent and, if so, with whom?

The teams also discussed whether or not the questions and scopes of these reviews were actually developed through a process of ‘co-production’.55,56 Co-production, in the context of research, is the process whereby stakeholders – those who may use or be affected by the outcomes of research – work as partners with researchers to create, shape and deliver research (or, indeed, syntheses of research evidence). Co-production does not necessarily refer exclusively to the relationship between researchers and the potential end-users of the service, as most commonly discussed, but might also relate to the relationship between the funder and research team.

Related to this, one of the models the group discussed related to the researcher-in-residence model, which held particular parallels with aspects of having ‘on-call’ review teams.57 Although the review teams were not physically co-located in the NIHR commissioning organisation, they did engage and interact in numerous ways to acquire a more complete understanding of the review commissioner’s needs. The teams saw three-way co-production between funders, researchers and users as an important aspiration that could yield benefits for all parties. Although all could point to instances of successful co-production, for example in including service users or care professionals as co-researchers and as co-authors on the reports, there was general recognition of the potential to extend this further. However, this was accompanied by an awareness that the brief and episodic interaction with some of the stakeholders often impairs more genuine or intensive co-production.

The team concluded by reflecting on the extent to which ‘true’ co-production can ever be possible within a rapid response review programme when faced with such accelerated timescales. Again, managing expectations extends to being open and honest with PPI representatives about their roles in informing the direction of reviews, and making it clear to other advisory panels and stakeholders that they are coming in on a process that has already undergone considerable stakeholder involvement and/or consultation. The teams highlighted the difference between informing the topic and informing the review. They also identified how the protocol was a genuine product of co-production, albeit within the context of being more important to the review team, but of relatively little detailed interest to the review users or policy customer (who were mainly interested in the final review output).

Interestingly, we did not directly discuss how we made our choices about how much and what content or service/policy context was required for each review project, although these choices clearly greatly influence the relevance of the scopes and the final review products. In general, each project team obtained as much and as diverse, relevant and co-operative content expertise as we could, given time and other resource constraints. However, efforts to identify and secure further or more specific expert knowledge after the agreement of the protocol would, inevitably, over time compete with efforts to deliver the work itself. This aspect of conducting rapid, responsive reviews is worthy of more explicit and prospective evaluation.

Although we did not discuss it in detail, it is possible that learning materials or templates could be developed that quickly clarify and communicate these typical expectations around scoping, explain key review terminology and provide ‘bite-size’ learning about common evidence synthesis methods and products. This was achieved, to some extent, by pointing new policy or service delivery stakeholders to our websites or relevant previous reviews. Within the case study projects, such knowledge sharing with the synthesis end-users happened in a more responsive and ad hoc way (i.e. in e-mails and teleconferences), which may have led to inconsistencies and duplicated effort. However, the time constraints on the senior health policy and service delivery professionals to use such learning materials would probably also mean a more bespoke and informal process of learning from the review team would prevail.

‘Pinning down’ versus ‘keeping open’

The teams recognised that, in contrast to a systematic review, where the review protocol is an opportunity to pin down and specify the intended final review, a responsive rapid review programme needs to build in contingency and conditionality. Although specifying the work to be carried out and the timescales and milestones within which it will be accomplished is a necessary part of negotiations, it was considered important to reserve the right, for both parties, to modify overall plans if necessary. This could be seen in changing the emphasis of results, prioritising a particular subquestion for earlier delivery or renegotiating the final review methodology. The three-phased mapping–scoping–review model, referred to in our previous report,11,58 not only optimised different methodologies for different stages of the question, but also offered multiple time points beyond which further plans might be framed conditionally.

Teams felt that the symbolic status of the protocol – analogous to a handshake to seal a deal – was often as important as the explicit technical information specified within it. Stakeholders, whether policy-makers or the public, often revealed little interest in the finalised protocols and the team and review commissioners tended to attach more value to the formative process rather than to the document as final artefact. Viewed in such a context, the scoping process is less a discernible phase of the review and more a pervasive issue that offers less flexibility as one nears the point of delivery. This also meant that the protocol was seen more as a ‘snapshot’ of the planned review and its intended direction at a point in this process (rather than a fixed blueprint for producing the defined end product). If this were a highly commercial contract between different parties wanting different goals, this would be a problem. However, in a context where the goals of the review teams and the policy customers are closely aligned (respectively, to produce and to receive a useful, independent and academically rigorous review of relevant evidence), such formal ‘contractual’ purposes of the protocol seem secondary. Scientifically, however, review protocols are also intended to minimise bias in review findings (through prespecifying and publishing the detailed aims and planned methods) and so the review protocols still fulfilled this important function.

Members from across the teams also pointed out how a similar, contingent need to keep plans open related to the involvement of stakeholders, whereby revisiting the consultative process might lead to further revision, refinement or reconceptualisation, ideally as a planned, transparent and documented activity. The teams reflected that this served to distinguish these commissioned review programmes from ‘classic’ research. They saw this contingent quality as an essential feature of policy support, more reflective of consultancy activities. Indeed, the team would also adopt the role of consultant when educating the end-user about synthesis process and products (consultative issue 4). This is seen when the review teams lay out the range of review products that they can provide and then advise the review commissioners on what might be most appropriate given the available time.

The role of the information specialist

Interestingly, within our case studies, less prominence was given to the contribution that different disciplines and skills provide within the scoping process. Information specialists on our teams could identify clear parallels between the skills they customarily employ when interacting, as internal knowledge brokers within a review team, and those that they utilised, as external knowledge brokers, as a ‘translator’ within discussions with the review commissioners. The different teams utilised diverse models of information specialist involvement (from nominated to the programme, through being dedicated to the programme and involvement in hybrid information specialist/reviewer roles). However, all three centres confirmed the need to ‘embed’ the information specialist within the scoping process and subsequent ‘negotiation’ to some degree or other. Distinct advantages were recognised from ensuring continuity in the information staff involved, particularly given that the types of topics explored through the NIHR HSDR contracts differed in their nature and requirements from ‘conventional’ systematic review topics. Indeed, an information specialist’s recognition of the ‘differentness’ of the review requirements could be as important as their technical knowledge in designing and conducting searches.

Although the need to involve information specialists in reviews, especially rapid reviews, has almost become a mantra in recent years, it is particularly interesting to isolate this specific contribution of their professional skills set beyond the technical knowledge of terminology, sources and databases. Information models associated with sense-making and the satisfying of information needs59 recognise the importance of relationship building, as evidenced in these review scoping processes. On the other hand, an information science perspective confirmed a recent observation that models of information need, typically spotlighted within information science, almost exclusively focus on an individual information need and not on the collective information transfer required by review teams.60

Abbreviated/accelerated review commissioning?

The review teams discussed the ‘upstream pipeline’ from ideas to commissioning briefs, as well as the ‘downstream’ pipeline from commissioning briefs to rapid review outputs. The constraints and practical, organisational and political considerations involved in managing and deciding priorities (consultative issue 1) within the review commissioning process contrast with the flexibility and agility that review teams can achieve once the topic is adopted in-house and comes within their control. Team members speculated if, ultimately, review commissioning might mirror the technical production of rapid reviews (i.e. in the form of abbreviated and accelerated versions of commissioning to match the concepts of abbreviated and accelerated review processes).

Team members expressed the view that the most satisfying and successful reviews in which they had participated typically involved a rapid path from ideas to briefs, such that the review outputs were contemporary, relevant and used in decision-making. In contrast, ‘testing the water’ with stakeholders as part of the scoping process sometimes revealed that they had not previously shaped the priority, or that time may have elapsed and priorities shifted. Team members felt that it could be useful to ‘check back the narrative’ (i.e. to ask if a previous high-priority question advanced for a review could be reaffirmed as a current high priority). On the other hand, the team recognised the need to reach a point at which a review question can be anchored and a protocol produced to draw a ‘line in the sand’, given that it would not be feasible to engage in a continual cycle of revisit, rescope and reprioritise. In some cases, the team may have invested considerable ‘sunk costs’ into the scoping process and may find it more feasible to make an existing priority work – by tailoring the scope – rather than pursue a de novo topic. In short, although review teams should understand that policy processes and priorities are often and necessarily dynamic, complex and political, review commissioners and users have to accept that a high-quality evidence synthesis can only be produced from a brief and question that, at some point in the review process, has to be fixed.

The ethics of commissioned reviews

The discussion concluded by reflecting on the ethics aspects of the scoping process. Team members felt a responsibility to produce reviews that genuinely meet the needs of health and social care services to avoid duplication and to ensure that public funding is effectively and efficiently spent. Reference was made to the Royal Society’s four ‘principles for good evidence synthesis for policy’: transparent, inclusive, accessible and rigorous.61 One of the centres had formally adopted these (with the addition of ‘appropriate’) as core values for use in its internal annual reviews. Reviewers have a responsibility to identify if a review might already exist or if a topic may be conceived too soon or articulated too late to be valuable. In such cases, the review commissioner might accept the availability and interpretation of evidence as it currently exists, commission primary research to meet the evidence need or set a time point when the value of a potential evidence review can be reconsidered. Note that none of the reviews that form the basis of our eight case studies required or obtained research ethics approval. This is accepted good research practice for systematic review projects (where the research ‘data’ is invariably already in the public domain).

Both a technical and a social process

Discussion within the teams agreed that the rapid review process extended beyond being a technical and exclusively information-driven process. Essentially, the relationship with commissioners of the reviews, whether the NIHR programme team or the end-users, is a social process that requires high-level social/research skills. The five themes (consultative issues 1–5) attest to the importance and prominence of this aspect of the rapid review commissioning process. This echoed the findings of Moore et al.,62 who identified 35 different skills, many of which were required at an experienced or very experienced level by knowledge brokers involved in a rapid review commissioning programme in New South Wales.

Where possible, all of the review team should be included as part of the ‘negotiation’ of scope. Staff described an ‘all hands on deck’ scenario at the initiation of projects when getting from the initial question, which may be difficult to deconstruct and translate, to a feasible review question. Team members needed to share and harness their collective experience so that they were able to recognise the specific requirements of this type of review work and have the confidence to propose and explore alternative methods and types of review. This could only happen through the social process of dialogue within the team about emerging understandings, misconceptions, confusions, anomalies and realisations, in relation to both the evidence and the end-user’s perspectives and needs.

Potentially, individual centre staff face a challenge in managing a personal portfolio that includes commissioned rapid review work that is potentially high impact, alongside more deliberative conventional academic projects and outputs for research excellence. Correspondingly, centre directors need to balance continuity, achieved through a small and focused team, with capacity, which may require access to a larger, more flexible and diffuse team. Continuity was needed to build up relationships internally within the team and with the commissioners, but also in terms of consolidating and extending skills in handling these complex and unclear types of topics. However, continuity could be delivered in diverse ways, from project direction and the assignment of dedicated staff across all projects through to standard involvement of the methodologists and identification of clear roles on each project. Capacity could also work out at an individual level as staff move from intensive involvement at particular stages of the project and then relinquish or re-engage at subsequent stages of the review pathway. These different team configurations to balance continuity and capacity influence the information exchange and topic-immersion aspects of conducting a review, and the trust and strength of relationships built with stakeholders.

Strengths and limitations

Strengths

This report builds on the detailed research experiences and insights from researchers working in three independent, university-based systematic review teams that were directly commissioned to conduct rapid and responsive evidence review projects to meet diverse policy and health service needs. The eight reviews from which we generated our shared reflections were on a diverse range of health and social care topics, and variously focused on effectiveness, cost-effectiveness, accessibility, implementation and patient and carer experiences. Some had well-specified initial review questions or topics and a clearly identified policy customer or end-user, others did not. For some reviewers and information specialists in all three teams, this work builds on experiences of developing and conducting rapid and responsive evidence syntheses for UK health policy organisations (such as the DHSC, NICE, Public Health England, Health Promotion Wales and the National Screening Committee) over several decades. This will inevitably have shaped how we work, and how we perceive and justify how we work, in this report.

Although this report has presented and reflected on the more recent experiences and lessons from eight review projects conducted between 2017 and 2020, for the York and Sheffield teams, this further builds on the experiences and a previous methodological report from similar commissioned roles between 2014 and 2017.11 However, in relation to review scoping processes, the previous methodological report11 highlighted three lessons: (1) the value of using a two-stage review approach (literature mapping/scoping, then review/synthesis), (2) the importance of disseminating review protocols as widely as possible and (3) the need ‘where feasible’ to include PPI in scoping reviews to ‘ensure their usefulness and relevance’. Although the first two of these emerged among our identified issues, it is conspicuous that PPI did not (see Limitations).

We have used an explicit process for describing, sharing and discussing our experiences of review scoping and question formulation (case studies described using a standardised approach). We have identified and structured our reflections using a simple framework based in previous research (of consultative, interactive and technical issues). In addition, we have situated our reflections and the proposed ‘lessons learned’ in relation to existing methods guidance on these early stages of systematic reviews.

We have endeavoured to identify common or recurring issues and principles and express them as useful lessons for others, but we have also sought to acknowledge complexity; in particular, the diversity of review user/decision-maker needs, the specificity or breadth of topics/uncertainties and the diverse and uncertain nature of available evidence. From the beginning, those involved in undertaking this work and writing this report have been open about and interested in the tension between scoping and systematic reviews as a technical and primarily information-driven process (i.e. scoping as ‘science’), and scoping as a collective social process of learning, negotiation and relationship building (i.e. scoping as ‘craft’) to create mutual understanding between reviewers, the intended review users and other stakeholders. Therefore, we hope it may also inform wider debates, such as those in science and technology studies, which seek to understand the interplay between formalised/standardised and informal practices in science,63,64 and between explicit, research-based evidence and the role of tacit, experience-based forms of knowledge in policy and practice.65

Limitations

Inevitably, our recollections and reflections may have suppressed or selectively reported negative experiences of stakeholder engagement or communication from research commissioners. Potential conflicts of interest relate to the need to preserve relations with stakeholders who might be important collaborators or commissioners in the future. Where this relates to the input of particular individuals, review teams took steps to anonymise those individuals and organisations with which we engaged. However, in general, such individuals are acknowledged in, and therefore identifiable from, the publication of the review.

More generally, this report describes and discusses the experiences and observations of researchers (review teams) working in a particular context (i.e. to produce commissioned, rapid reviews to respond to policy and health service users) in relation to the critical early stages of conducting systematic review (i.e. topic scoping and question formulation). Unlike the review scoping processes themselves, we have not involved the ultimate review users in this reflective exercise and they might have identified different issues, challenges and solutions. A prospective study of such processes should aim for greater cross-validation from different stakeholders and reflections from others external to the team. However, this would also require the scoping processes and choices to be more fully documented for others to be able to comment or reflect on them.

Although we prespecified some issues using a plausible framework, and used these to spot similarities and differences between the case studies, this was neither a formal thematic analysis nor a comparative case study. In addition, although our initial basic four-factor model (see Figure 1) usefully shaped our early discussions and consensus across the teams, and was not contradicted by what we found, it proved too basic and was effectively superseded by the 14 more specific scoping process considerations. This report cannot be considered ‘research evidence’ or judged against the usual standards of rigour and relevance. Nevertheless, given the transparency and reflexivity with which we have described and shared our experiences, both within this report and in the discussions that enabled us to produce it, we hope that it is an authentic, readable and useful account of the principles and key issues of scoping that have emerged within our experienced community of practice.

Some readers may express surprise at the apparent lack of PPI in the case studies presented here. All of the centres view PPI as a key contributor to the overall review process. However, we also acknowledge that opportunities to elicit patient and public experiences and preferences are perhaps more limited and, indeed, more challenging during scoping than at other stages in the process. Of the eight review projects that are the focus of this report, all but one included some PPI. However, of those that did make use of PPI, patients or the public were involved in review scoping and informing questions in only four case study reviews (case studies 4, 6, 7 and 8) (see Appendix 1, Table 13). Topics, as presented to the team, had, in many cases, already undergone input from PPI representation at earlier points of the process. Some topics (such as case study 5 in the adult social care field) were operationalised within patient/service areas where major national research prioritisation exercises involving patients and the public had recently concluded (such as James Lind Alliance Priority Setting Partnerships). Topic selection processes within the DHSC and the NIHR HSDR programme seek to include PPI or lay member input before being allocated to the review team. At the point of scoping, these topics were only ‘semi-malleable’, recognising the primary purpose of our centres in summarising and synthesising evidence to inform health and social care policy and service commissioning.

Rapid reviews, by definition, include rapid scoping processes, and identifying and recruiting appropriate patients or carers to any rapid project is challenging. At least one of the centres maintains a standing group of trained and pre-identified PPI representatives. However, ensuring that representation captures the values and perspectives of target populations, such as those with IDs or carers of those from ethnic minority populations, is challenging and may require specific targeted responses. Indeed, some reviews may not have narrowed their remit to particular types of patients or members of the public, making it difficult to prespecify what experiences, of conditions or of health and social care use, it would be appropriate to involve.

These possible explanations for little mention of PPI stakeholders in our case studies should not detract from the imperative of exploring and conducting PPI as soon as feasible within topic scoping. Indeed, where an immediate primary policy/service stakeholder has not been identified, or amid many and competing priorities, PPI representatives demonstrate the potential to provide critical insights about where the need for better research evidence is greatest.

We have not explored as much as we could the ultimate question of how and why the nature of the main subject of our reviews – essentially, different ways of organising and delivering health and social care services – is associated with specific challenges in formulating review questions and devising what review scope and approach to synthesis will best meet users’ needs. There is a growing literature on the challenges of evidence synthesis of complex interventions, including the issue of developing good review questions and review scopes.66–68 Such debates and emergent guidance encompass the recurring tensions in systematic review methods, such as how to capture and explain (vs. suppress) heterogeneity, how to capture the impact of contexts, ‘lumping’ versus ‘splitting’, whether to focus on the whole package/intervention or try to uncover associations with underlying components and mechanisms, and the critical role of theory and logic models in evidence synthesis. We recognise all of these features of complexity in the reviews we have conducted and the diverse, messy, imperfect real-world evidence of which we have sought to make sense. It would probably take a prospective research study, or at least a series of workshops among reviewers who do this kind of work, to tease out and refine the connections and lessons from our work to build on these complex interventions synthesis ideas.

Lessons learned

Our conclusions have been expressed as ‘lessons learned’, practical principles and suggestions, and as reminders to ourselves and guidance to others for improving scoping processes in the context of conducting rapid reviews for policy-makers and service commissioners and managers. Some are specific to how review teams can be more effective at question formulation and topic scoping, and others may be useful for review users and commissioners. Others simply (and humbly) describe challenges or common constraints in conducting reviews for policy and service delivery. We contend that if these lessons were more widely acknowledged – by evidence reviewers, prospective evidence users and others who may be affected by any decisions informed – these might foster more realistic expectations.

We present the lessons learned as a table (Table 12), mapping them to particular issues identified across our case studies. Particular lessons can be traced back to Chapter 4 and Table 11 to identify the case studies and reflections from which they are drawn.