Multicriteria decision analysis

In this chapter, the MCDA as a tool for option comparison is illustrated.

There are various HEE interventions that can be used to decrease household energy use for heating while helping to maintain or improve internal temperature control for thermal comfort and well-being. Examples include loft insulation, double glazing, cavity wall insulation, solid wall insulation, or a combination of such measures. The decision on which intervention or package of interventions is ‘optimal’ depends on the objectives of the policy and the criteria by which it is to be measured. Pertinent criteria might include, for example, benefits to health, NHS cost savings associated with avoided health burdens and energy savings.

Our aim was to develop and apply a method for comparing choices of HEE interventions in the UK.

Methods

The MCDA is a method that allows the comparative evaluation of interventions or policy options across multiple criteria. Of the several MCDA methods that can be used for this purpose [e.g. multiattribute/multiutility theory, Analytical Hierarchy Process, outranking methods and Simple Multiattribute Rating Techniques (SMART)], we used the SMART method implemented in the Annalisa software (Maldaba Ltd, London, UK), which we chose because it is easy to use in interactions with stakeholders and it is transparent.85,86

The SMART MCDA procedure has five steps, which are shown schematically in the diagram (Figure 24). The first step defines the set of feasible HEE interventions for comparative evaluation [shown in the diagram as a set of three interventions (I), I₁, I₂ and I₃].

FIGURE 24 Schematic diagram of the MCDA process. C, criteria; H, normalised impact; I, intervention; W, weight. (Figure not reproduced here; see the original article.)

The second step defines the criteria (C) that are used to compare them (C₁, C₂, C₃ and C₄ in Figure 24), which ideally should be independent. These criteria should also be defined in the same way, in the sense that either they are all ‘gains’ or they are all ‘losses’. In our analysis, they are defined them as gains.

The third step uses a model (or a set of models) to calculate the impacts that each intervention had on each criterion. The impacts model used in this MCDA is the same as that described in the preceding sections for modelling the impact of housing energy efficiency interventions on the indoor environment and health.

The different criteria of impact naturally have different units of measurement. For example, C₁ might be the NHS cost saving measured in pounds (£) and C₂ health gains might be measured in QALYs. To ensure that they can be more readily combined for analysis, all impacts are normalised to dimensionless quantities with a range of possible values between zero and unity. The method that we used for normalisation is detailed in Appendix 3. In Figure 24, H₁₁, H₂₁ and H₃₁ are the normalised impacts that interventions I₁, I₂ and I₃ had on criterion C₁; H₁₂, H₂₂ and H₂₃ are the normalised impacts that interventions I₁, I₂ and I₃ had on criterion C₂, and so on.

The fourth step assigns the relative importance (weight) to be attached to the criteria (W₁, W₂, W₃ and W₄ in the diagram, corresponding, respectively, to criteria C₁, C₂, C₃ and C₄). These weights should add up to unity. They can be interpreted as indicating the ratio of importance of one criterion against another. For example, if three criteria C₁, C₂ and C₃ have weights 0.1, 0.3 and 0.6 (which sum to 1) then criterion C₂ is considered three times more important than C₁ and half as important as C₃. C₃ is deemed to be six times more important than C₁. The relative weights are ideally elicited from policy-makers or stakeholders.

The fifth step combines the normalised impacts with the relative weights to produce a performance score for each intervention. If the impacts are gains, then the intervention with the highest performance score is the ‘optimal’ intervention across all the selected criteria; if the impacts are losses, then the intervention with the lowest performance score is the optimal intervention.

Different sets of weights and normalisation procedures can be explored to determine the sensitivity of the optimal intervention determined from them.

Impacts model

The impacts model provides estimates of the health impact, as well as costs of changes in health, that arises from installing energy efficient interventions in domestic dwellings. Informed by discussion with stakeholders, we selected the following criteria for comparing housing interventions:

1. mortality gain (QALYs)
2. morbidity gain (QALYs)
3. NHS costs (£)
4. space heating (kWh)
5. impact on inequalities, measured in relation to:
   1. socioeconomic deprivation (quartile of the Index of Multiple Deprivation) – the difference in the change in health (Δ health) in the most deprived compared with the least deprived: Δ health (most deprived) – Δ health (least deprived)
   2. age (those aged < 65 years and those aged ≥ 65 years) – the difference in the change in health in those ≥ 65 years compared with those < 65 years: Δ health (≥ 65 years) – Δ health (< 65 years)
   3. FP (yes/no) – the difference in the change in health for those in FP compared with those not in FP (≈ FP): Δ health (FP) – Δ health (≈FP).

A description of the building physics (exposure) model used to estimates of intervention-related changes to the indoor environment, as well as the method for estimating the health impacts arising from them, is summarised in Hamilton et al.32 The changes to indoor exposures related to changes to the thermal, ventilation and heating performance of the dwelling and related energy expenditure. These exposures were temperature (cold and overheating), particle air pollution (PM_2.5), second-hand (environmental) tobacco smoke, radon and moisture (as a precursor of mould growth). The impacts that these exposures had on health were estimated (with time lags when relevant) for outcomes whose current disease rates were determined from published sources. These outcomes included mortality from cerebrovascular and cardiopulmonary disease, lung cancer and winter mortality, and morbidity for the same causes, with the addition of common mental disorder (related to winter cold) and asthma in children. These impacts were quantified using life table methods applied to individuals within the EHS data, combined with more direct estimates of changes in disease prevalence, to derive estimates of change in QALYs.

The impact that these had on NHS health-care services was assessed by deriving estimates of the costs of associated changes in NHS contacts, following the method described by Stafford.87 The categories that the NHS uses as a framework for estimating NHS expenditure across health-care conditions (NHS programme budgeting categories, and their published aggregate primary care trust expenditure figures) were based on data for the period 2012–13 (latest available data). These were used to derive total aggregate costs for selected health outcomes for England for primary care, secondary care, emergency care and social and community care. Estimates of national prevalence in 2012–13 for the selected disease outcomes were then used to derive the total unit NHS contact costs. Unit costs for each health outcome were estimated by dividing the total expenditure for each disease outcome for England from these budgetary categories by the total number of people on each disease outcome register for England. The health impacts (owing to changes in exposures), in terms of QALYs, estimated by the impact model were then converted into annual NHS contacts, and the NHS treatment unit costs were applied to hospital contacts for the selected diseases.

Using these criteria, we compared the following specific energy efficiency measures: loft, cavity wall and solid wall insulation, double-glazing replacement, boiler/heating system upgrade and draft stripping. These were assessed with and without installation of additional ventilation measures (i.e. installation of trickle vents and/or extraction fans intended to provide additional (controllable) ventilation that partly offsets the reduced air exchange created by other measures). The assessment with and without compensatory ventilation was done as energy efficiency measures that increase airtightness partly protect against the ingress of pollutants of outdoor origin but also tend to increase exposure to pollutants of indoor origin (including some particles, STS and radon) and may adversely affect mould growth.88 Boiler replacements, which are not considered to alter airtightness, were modelled without additional ventilation.

Results

The results of the health impacts, NHS cost savings, energy savings and the difference in health impacts between the three measures of inequality (socioeconomic status, FP and older age) are given in Table 9 for interventions with additional ventilation and in Table 10 for interventions without additional ventilation. The corresponding results of the MCDA for the scenarios with and without compensatory ventilation are shown in Figures 25 and 26, respectively: the summary scores are the horizontal bars in the top panel of each set of results, with the ‘optimal’ intervention shown by a darker orange bar.

FIGURE 25 The MCDA scores results. (a) Equal weighting; and (b) non-equal weighting. (Figure not reproduced here; see the original article.)

FIGURE 26 The MCDA scores results (without ventilation). (a) Equal weighting; and (b) non-equal weighting (but different from Figure 25). (Figure not reproduced here; see the original article.)

For scenarios with additional ventilation and using equal weighting for each criterion (as shown in Figure 25b), the optimum intervention was double glazing. However, when greater weight was given to NHS cost savings, energy savings and the reduction of health inequalities (in terms of socioeconomic group), boiler replacement was identified as the optimal intervention – ahead of intervention based on all measures combined.

In the absence of compensatory ventilation, the MCDA analysis identified ‘all possible interventions’ as being optimal when based on criteria with equal weightings, but the optimal choice was, again, replacement boilers when the weights were altered to give more emphasis to health impacts and energy savings.

The absence of additional ventilation to compensate for any reduction in air exchange resulting from the energy efficiency measures leads to an increase in the indoor concentration of pollutants of indoor origin with a net negative effect for health. This can be seen by the generally negative figures for mortality impacts in Table 10 and the larger NHS costs savings associated with interventions when compensatory ventilation is applied (see Table 9). With compensatory ventilation, the size of health impacts is higher, with a positive (i.e. beneficial) change in terms of QALYs gained from improved mortality and morbidity.

As expected, the energy saving is considerably less for interventions with added ventilation; however, there is still an appreciable saving when compared with no intervention (with the smallest gain being for draught proofing, followed by loft insulation, and the largest gain for ‘all possible interventions’). With regard to measures of inequality, the intervention (except for draught-proofing) had a slightly lower positive health impact for people living in a more deprived area but had a greater impact for all measures for those aged ≥ 65 years than for those aged < 65 years and, similarly, for the FP than for non-FP households (with the exception of double glazing).

Workshop

We elicited inputs to the MCDA work through three workshops: two (one small and one larger) held at the London School of Hygiene & Tropical Medicine (LSHTM) and one hosted by Age UK. In each case, we invited a range of stakeholders from non-governmental organisations, local and national government departments, academia and others. In the first workshops (10 participants: four academic, four from charities and two housing organisations), we discussed plans for applying the MCDA method and asked for inputs on the policy comparisons and the proposed criteria for their assessments, which were further elaborated in a more general stakeholder workshop (26 participants: 11 academic, five from charities, two from housing organisations, four from national or local government, five from Public Health England/National Institute for Health and Care Excellence and one from an international organisation). In the last workshop (formed of 24 participants: 11 academic, three from charities, three from housing organisations/consultancies, three from national or local government and three from Public Health England), we showed the use of the MCDA tool in a ‘live’ demonstration and sought inputs on the interpretation of its results. We initially showed results using analyses performed ‘offline’ ahead of the workshop using equal relative weightings for the selected criteria but then used it live to explore the results interactively. This showed how the results (in terms of the overall intervention scores and, hence, the choice of which intervention performs best) are sensitive to the values of the relative weights and the normalised impacts.

The stakeholders were broadly in agreement with the choice of intervention options to compare, but the choice of the criteria and how they altered the option comparisons generated considerable debate. The main issue of discussion was the extent to which the overall performance scores changed with different choice of criteria weightings and hence the potential variation that could arise given the different perspectives/priorities of different stakeholders. Our criteria reflected several important perspectives, including public health (reduction in mortality and morbidity), NHS (increasing cost savings associated with diseases averted), climate change mitigation (reduction in energy use) and inequality (reduction in health inequality between FP vs. non-FP, least deprived vs. most deprived and pensioners vs. non-pensioners). However, we discussed that an alternative set or a subset of criteria could have been chosen, would have been equally valid and would have produced different performance scores for the interventions. There was discussion that the value of the MCDA approach is, therefore, not so much the single headline ‘answer’ that any one analysis generates but that it may yield understanding (if used as an interactive tool) about the importance of different factors in reaching particular results.

We showed online how the performance of the interventions in terms of their ranking depend on the weights assigned to the criteria and the values of the normalised impacts. The participants appreciated the transparency that the MCDA tool brings, particularly in relation to the influence of the relative weights attached to different criteria (which reflects the clear meaning attached to the changes in the weights). However, they were more concerned with the sensitivity of the results to the estimates of normalised impacts, the calculations of which are model-dependent and more opaque. This emphasised the potential value to decision-makers not just of the results but of understanding how different exposures and outcomes are affected by different interventions.

In summary, participants engaged actively with the MCDA tool because the evidence of impacts, the importance attached to the specific criteria (relative weights) and the overall performance scores were clearly displayed and the changes in performance scores could be tested against changes to the criteria weightings. This provided insight into how the results of the optimal choice depend on the priority given to different criteria. There was some desire also to have further detail on how interventions result in the specific exposure changes and outcomes.

Discussion

The results of these MCDAs provide indication of how specific HEE intervention measures, or all interventions combined, compare with each other when assessed against agreed criteria and criteria weightings.

They suggest that the optimal intervention depends not only on the criteria weightings but also on the presence or absence of compensatory ventilation to offset part of the reduced air exchange that would otherwise accompany most interventions. Without compensatory ventilation, many health impacts are negative and the gains for morbidity and NHS cost savings are lower than with compensatory intervention (an effect driven by net changes to indoor air quality).

When there is compensatory ventilation, and the particular set of equally weighted criteria relating to health, environment and health inequalities are used the results suggest double glazing to be the optimal intervention, while without compensatory ventilation, ‘all possible interventions’ combined appeared optimal. However, the results are sensitive to changes to the priority (weights) given to particular criteria and, potentially, to the set of criteria used.

However, the MCDA should not be viewed as a process providing a single answer but rather as an interactive engagement tool to help decision processes. Its use helps the transparency of decision-making in that it shows how priorities depend on the assessment criteria and their weightings and on evidence of impacts. It allows stakeholders to be more engaged with the process so that they have greater sense of ownership of the decision proposed. Its use is encouraged by the UK Government.89

Information needs for improved decision support

The analyses suggest areas in which better evidence would be helpful to improved decisions on HEE interventions. One of those is evidence on the consequences of interventions for air exchange. For example, as Figures 25 and 26 show, in circumstances where there is compensatory ventilation, the six insulation measures [cavity wall insulation, draught proofing, (double) glazing, loft insulation, solid wall insulation] were broadly similar with regard to the impact that they had on mortality, morbidity and NHS cost savings and were appreciably better than that of boiler replacement for these criteria, although the impact had on inequalities was generally worse. However, the situation is almost reversed when there is no compensatory ventilation, with boiler replacement having a mostly greater impact with respect to mortality, morbidity and NHS cost savings.

Therefore, the accuracy and validity of evidence on ventilation changes appear central to the assessment of the relative position of intervention options, and indeed to the overall benefit of intervention measures. However, there remains uncertainty over the quantification of ventilation changes associated with each form of intervention. Hence, obtaining improved empirical evidence on the consequences of interventions for ventilation, and how those consequences may vary with occupant behaviour, is important and merits further research, especially as adverse effects on ventilation may be detrimental to health.

A second area relates to criteria weightings. The outputs of the MCDA are sensitive to the weights given to each criterion (and also to the criteria chosen), as illustrated by altered patterns when alternative weights were used. For example, in analyses that assume compensatory ventilation, the optimal choice was double glazing when equal weight was assigned to each criterion, but replacement boilers were favoured when there was heavier weighting on energy savings (e.g. as might be appropriate for stakeholders with greater concern with carbon impact or FP). As the workshop discussions emphasised, the question of perspective (i.e. whose views) is potentially very important. But there are also complexities here, because the benefits and costs of interventions potentially accrue for different sections of society and government, and so the framing of the options is also important. If the MCDA is used to support ‘real-life’ decision-making, it would be valuable to develop through participatory processes, greater shared understanding of the ultimate objectives of HEE programmes (e.g. in relation to such outcomes such as health, FP, climate change and inequalities) and to elicit criteria weights appropriate for balancing those objectives. In the analyses presented in this report, we focused on the health, environment and inequality impacts from a broadly societal perspective.

A third area needing improved evidence is the transparency and empirical basis of evidence on the normalised impacts (on health and other criteria). This was an issue of some discussion among workshop participants because many of the relevant data were generated by models. Discussants generally wanted to understand more about how model results were derived and the underlying assumptions so that they could judge the reasonableness of suggested impacts. To some extent this can be addressed by showing intermediate model outputs and underlying empirical relationships, but there is a wider issue of needing more empirical evidence on the relationship between housing characteristics and dwelling performance, to provide both further scientific validation and greater transparency for non-expert stakeholders.

A further issue is the characterisation and presentation of uncertainty. For this work, we used only simple sensitivity analyses to explore the effect of variation in preferences. Methods to represent uncertainty in model and other outputs is desirable but also complex because of the many factors (both structural and parametric) that potentially contribute to the uncertainties in these MCDAs. Conventional Monte Carlo methods are difficult to implement with the fairly complex sets of model-based outputs needed for assessing HEE interventions against multiple criteria. Again, this is an area that would benefit from further research.

In conclusion, the MCDA provides an analytical framework of potential value as a tool to support decisions on HEE interventions, especially if used in iterative fashion to explore options with relevant stakeholder groups. The robustness of those comparisons would be improved by strengthened empirical evidence, specifically on the effect of interventions on ventilation changes, but also by participatory engagements with stakeholders on the framing of the objectives of intervention programmes and the criteria by which they should be judged.