Reports included in the economic evaluation synthesis
Only one economic evaluation report was eligible for inclusion in the review,132 which reported on the TXT-Auto intervention. Targeting sexual health and substance use outcomes, TXT-Auto was an open-ended intervention with core content organised by user assessment. Characteristics of the included study are reported in Table 10.
| Item | Description |
|---|---|
| Programme: TXT-Auto (Reback et al.115) | |
| Research question | What were the costs to the health-care sector for benefits achieved through the application of theory-based, interactive text messages, with and without real-time interactive peer text conversations, among non-treatment-seeking, methamphetamine-using MSM? |
| Intervention | TXT-Auto: for 8 weeks, participants received five automatically transmitted unidirectional text messages a day and a once-weekly assessment on methamphetamine use and HIV sexual behaviours in the previous 7 days |
| Comparator(s) and whether or not this represents standard practice in the UK | TXT-PHE: peer health educators engaged in bidirectional interactive text messaging conversations with participants; participants also received five automatically transmitted unidirectional text messages a day, as well as a once-weekly assessment on methamphetamine use and HIV sexual behaviours in the previous 7 days |
| AO comparator: participants received only the once-weekly assessment on methamphetamine use and HIV sexual behaviours in the previous 7 days | |
| Note that neither of the above is standard practice in the UK | |
| Base-case population characteristics and analysed subgroups | MSM aged 18–65 years reporting methamphetamine use and CAI with a non-primary partner in the previous 3 months, who had access to a mobile phone with unlimited texting service, and who were not enrolled or seeking enrolment in a programme for methamphetamine use, and not considered to have a serious psychiatric condition. No subgroups were analysed in the economic evaluation |
| Form of economic evaluation | Cost-effectiveness analysis |
| If cost–utility analysis, were QALYs reported | NA |
| Primary outcome measure(s) for the economic evaluation | Number of self-reported days of methamphetamine use in the previous 30 daysNumber of episodes of sex while on methamphetamines in the previous 30 daysNumber of self-reported episodes of CAI with any partnerRisk behaviours were reported at 8 weeks and at 3, 6 and 9 months following baseline. Interpolation occurred for the months not specifically reported to obtain an estimate of the total number of risks acts over the 9 months and converted this to an average number of risk behaviours per month from baseline to month 9. This monthly average was subtracted from the figure reported in the baseline for risk behaviours over the prior month to measure incremental improvement in outcomes |
| Methods used to value health states and other benefits | Health outcomes were not valued |
| Methods and sources of information used to estimate resource use | Costs of delivery were collected retrospectively using a modified UNAIDS template from a health-care system perspective. Monthly costs of the ongoing programme were calculated based on programme expenditures from February 2014 to January 2015. Costs for facilities, other office and medical costs, and the programme director were obtained from administrative records. The costs for the director included salary, benefits and retirement contributions. In the base-case analysis, these costs are evenly split among the three study arms. A sensitivity analysis did not allocate facility or director costs to the comparator condition. Costs were price-adjusted back to year 1 of the study (2014), using the medical care component of the consumer price index. The cost of the text messaging platform was allocated based on the proportions of messages received per arm |
| Did the study include start-up provider costs? | No |
| Did the study include ongoing provider costs? | Yes |
| Did the study include provider costs per contact? | Yes. These were calculated by first calculating the monthly average costs for each arm by dividing its total monthly cost by the average number of participants per month. TXT-Auto averaged 10.2 participants per month. Monthly cost per participant was multiplied by 2 to account for the 2-month enrolment period. In the base-case analysis, the average intervention cost per participant in the TXT-PHE arm was US$3478 |
| Did the study include costs to patients? | Yes, but these were reported as minimal |
| Currency and price year | Costs were expressed in US dollars and price-adjusted back to year 1 of the study (2014), using the medical care component of the consumer price index |
| Details of model used and key structural issues and assumptions | No decision modelling was performed |
| Justification for model used | NA |
| Base-case time horizon | 9 months |
| Base-case discount rates for costs and benefits | NA |
| Statistical test(s) and CI(s) for stochastic data | Not reported |
| Sensitivity analyses | One sensitivity analysis did not allocate facility or director costs to the comparator condition. Another tested the effect of using reported risk behaviours at 9 months vs. using average monthly risk behaviours. This raised the cost of the two texting arms and reduced the cost of the comparator. This moderated some of the outcome effects, but showed the same pattern as the base-case analysis. Other one-way sensitivity analyses were undertaken, but they appear to have been based on average costs and effects, rather than ICERs |
| Base-case ICER | Sex on methamphetamine: US$426 per episode avertedCAI: US$37 per episode averted |
| ICERs for specified subgroups | Not reported |
| Author conclusions | Both intervention arms outperformed the comparator in reducing HIV risk behaviours, but the TXT-Auto arm dominated the TXT-PHE arm in achieving greater reductions in days of methamphetamine use and episodes of CAI at lower cost. Sensitivity analyses showed that results were robust to a number of changes in assumptions |
Quality of study
We assessed the quality of the included economic evaluation on 10 main items, each of which contained between two and six sub-items for a total of 31 sub-items (see Appendix 16). Reviewer agreement was high, with the two reviewers agreeing on 25 of the 31 sub-items (81%) and all of the main items. The study was judged as meeting five main quality items: well-defined question in answerable form, comprehensive description of competing alternatives, the effectiveness of the programme was assessed, costs and consequences were valued credibly and an incremental analysis of costs and consequences of alternatives was performed. The quality criterion of costs and consequences adjusted for differential timing was judged to be inapplicable to the study. The study was judged as not meeting four of the main quality items: all important and relevant costs and consequences for each alternative identified, costs and consequences measured accurately in appropriate physical units, allowance made for uncertainty in estimates of costs and consequences and discussion of results includes all issues of concern to users.
Summary of this study
The included study (see Table 10) assessed the cost-effectiveness of two text-based interventions to reduce the frequency of methamphetamine use and HIV sexual risk behaviours among MSM: (1) an interactive messaging service delivered by peer health educators with an automated text messaging service (TXT-PHE); and (2) an automatic text-based messaging service without peer interaction (TXT-Auto). It is the latter intervention that met the inclusion criteria for this systematic review. Both interventions were compared with a weekly assessment of risk behaviours that did not include a text-based messaging element (AO).
The evaluation involved a RCT conducted in the USA, which enrolled participants between March 2014 and January 2016.
The economic evaluation was a cost-effectiveness analysis with outcomes expressed in terms of numbers of episodes of CAI with any partner (which was not reported in the previous outcome evaluations), days of methamphetamine use (for which there was not a significant difference between arms, as reported in the previous outcome evaluation) and episodes of sex while using methamphetamines (for which, as described above, the previous outcome evaluation reported a difference favouring TXT-Auto, but only at p < 0.1 significance), all within the previous 30 days. The analysis was from a health services cost perspective over a 9-month time horizon (the duration of follow-up of the RCT), with costs expressed in US dollars at 2014 prices. Although the analysis was based on the results of the RCT, the costs were stated to have been collected retrospectively. They included the costs of providing facilities/offices, medical resources and general administration of the texting services. The economic evaluation did not include any decision modelling, which is a quantitative method of synthesising information from different sources and extrapolating results into the longer term.
In the base-case analysis, TXT-Auto was reported to have cost an additional US$426 per reduction in episodes of sex while using methamphetamines, compared with AO, although the authors acknowledged that that there was not a statistically significant difference between the two conditions. The economic evaluation also reported that the cost to achieve a reduction in days of methamphetamine use was actually higher for TXT-Auto group than for the AO control group. The economic evaluation reported that TXT-Auto was more cost-effective than the AO control in reducing episodes of CAI with any partner, reporting that the cost per difference in CAI with any partner between the two conditions was US$37 per episode averted, but did not provide an estimate of the significance uncertainty of the difference between the conditions in CAI with any partner. Several one-way sensitivity analyses were conducted around intervention costs and changes in risk behaviour. However, the impact of these changes on the base-case ICERs were not directly reported; they relate only to the average treatment costs and effects. A probabilistic sensitivity analysis was not undertaken.
Critical appraisal of this study
A strength of the analysis is that it was based on results from a reasonably well-conducted RCT. However, it contains a number of problematic issues that limit its usefulness in terms of a UK decision-making context. First, the control arm was not judged to be reflective of routine UK practice, meaning that it is difficult to interpret the ICERs, even if the study had been judged as technically strong. Second, no attempt was made to extrapolate the trial results beyond the 9-month follow-up period using modelling techniques. This can be particularly important in the context of infectious diseases because there is a possibility of a prevention benefit, as well as a benefit to participants directly. A related issue is that the results were reported as incremental costs per reduced episode of CAI and reduced episodes of sex while on methamphetamines in the previous 30 days. Even though the base-case ICERs are low in monetary terms (e.g. US$37 per episode of CAI averted), it would have been preferable to have attempted to express health benefits in terms of a generic outcome measure such as a quality-adjusted life-years, that is to have estimated and valued the potential longer-term health benefits of reduced risk behaviours, and to have expressed them using a metric for which willingness-to-pay thresholds have been estimated. The RCT did not collect information on HIV or any type of infection incidence. Thus, even though TXT-Auto was associated with a lower incidence of CAI than AO, the impact of this benefit on health is unknown. Most of these effects were significant only at p < 0.1, and so were not considered significant effects in our synthesis of outcomes. Thus, if a probabilistic sensitivity analysis had been conducted, it could not have led to complete certainty in terms of decision-making at any willingness-to-pay level. An assessment of the uncertainty around the base-case cost-effectiveness analysis using a deterministic approach is an important component of any economic evaluation. The deterministic sensitivity analysis reported in the economic evaluation is of poor quality in that it was limited to a few cost and effect parameters; it contained only one-way analyses; and the direct impact of alternative parameter values was not reported for the actual base-case ICERs, but only for the average treatment costs and effects. It also appears as though only a proportion of costs attributable to each treatment option have been included in the ICER calculations, although it is unclear why. If the total cost of the treatment arms had been included, as is more normal practice, the ICER values would have been higher than those reported. In summary, as there is weak evidence to suggest that TXT-Auto reduced the number of episodes of CAI and other outcomes, it is difficult to know whether or not it is a cost-effective use of resources based on the results from this analysis. At best, however, the results suggest that its cost-effectiveness is uncertain.