Parts of this chapter have been reproduced or adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes additions and minor changes to the original text.
Reports included in the outcome evaluation synthesis
We included 14 outcome evaluations of 13 interventions described across 16 reports, published between 2006 and 2020.99–102,109,110,113–117,122–125,127 Of these, 14 reports were eligible for inclusion in the outcome evaluation synthesis,99–102,110,113,115–117,122–125,127 and two were abstracts109,114 presenting findings identical to those presented in included papers.110,115 All included studies were randomised trials allocating individuals. Of the 14 included trials and included arms within those trials, 13 compared interventions with no-treatment or attention-only controls100–102,110,113,115–117,122–125,127 (which receive the same degree of engagement as intervention participants but no other intervention contents) and two compared active interventions.99,102 Comparing two intervention arms and one control arm, the report by Davidovich et al.102 fell into both categories. Thirteen eligible studies reported sexual health outcomes,99–102,110,113,115–117,122–125,127 two reported substance use outcomes alongside sexual health outcomes115,125 and none reported mental health outcomes.
See Table 2 for a list of all reports included in the outcome evaluation synthesis, Appendix 10 for characteristics of included outcome evaluations and Table 4 for the health outcomes examined in this review that were targeted by each intervention.
Risk of bias and quality of studies
The risk-of-bias assessment covered a range of domains, as prescribed by the Cochrane Collaboration risk-of-bias tool.63 Risk-of-bias judgments are presented for active versus control trials and, subsequently, for active versus active trials.
Appendix 10 shows the results of the risk-of-bias assessment for each outcome evaluation study, and Figure 8 shows a risk-of-bias graph by domain.
FIGURE 8 Possible sources of bias, by intervention. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
Synthesis of studies comparing active intervention with control arm
The narrative synthesis included 13 trials100–102,110,113,115–117,122–125,127 comparing interventions with no-treatment or attention-only controls. Two studies113,125 reported outcomes for HIV infections, three studies110,113,125 reported outcomes for STIs and 12 studies100–102,113,115–117,122–125,127 included outcomes for sexual risk behaviour. Two studies115,125 reported outcomes for drug use. No included outcome evaluations reported outcomes for alcohol use, depression or anxiety. Only one study, Cheng et al.,124 reported outcomes by equity-relevant characteristics. See Tables 5–8 for overviews of measures used in included outcome evaluations.
Risk of bias of active intervention versus control arm trials
Sequence generation
Of the 13 trials included, eight100,101,113,116,122–125 had adequate sequence generation and five102,110,115,117,127 did not state how sequence generation was undertaken. In most cases, sequence generation was undertaken using computerised methods, given that interventions were delivered electronically. However, one study included rolling of dice corresponding to a random number table.125 Several studies reported using randomisation or minimisation stratified by factors such as race or age.100,113,122
Allocation concealment
Of the 13 trials included, only one did not state how allocation concealment was undertaken.115 The remaining 12 trials provided information on how allocation was concealed; in all cases it was concealed by automatic online randomisation.99–102,110,113,116,117,122–125,127
Blinding of participants and personnel
Three trials reported blinding of participants and study personnel in respect of treatment allocation.113,122,125 One trial did not report information with which to make a judgement.127 Nine trials did not include blinding of participants, personnel, or both.100–102,110,115–117,123,124 In three trials, neither participants nor personnel were blinded.110,115,117 In five trials personnel were not blinded.100–102,110,123 In two trials, participants were not blinded.116,124
Blinding of outcome assessors
Seven trials reported blinding of outcome assessors.100,102,110,113,122,123,125 In two trials,110,113 this was because the key outcomes were biological in nature. In the remaining five trials,100,102,122,123,125 outcomes were self-reported but participants were blinded. Four trials stated that outcome assessors were not blinded.115–117,124 This was because outcomes were self-reported and participants were not blinded in these trials. Two trials did not provide enough information to judge blinding of outcome assessors.101,127
Complete outcome data
Nine trials110,113,115,116,122–125,127 presented complete outcome data, whereas four trials100–102,117 did not. Complete outcome data were defined as balanced retention in study arms with attrition of < 30%. Of the four trials without complete outcome data, all had high levels of attrition; two trials102,117 also had notably imbalanced attrition between arms.
No selective outcome reporting
Seven trials100,101,110,113,117,123,127 had evidence of selective outcome reporting, whereas six trials102,115,116,122,124,125 did not. Of the trials with evidence of selective outcome reporting, one117 did not present analysis for outcomes because of sample size; three100,101,110 did not present findings in sufficient detail to estimate effectiveness; and a further three113,123,127 did not report outcomes that were in the protocol for each trial.
Accounted for clustering
None of the trials was cluster randomised; thus, there was no need to adjust for clustering.
Reduction of other sources of bias
Three trials did not reduce other sources of bias, in every case arising from unclear or inappropriate analysis methods.101,110,117 Ten trials100,102,113,115,116,122–125,127 reduced other sources of bias by using a combination of geographically diverse recruitment and multiple methods of sample recruitment. Specifically, Bauermeister, et al.,123 Carpenter et al.,100 Cheng et al.,124 Chiou et al.,125 Mustanski et al.,113 Hirshfield et al.127 and Rosser et al.116 each used geographically diverse samples with respect to the country where the trial was undertaken. Davidovich et al.,102 Reback et al.,115 Rosser et al.,116 Hirshfield et al.127 and Mustanski et al.113,122 each used multiple methods of recruitment.
Effects on human immunodeficiency virus infections
Two studies presented estimates for HIV infection: Chiou et al.,125 which presented short-term (< 3 months) estimates, and Mustanski et al.,113 which presented mid-term (3 months–1 year) estimates (Table 5). Across studies, there was no consistent evidence of interventions reducing new HIV infections, although sparse events and short time scales may have precluded clearer evidence of an effect.
| Intervention | Measure | Notes |
|---|---|---|
| Keep it Up!113 | Incident HIV infections (over 12 months) | Self-reported test result |
| Safe Behaviour and Screening125 | Incident HIV infections (6-month follow-up, post intervention) | Self-reported test result |
Study-level results
Chiou et al.125 reported three new HIV infections (n = 130) in the group that received Safe Behaviour and Screening, an open-ended intervention with general content, compared with two infections (n = 135) in the control group. At 6 months post randomisation (i.e. at post-intervention follow-up), this yielded an incidence rate ratio of 1.56 [95% confidence interval (CI) 0.26 to 9.56]. Evaluating the Keep It Up! time-limited, interactive online modular intervention, incident HIV diagnoses at 12 months post randomisation (6 months post intervention) in Mustanski et al.113 were not different in the intervention arm (nine diagnoses over 384 person-years), compared with the control arm (eight diagnoses over 410 person-years).113
Meta-analysis
Effect sizes for HIV infection are presented in Figure 9 and drew from two studies, each contributing one effect size.113,125 We undertook an overall analysis, rather than analyses by time, as only one effect size from each follow-up category was included. Included interventions did not have an overall impact on HIV infections, with an increase in incidence of HIV infections equivalent to 0.12 standard deviations, but with a wide estimated CI that included the point of no effect (95% CI –0.34 to 0.59). Heterogeneity was not meaningfully present in this meta-analysis (I2 = 0%). Certainty in the assessment of the evidence ranged from very low to low because of the risk of bias (selective outcome reporting) and the imprecision of effect estimates (see Grading of Recommendations Assessment, Development and Evaluation analysis for detailed reporting on the level of certainty for outcomes reported).
FIGURE 9 Effect sizes and meta-analysis for HIV infections in included studies. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
Effects on sexually transmitted infections
Three studies presented estimates for STIs: Chiou et al.,125 Milam et al.110 and Mustanski et al.113 (Table 6). Of these, Chiou et al.125 and Milam et al.110 presented short-term results, whereas Mustanski et al.113 presented mid-term results. There was no evidence of short-term impacts on incident STIs, whereas there was some evidence, albeit from one study, of mid-term impacts on incident STIs.
| Intervention | Measure | Notes |
|---|---|---|
| Internet-based safer sex intervention110 | Incident STIs (over 12 months) | Clinical testing (syphilis, chlamydia and gonorrhoea) |
| Keep it Up!113 | Urethral chlamydia (over 12 months) | Clinical testing |
| Urethral gonorrhoea (over 12 months) | Clinical testing | |
| Rectal chlamydia (over 12 months) | Clinical testing | |
| Rectal gonorrhoea (over 12 months) | Clinical testing | |
| Any STI (over 12 months) | Positive clinical test for urethral or rectal chlamydia or gonorrhoea | |
| Safe Behaviour and Screening125 | Incident syphilis (6-month follow-up, post intervention) | Self-reported test result |
Short-term results
At 6 months post randomisation (i.e. at post intervention), Chiou et al.125 reported four incident syphilis infections in the group (n = 130) receiving the Safe Behaviour and Screening open-ended general content intervention, compared with three incident syphilis infections in the control group (n = 135). This translated to an incidence rate ratio of 1.39 (95% CI 0.31 to 6.37). Evaluating an open-ended intervention with content organised by assessment, Milam et al.110 reported rates of any incident bacterial STIs (syphilis, gonorrhoea or chlamydia) over 12 months, which was the intervention period. In the intervention group (n = 90), 27 participants reported incident STIs, whereas in the control group (n = 89), 22 participants reported incident STIs. These proportions (30% vs. 25%) were not statistically significantly different (p = 0.50), nor was the distribution of visits with new STIs per subject different between arms (p = 0.57). A logistic regression that controlled for STI diagnosis, meth use and use of antiretroviral therapies as baseline covariates yielded an odds ratio (OR) of 1.35 (95% CI 0.68 to 2.70).
Mid-term results
Assessing the online modular intervention Keep it Up!, Mustanski et al.113 reported results for several STIs, both individually and as a composite outcome, at 12 months post randomisation (6 months post intervention). Analyses included between 356 and 359 intervention participants (359 for urethral STIs, 356 for rectal STIs) and 374 control participants. Findings were principally reported as risk ratios (RRs), and suggested a statistically significant 40% difference in risk of any STI diagnosis (RR 0.60, 95% CI 0.38 to 0.95). Findings for individual STIs were not significant: urethral chlamydia (RR 0.60, 95% CI 0.13 to 2.34), urethral gonorrhoea (RR 0.35, 95% CI 0.01 to 4.33), rectal chlamydia (RR 0.61, 95% CI 0.34 to 1.06) or rectal gonorrhoea (RR 0.91, 95% CI 0.40 to 2.05). Another analysis of the outcome drew on a matched-pair analysis and estimated a within-subject reduction in risk of 68% for any STIs (95% CI 0.40 to 0.83).
Meta-analysis
Effect sizes for STIs are presented in Figure 10 for outcomes at < 3 months post intervention and in Figure 11 for outcomes between 3 months and 1 year post intervention. In both plots, negative effect sizes represent benefits. A meta-analysis of effect sizes with follow-up of < 3 months included two effect sizes from two studies110,125 and also suggested a non-significant increase in the number of STIs in the intervention group, compared with the control group (d = 0.17, 95% CI –0.18 to 0.52), with heterogeneity not meaningfully present in this meta-analysis (I2 = 0%).110,125 The certainty of evidence was very low as a result of risk of bias (details of randomisation, selective outcome reporting) and imprecision of effect estimates. The overall analysis across short- and medium-term follow-ups and intervention types drew on three studies110,113,125 contributing seven effect sizes, and suggested a small and non-significant increase in STIs in the intervention group, compared with the control group (d = 0.07, 95% CI –0.79 to 0.94) and low heterogeneity (I2 = 16%). We did not meta-analyse effect sizes with follow-ups of 3 months to 1 year as only one study would have contributed to this.113
FIGURE 10 Effect sizes and meta-analysis for STIs in included studies, < 3 months post intervention. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
FIGURE 11 Effect sizes for STIs in included study, 3 months to 1 year post intervention. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
Effects on sexual risk outcomes
A total of 11 studies presented estimates for sexual risk outcomes (Table 7).100,102,110,113,115,116,122–125,127 Nine studies100,101,113,115,116,122,123,125,127 presented short-term results, whereas six studies102,113,115,116,124,127 presented mid-term results. One study117 intended to present short-term results relating to sexual risk outcomes, but did not estimate an effect because of an unexpectedly low sample size. The primary method of analysis in Reback et al.115 was a longitudinal regression model, and we present these estimates as well. There was some indication of short-term impacts on sexual risk in included interventions, but no such effect was apparent in mid-term results.
| Intervention | Measure | Notes |
|---|---|---|
| China–Gate HIV Prevention Program124 | CAI in previous 3 months (6-month follow-up) | |
| Cognitive Vaccine Approach102 | Condom use (6-month follow-up), tailored intervention | Dichotomised to always or not always using condoms with steady partner |
| Negotiated safety (6-month follow-up), tailored intervention | Dichotomised to practising or not practising negotiated safety with steady partner; negotiated safety defined as CAI between partners who are both HIV negative, agree to be monogamous or have no CAI with other partners and agree to tell each other if they do have an episode of CAI with another partner | |
| Condom use (6-month follow-up), non-tailored intervention | Dichotomised to always or not always using condoms with steady partner | |
| Negotiated safety (6-month follow-up), non-tailored intervention | Dichotomised to practising or not practising negotiated safety with steady partner; negotiated safety defined as CAI between partners who are both HIV negative, agree to be monogamous or have no CAI with other partners and agree to tell each other if they do have an episode of CAI with another partner | |
| Hot and Safe M4M100 | Insertive CAI, positive/unknown serostatus (3 months post baseline) | Number of acts |
| Receptive CAI, positive/unknown serostatus (3 months post baseline) | Number of acts | |
| CAI, positive/unknown serostatus (3 months post baseline) | Number of acts | |
| Insertive CAI, any partner (3 months post baseline) | Number of acts | |
| Receptive CAI, any partner (3 months post baseline) | Number of acts | |
| CAI, any partner (3 months post baseline) | Number of acts | |
| Insertive condomless oral intercourse, positive/unknown serostatus (3 months post baseline) | Number of acts | |
| Receptive condomless oral intercourse, positive/unknown serostatus (3 months post baseline) | Number of acts | |
| Insertive condomless oral intercourse, any partner (3 months post baseline) | Number of acts | |
| Receptive condomless oral intercourse, any partner (3 months post baseline) | Number of acts | |
| Keep it Up!122 | Condom failures (12-week follow-up) | Condom failure (e.g. condom breaking during sex) assessed on Likert scale |
| Total number of CAI acts in previous 6 weeks (12-week follow-up) | ||
| Condom use errors (12-week follow-up) | Condom use errors (e.g. using oil-based lubricant) assessed on Likert scale | |
| Keep it Up!113 | Any CAI (3-month follow-up post baseline) | Dichotomised based on partner-level reports |
| Any CAI in previous 3 months (6-month follow-up post baseline) | Dichotomised based on partner-level reports | |
| Any CAI in previous 3 months (12-month follow-up post baseline) | Dichotomised based on partner-level reports | |
| MyDEX123 | Insertive CAI (90-day follow-up) | Dichotomised |
| Receptive CAI (90-day follow-up) | Dichotomised | |
| Serononconcordant receptive CAI (90-day follow-up) | Dichotomised; serostatus based on whether or not participants knew whether or not partners were on PrEP (if HIV-negative) and/or virally suppressed (if HIV-positive) | |
| Serononconcordant insertive CAI (90-day follow-up) | Dichotomised; serostatus based on whether or not participants knew whether partners were on PrEP (if HIV negative) and/or virally suppressed (if HIV positive) | |
| Safe Behaviour and Screening125 | Condom use during anal intercourse in previous 3 months (6 months post intervention) | Percentage of encounters |
| Sex Positive!127 | CAI, serodiscordant partners (3-month follow-up post baseline) | Based on last encounter with each of up to three most recent anal intercourse partners; serodiscordant partners were known to have a HIV-negative status. Estimates are change scores |
| CAI, serodiscordant partners in previous 3 months (12-month follow-up post baseline) | Based on last encounter with each of up to three most recent anal intercourse partners; serodiscordant partners were known to have a HIV-negative status. Estimates are changes scores | |
| CAI, unknown serodiscordant partners (3-month follow-up post baseline) | Based on last encounter with each of up to three most recent anal intercourse partners. Estimates are change scores | |
| CAI, unknown serodiscordant partners in previous 3 months (12-month follow-up post baseline) | Based on last encounter with each of up to three most recent anal intercourse partners. Estimates are change scores | |
| Sexpulse116 | CAI in the previous 90 days (3 months post baseline) | Number of partners |
| CAI in the previous 90 days (6 months post baseline) | Number of partners | |
| CAI in the previous 90 days (9 months post baseline) | Number of partners | |
| CAI in the previous 90 days (12 months post baseline) | Number of partners | |
| TXT-Auto115 | Episodes of CAI with exchange partner in previous 30 days (8 weeks post baseline) | |
| Episodes of CAI with exchange partner in previous 30 days (3 months post baseline) | ||
| Episodes of CAI with exchange partner in previous 30 days (6 months post baseline) | ||
| Episodes of CAI with exchange partner in previous 30 days (9 months post baseline) | ||
| Episodes of CAI with casual partner in previous 30 days (8 weeks post baseline) | ||
| Episodes of CAI with casual partner in previous 30 days (3 months post baseline) | ||
| Episodes of CAI with casual partner in previous 30 days (6 months post baseline) | ||
| Episodes of CAI with casual partner in previous 30 days (9 months post baseline) | ||
| Episodes of CAI with methamphetamines in previous 30 days (8 weeks post baseline) | ||
| Episodes of CAI with methamphetamines in previous 30 days (3 months post baseline) | ||
| Episodes of CAI with methamphetamines in previous 30 days (6 months post baseline) | ||
| Episodes of CAI with methamphetamines in previous 30 days (9 months post baseline) | ||
| Episodes of sex with methamphetamines in previous 30 days (8 weeks post baseline) | ||
| Episodes of sex with methamphetamines in previous 30 days (6 months post baseline) | ||
| Episodes of sex with methamphetamines in previous 30 days (9 months post baseline) | ||
| Episodes of CAI with anonymous partner in previous 30 days (8 weeks post baseline) | ||
| Episodes of CAI with anonymous partner in previous 30 days (3 months post baseline) | ||
| Episodes of CAI with anonymous partner in previous 30 days (6 months post baseline) | ||
| Episodes of CAI with anonymous partner in previous 30 days (9 months post baseline) | ||
| Episodes of CAI with main partner in previous 30 days (8 weeks post baseline) | ||
| Episodes of CAI with main partner in previous 30 days (3 months post baseline) | ||
| Episodes of CAI with main partner in previous 30 days (6 months post baseline) | ||
| Episodes of CAI with main partner in previous 30 days (9 months post baseline) |
Short-term results
Effect estimates presented for short-term results fell into four categories: condomless sex (at the time of the included studies’ publication this was usually referred to as unprotected sex acts), condom use, serononconcordant sex acts and sex acts under the influence of drugs.
Condomless sex
Six studies100,113,115,116,122,123 presenting short-term results for condomless sex yielded inconsistent evidence as to the effectiveness of interventions on this outcome. First, considering evaluations of time-limited interactive online modular interventions, in their evaluation of the myDEx intervention, Bauermeister et al.123 found that, at 90 days post randomisation (i.e. at post intervention), intervention recipients (n = 95) had significantly lower odds than attention control recipients (n = 28) of any condomless receptive anal intercourse during the prior 3 months (OR 0.43, 95% CI 0.20 to 0.94). The reduction was lower in magnitude and non-significant for any condomless insertive anal intercourse (OR 0.64, 95% CI 0.28 to 1.44). However, at 3 months post baseline, Carpenter et al.100 did not find that the Hot and Safe M4M intervention generated significant differences between groups on any CAI, condomless insertive anal intercourse, condomless receptive anal intercourse, condomless insertive oral intercourse or condomless receptive oral intercourse (intervention group, n = 59; control group, n = 53).100 Specific significance tests per outcome were not provided, although we were able to calculate standardised mean differences (see Figure 12 for specific estimates), none of which suggested a significant impact of the intervention. In the first evaluation of the Keep it Up! intervention, Mustanski et al.122 found that, at 12 weeks post intervention, those who received the intervention had a lower rate of CAI acts of borderline statistical significance (rate ratio 0.56; p = 0.04, n = 63). However, in the second evaluation of Keep it Up!,113 there was no significant difference between groups for any CAI acts, number of male CAI partners overall or number of casual CAI partners at 3 months post randomisation (intervention group, n = 367; control group, n = 410), although specific significance tests were not reported for these outcomes. Calculated standardised mean differences between groups on reports of any CAI did not suggest a significant difference (d = –0.10, 95% CI –0.26 to 0.06). Finally, Rosser et al.116 estimated that the Sexpulse intervention reduced the number of male CAI partners by 16.8% at 3 months post baseline, although this effect was only marginally significant (95% CI 0.691 to 1.000, intervention group, n = 267; control group, n = 293).116
Considering time-limited interactive computer games, a study by Christensen et al.101 examined the mediating impact of shame on the number of unprotected anal intercourse events in the preceding 3 months in the SOLVE intervention at 3 months post baseline (intervention group, n = 437; control group, n = 484). Estimates of the intervention’s total impact on CAI were not presented, but the significant reported indirect effect on CAI through shame suggests a significant total effect of the intervention on CAI. However, these estimates are not directly comparable to the other tests of intervention impact presented here.
Turning to open-ended interventions with content organised by assessment, Reback et al.115 did not undertake end point-specific tests for CAI outcomes; however, we calculated that the intervention did not reduce episodes of CAI (i.e. the number of times that people had CAI) with main partners, anonymous partners, partners for transactional sex or casual partners at 8 weeks or 3 months post baseline (Figure 12 presents specific estimates: at 8 weeks, intervention group, n = 82; control group, n = 79; at 3 months, intervention group, n = 82; control group, n = 83). There was some signal of a harmful effect in terms of the intervention group having a higher number of CAI episodes with casual partners at 8 weeks post baseline, but this may have been due to substantial baseline imbalance.
Condom use
Two studies122,125 presented short-term results for condom use and together suggested an inconsistent indication of effectiveness on this outcome. At 6 months post baseline, Chiou et al.125 found that the Safe Behaviour and Screening open-ended intervention with general content increased the proportion of anal intercourse encounters in which condoms were used by 20.7% [standard error (SE) 0.058; p = 0.001, intervention group, n = 130; control group, n = 135]. Similarly, in the first evaluation of the Keep it Up! time-limited interactive online modular intervention, Mustanski et al.122 showed that the intervention reduced the number of condom use errors (d = –0.19; p = 0.56) and condom failures (d = –0.22; p = 0.30), but not significantly so. This analysis included 50 intervention and 52 control participants.
Human immunodeficiency virus serononconcordant sex
Three studies presented short-term results for HIV serononconcordant sex acts with other men, and yielded mixed evidence on the effectiveness of time-limited/modular interventions for this outcome.100,123,127 First considering time-limited interactive online modular interventions, in their evaluation of the myDEx intervention, Bauermeister et al.123 found that, at 90 days post randomisation (i.e. at post intervention), intervention recipients (n = 95) had lower odds than attention control recipients (n = 28) of any condomless receptive anal intercourse with serodiscordant or serounknown partners not known to be on PrEP or virally suppressed during the preceding 3 months, but not significantly so (OR 0.44, 95% CI 0.15 to 1.31); a similar pattern was found for insertive anal intercourse (OR 0.49, 95% CI 0.17 to 1.33).123 However, at 3 months post baseline, Carpenter et al.100 found that the Hot and Safe M4M intervention generated greater reductions in all CAI events with partners of positive or unknown serostatus [group by time F7,59, degrees of freedom (df) = 1101; p = 0.007], including condomless insertive anal intercourse (group by time F7,24, df = 1101; p = 0.008), but not condomless receptive anal intercourse (group by time F1,35, df = 1101; p = 0.248). The intervention group also reported reduced condomless insertive oral intercourse events (group by time F7,45, df = 1101; p = 0.007) and reduced condomless receptive oral intercourse events with partners of positive or unknown serostatus (group by time F8,45, df = 1101; p = 0.004), with analyses drawing on 59 intervention group and 53 control group participants.100
Turning to time-limited non-interactive video series interventions, Hirshfield et al.127 found that, at 3 months post baseline, the Sex Positive! intervention did not reduce either the number of CAI partners known to be serodiscordant (adjusted standardised β = 0.003, 95% CI –0.168 to 0.178) or the number of CAI partners not known to be seroconcordant (adjusted standardised β = –0.073, 95% CI –0.332 to 0.051).
Sex acts under the influence of drugs
Reback et al.115 was the only study to present short-term results for this category of sexual risk outcomes, evaluating an intervention with open-ended content organised by assessment. We calculated differences using end-point means. At neither 8 weeks post baseline (d = 0.23, 95% CI –0.08 to 0.54, intervention group, n = 82; control group, n = 79) nor 3 months post baseline (d = 0.08, 95% CI –0.23 to 0.38, intervention group, n = 82; control group, n = 83) was there a significant difference between groups for episodes of sex while on methamphetamines.
Mid-term results
Effect estimates presented for mid-term results fell into four categories: condomless sex, condom use, serononconcordant sex acts and sex acts under the influence of drugs.
Condomless sex
Four studies presenting mid-term results for condomless sex yielded inconsistent evidence as to the effectiveness of interventions on this outcome.113,115,116,124 First, considering a time-limited interactive online modular intervention, in the evaluation of the China–Gate HIV Prevention Program,124 at 6 months post baseline, intervention participants were less likely than control participants to report CAI in the previous 3 months, with a risk difference of 9.3% (95% CI 1.1% to 17.5%, intervention group, n = 501; control group, n = 485); estimates using multiple imputation to include the entire sample (intervention group, n = 550; control group, n = 550) generated a similar estimate (8.9%, 95% CI 1.2% to 16.6%). In the second evaluation of Keep it Up!, there was no significant difference between groups for the numbers of male casual CAI acts and the number of CAI partners at 6 or 12 months post randomisation, although specific significance tests were not reported for these outcomes.113 However, at 12 months post randomisation, intervention participants were 17% less likely to report any CAI in the previous 3 months (95% CI 0.70 to 0.99, intervention group, n = 366; control group, n = 391). Rosser et al.116 estimated that the Sexpulse intervention did not reduce the number of male CAI partners at 12 months post baseline (incidence rate ratio 0.998, 95% CI 0.952 to 1.046, intervention group, n = 276; control group, n = 278). We also calculated that there was no evidence of a significant effect at 6 months (d = –0.13, 95% CI –0.29 to 0.04) or 9 months (d = –0.10, 95% CI –0.27 to 0.06) post baseline.
Turning to open-ended interventions with content organised by assessment, Reback et al.115 did not undertake end point-specific tests for CAI outcomes; however, we calculated that the intervention did not reduce episodes of CAI with main partners, anonymous partners, partners for transactional sex or casual partners at 6 or 9 months post baseline (see Figure 13 for specific estimates; at 6 months: intervention group, n = 83; control group, n = 78; at 9 months: intervention group, n = 85; control group, n = 84).
Condom use and serononconcordant sex acts with other men
Davidovich et al.102 presented mid-term results for condom use with no evidence of effectiveness, whereas two studies102,127 presenting mid-term results for serononconcordant sex acts with other men yielded inconsistent evidence as to the effectiveness of interventions on this outcome. In a time-limited non-interactive online modular intervention termed the Cognitive Vaccine Approach, intervention participants receiving the tailored intervention (n = 128) were significantly more likely than control participants (n = 140) to practise negotiated safety (seroconcordant CAI, or no condom use only in the context of monogamous relationships) than to have CAI with partners of unknown HIV concordance (OR 10.50, 95% CI 1.19 to 92.72); however, intervention participants receiving the non-tailored intervention did not show a significant difference on this outcome compared with control recipients (OR 1.62, 95% CI 0.14 to 19.07).102 In this same study, intervention participants did not have odds of condom use that were significantly different from those of the control participants at 6 months post baseline. This was the case comparing either the tailored version of the intervention (OR 1.66, 95% CI 0.68 to 4.02) or the non-tailored version (n = 107) of the intervention (OR 0.55, 95% CI 0.22 to 1.37) with control (n = 140), with OR values > 1 suggesting increased condom use. However, it should be noted that, in this analysis, negotiated safety, condom use and other CAI were mutually exclusive categories estimated as part of a multinomial regression model. In addition, Hirshfield et al.127 found that, at 12 months post baseline, the Sex Positive! time-limited non-interactive video series intervention did not significantly reduce either the number of CAI partners known to be serodiscordant (adjusted standardised β = –0.073, 95% CI –0.332 to 0.051) or the number of CAI partners not specifically known to be seroconcordant (adjusted standardised β = –0.084, 95% CI –0.399 to 0.045).
Sex acts under the influence of drugs
Reback et al.115 was the only study to present mid-term results for this category of sexual risk outcomes, for an intervention with open-ended content organised by assessment. We calculated differences using end-point means. At neither 6 months post baseline (d = –0.10, 95% CI –0.41 to 0.21, intervention group, n = 83; control group, n = 78) nor 9 months post baseline (d = –0.18, 95% CI –0.48 to 0.13, intervention group, n = 85; control group, n = 84) was there a significant difference between groups for episodes of sex while using methamphetamines.
Overall results
Reback et al.115 used, as their primary method of analysis, a longitudinal regression model with a continuous variable for follow-up (range 0–4), with the test of intervention effectiveness being the interaction between intervention condition and time. In the regression models presented, the interactions between allocation to TXT-Auto and time were not significant for episodes of CAI with main partners (β = –0.16, 95% CI –0.31 to 0.003), episodes of CAI with casual partners (β = –0.01, 95% CI –0.06 to 0.03), episodes of CAI with anonymous partners (β = –0.05, 95% CI –0.10 to 0.003) or episodes of CAI with partners for transactional sex (β = –0.03, 95% CI –0.18 to 0.11). Similarly, the regression model for episodes of sex while using methamphetamines did not yield a significant interaction (β = –0.05, 95% CI –0.11 to –0.009). This coefficient is interpreted as the change between groups between follow-up periods in standard deviations. Thus, those allocated to TXT-Auto had a faster reduction in risk in all outcomes, but this difference in rate of change was not significant.
Equity-relevant characteristics
Only Cheng et al.124 presented results broken down by equity-relevant characteristics in relation to the evaluation of the interactive online modular China–Gate HIV Prevention Program. On the primary outcome, CAI in the previous 3 months, there was no evidence of effect modification by annual income (p = 0.445); however, there was significant effect modification by educational attainment (p = 0.012), with those in the category of least education enjoying a greater benefit than those with a university education or above (difference in effects 2.6%).
Meta-analysis
Effect sizes for sexual risk outcomes are presented in Figure 12 for outcomes of < 3 months post intervention (short term) and in Figure 13 for outcomes between 3 months and 1 year post intervention (mid-term). In both plots, negative effect sizes represent benefits. A meta-analysis drawing on 32 effect sizes from eight studies100,113,115,116,122,123,125,127 with < 3 months’ follow-up found a suggestion of effectiveness, albeit not statistically significant (d = –0.14, 95% CI –0.30 to 0.03) with substantial heterogeneity (I2 = 61%). The certainty of evidence for this meta-analysis was graded as very low because of risk of bias (details of randomisation, selective outcome reporting), inconsistency of studies, and publication bias arising from the non-inclusion of two studies.101,117 A meta-analysis drawing on 22 effect sizes from six studies102,113,115,116,124,127 with 3 months to 1 year of follow-up suggested a significant impact on reducing sexual risk (d = –0.12, 95% CI –0.19 to –0.05), but with low heterogeneity (I2 = 27%). The certainty of evidence for this meta-analysis was graded as low because of risk of bias (details of randomisation, selective outcome reporting).
FIGURE 12 Effect sizes and meta-analysis for sexual risk outcomes in included studies, < 3 months post intervention. RVE, robust variance estimation. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
FIGURE 13 Effect sizes and meta-analysis for sexual risk outcomes in included studies, 3 months to 1 year post intervention. RVE, robust variance estimation. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
We then pooled estimates regardless of follow-up time. Based on 54 effect sizes from 10 studies,100,102,113,115,116,122–125,127 interventions significantly reduced sexual risk compared with control groups (d = –0.15, 95% CI –0.26 to –0.05). This meta-analysis had substantial heterogeneity (I2 = 56%). To explore this heterogeneity, we compared interactive interventions100,113,115,116,122–125 with non-interactive interventions.102,127 A random-effects meta-regression did not suggest that this characteristic accounted for heterogeneity with non-interactive interventions not meaningfully worse than interactive interventions (β = 0.12, 95% CI –0.66 to 0.89).
Effects on alcohol and drug use
Two studies presented estimates for drug use: Chiou et al.125 and Reback et al.115 (Table 8), both focused on open-ended interventions. Neither quantified outcomes on alcohol use specifically. Although both studies presented short-term results, only Reback et al.115 presented mid-term results. However, the primary method of analysis in that study was a longitudinal regression model; we present these estimates as well. Across studies, there was no consistent evidence of effectiveness of interventions in reducing drug use.
| Intervention | Measure | Notes |
|---|---|---|
| Safe Behaviour and Screening125 | Recreational drug use in previous 3 months (6-month follow-up post intervention) | Asked about various recreational drugs (not all of which were specified in the report, but among which alcohol was included) and assessed on Likert scale |
| TXT-Auto115 | Days of methamphetamine use in previous 30 days (8 weeks post baseline) | |
| Days of methamphetamine use in previous 30 days (3 months post baseline) | ||
| Days of methamphetamine use in previous 30 days (6 months post baseline) | ||
| Days of methamphetamine use in previous 30 days (9 months post baseline) |
Short-term results
After 6 months of app use (i.e. at 6 months post randomisation), Chiou et al.125 found that the Safe Behaviour and Screening open-ended intervention with general content reduced drug use, as measured on a five-point Likert scale (β = –1.19, SE 0.204; p < 0.001; intervention group, n = 130; control group, n = 135). We used estimates from table 2 in Reback et al.115 to compare the TXT-Auto intervention, an open-ended intervention with content organised by assessment, with the assessment only (AO) condition on the outcome of days of methamphetamine use. Neither at the post-intervention follow-up at 8 weeks post randomisation (d = 0.15, 95% CI –0.16 to 0.45; intervention group, n = 83; control group, n = 79) nor at 3 months post randomisation (d = –0.03, 95% CI –0.34 to 0.28; intervention group, n = 82; control group, n = 83) was there evidence of a difference between the conditions. Statistical heterogeneity precluded meta-analysis of results, and certainty of the assessment of evidence was graded as being very low because of risk of bias (details of randomisation), inconsistency of studies and imprecision of effect estimates.
Mid-term results
We similarly estimated mid-term effects in Reback et al.115 for the outcome of days of methamphetamine use. At 6 months post randomisation, there was no evidence of a difference between the TXT-Auto intervention and the AO condition on this outcome (d = 0.23, 95% CI –0.07 to 0.54; intervention group, n = 83; control group, n = 78). A similar estimate was produced at 9 months post randomisation (d = 0.28, 95% CI –0.02 to 0.59; intervention group, n = 85; control group, n = 84). Both estimates suggested a possible, although not significant, intervention effect of increased days of methamphetamine use in the intervention arm; however, authors noted that this could have been due to baseline imbalance between arms.
Overall results
Reback et al.115 used, as their primary method of analysis, a longitudinal regression model with a continuous variable for follow-up (range 0–4), with the test of intervention effectiveness being the interaction between intervention condition and time. In the regression model presented, the interaction between allocation to TXT-Auto and time was not significant (β = 0.01, 95% CI –0.04 to 0.06). This coefficient is interpreted as the change between groups between follow-up periods in standard deviations. The main effect for time was significant, suggesting a decrease in days of methamphetamine use between follow-ups of 0.10 standard deviations (β = –0.10, 95% CI –0.14 to –0.06). Thus, those allocated to TXT-Auto had a slower rate than the control group of decrease in days of methamphetamine use of 0.01 standard deviations, but this difference in rate of change was not significant.
Meta-analysis
Effect sizes for alcohol and drug use are presented in Figure 14 for outcomes of < 3 months post intervention and in Figure 15 for outcomes between 3 months and 1 year post intervention. In both plots, negative effect sizes represent benefits. We do not present a pooled effect size for these estimates as meta-analyses included both substantial heterogeneity (I2 = 95% both overall and only including effect sizes of up to 3 months’ follow-up) and too few studies to explore this heterogeneity.
FIGURE 14 Effect sizes for alcohol and drug use in included studies, < 3 months post intervention. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
FIGURE 15 Effect sizes for drug use in included studies, 3 months to 1 year post intervention. This figure has been adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)
Grading of Recommendations Assessment, Development and Evaluation analysis
Findings from a GRADE analysis of included outcomes are presented in Table 9. The evidence generated a score of low or very low confidence for most outcomes. This was primarily because of risk of bias and imprecision in the evidence.
| Certainty assessment | Summary of effects | Certainty | Importance | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Studies (n) | Study design | Risk of bias | Inconsistency | Indirectness | Imprecision | Other considerations | |||
| Drug use (short term) | |||||||||
| 2 | Randomised trials | Seriousa | Very seriousb | Not serious | Very seriousc | None | Estimates could not be pooled owing to high heterogeneity and few effect sizes | ⨁◯◯◯ Very low | Critical |
| Drug use (mid-term) | |||||||||
| 1 | Randomised trials | Seriousa | Not serious | Not serious | Very seriousc | None | Estimates drew from one study, Reback et al.115 This study did not provide evidence of effectiveness | ⨁◯◯◯ Very low | Critical |
| HIV infections (short term) | |||||||||
| 1 | Randomised trials | Not serious | Not serious | Not serious | Very seriousc | None | Estimates drew from one study, Chiou et al.125 This study did not provide evidence of effectiveness | ⨁⨁◯◯ Low | Critical |
| HIV infections (mid-term) | |||||||||
| 1 | Randomised trials | Seriousd | Not serious | Not serious | Very seriousc | None | Estimates drew from one study, Mustanski et al.113 This study did not provide evidence of effectiveness | ⨁◯◯◯ Very low | Critical |
| STIs (short term) | |||||||||
| 2 | Randomised trials | Very seriousa,d | Not serious | Not serious | Very seriousc | None | A pooled estimate of both studies suggested a non-significant increase in STIs as a result of interventions (d = 0.17, 95% CI –0.18 to 0.52) | ⨁◯◯◯ Very low | Critical |
| STIs (mid-term) | |||||||||
| 1 | Randomised trials | Seriousd | Not serious | Not serious | Not serious | None | Estimates drew from one study, Mustanski et al.113 The pooled outcome of ‘any STI’ suggested a significant reduction in risk of STIs (RR 0.32, 95% CI 0.40 to 0.83) | ⨁⨁⨁◯ Moderate | Critical |
| Sexual risk behaviour (short term) | |||||||||
| 8 | Randomised trials | Very seriousa,d | Seriouse | Not serious | Not serious | Publication bias strongly suspectedf | Pooled estimates suggested a non-significant decrease in sexual risk behaviour (d = –0.14, 95% CI –0.30 to 0.03) | ⨁◯◯◯ Very low | Critical |
| Sexual risk behaviour (mid-term) | |||||||||
| 6 | Randomised trials | Very seriousa,d | Not serious | Not serious | Not serious | None | Pooled estimates suggested a significant reduction in sexual risk behaviour (d = –0.12, 95% CI –0.19 to –0.05) | ⨁⨁◯◯ Low | Critical |
Synthesis of studies comparing active interventions
Two studies included comparisons between active interventions.99,102 Both studies reported sexual risk outcomes only, and only Davidovich et al.102 presented extractable outcome data. As a result, a meta-analysis was not undertaken.
Risk of bias of trials comparing active interventions
Sequence generation
Neither of the two trials presenting active versus active comparisons presented enough information to judge sequence allocation.99,102
Allocation concealment
Both trials reported information that suggested that allocation was sufficiently concealed, owing to online randomisation.99,102
Blinding of participants and personnel
Davidovich et al.102 stated that participants were blinded but study personnel were not. Bowen et al.99 did not present enough information to judge the blinding of participants or personnel.
Blinding of outcome assessors
Davidovich et al.102 was rated as having a low risk in this domain because, even though outcomes were self-reported, participants were blinded. Bowen et al.99 did not present enough information to judge this item.
Complete outcome data
Both trials had high levels of attrition (> 30%) and imbalanced attrition between arms.99,102
No selective outcome reporting
Davidovich et al.102 reported all stated outcomes and was thus rated as being at low risk of bias for this domain. Bowen et al.99 was rated as being at high risk of bias for this domain as no concrete estimates for between-group differences were presented.
Accounted for clustering
Neither study required accounting for clustering.
Reducing other sources of bias
Davidovich et al.102 was rated as being at low risk of bias for this domain as the trial included multiple methods of recruitment. Bowen et al.99 was rated as being at high risk of bias for this domain as no clear evidence of effectiveness or ineffectiveness was presented in this trial.
Narrative summary of trials comparing active interventions
Davidovich et al.102 included a three-arm trial testing tailored and non-tailored versions of their time-limited non-interactive online modular interventions against a no-treatment control group. We calculated that participants receiving the tailored intervention (n = 128) were more likely than participants receiving the non-tailored intervention (n = 107) to report, at 6 months post baseline, the practice of negotiated safety (defined above), compared with CAI with other partners (OR 6.50, 95% CI 2.49 to 16.90), and the practice of condom use, compared with CAI (OR 2.98, 95% CI 1.74 to 5.12).
In their evaluation of time-limited interactive online modular interventions, Bowen et al.99 tested the impact of ordering modules with content about HIV knowledge, content about risk in casual or new partnerships and content about contexts of risk on the proportion of anal intercourse partners with whom a condom was used every time. At post intervention, there was no statistical difference between modules on sexual risk, but specific group differences were not presented.