About this chapter
Parts of this chapter have been reproduced or adapted with permission from Meiksin et al.57 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes additions and minor changes to the original text.
Reports included in process evaluation synthesis
Twelve reports on 11 studies of eight unique interventions were eligible for inclusion in the process evaluation synthesis.96,98,103,107,108,111,118,119,121–123,130 Appendix 9 summarises characteristics of included process evaluations and Table 4 provides information on the health outcomes of interest for this review that were targeted by all included interventions. Included process evaluation reports presented findings on how intervention receipt (but not delivery) varied by characteristics of the intervention,96,103,107,108,111,118,119,121–123,130 participants96,103,107,108,118,119,121,130 and context,98,118,121 but not providers. Included studies reported on interventions that span three intervention types identified in our typology. These include online modular (eight reports,96,98,103,111,121–123,130 five interventions) and computer game (two reports,107,108 one intervention) interventions, both of which were interactive time-limited interventions, as well as open-ended interventions with general content with core content not tailored by assessment (two reports,118,119 two interventions). Three interventions addressed sexual health only,98,111,118,121 two addressed mental health only,96,107,108 one addressed sexual health and substance use103,122,130 and two addressed all three outcomes of interest for this review.119,123
Four interventions targeted sexual minority youth or young adults,103,107,108,111,122,123,130 two targeted MSM more generally,96,118 one targeted rural MSM98,121 and one targeted people living with HIV.119 Five were delivered via the internet,96,98,103,111,121–123,130 two via smartphone app118,119 and one via computer CD-ROM.107,108 Process evaluations for seven of the included interventions took place in the USA96,98,103,111,118,119,121–123,130 and one took place in New Zealand.107,108
Direct quotations from participants were limited in the included reports and are therefore limited in our reporting on the themes identified in our analysis.
Quality of studies
Appendix 9 shows the results of our quality assessments of each process evaluation report. Overall, quality varied, with most reports assessed as being of medium or high quality. In terms of the reliability or trustworthiness of their overall findings, four reports were assessed as being of medium quality96,111,118,119 and eight as high quality.98,103,107,108,121–123,130 In terms of their overall usefulness for addressing our research questions, four were assessed as being of low quality,98,111,121,123 three as medium quality107,108,122 and five as high quality.96,103,118,119,130 Only two reports were assessed as being of high quality in terms of both reliability/trustworthiness and usefulness103 and all were assessed as being of medium or high quality in at least one of these two domains.96,98,103,107,108,111,118,119,121–123,130
Quality varied across the six individual criteria feeding into these overall assessments. The vast majority of the included process evaluations took steps to (1) minimise bias and error/increase rigour in sampling (n = 11),98,103,107,108,111,118,119,121–123,130 (2) minimise bias and error/increase rigour in data collection (n = 10)96,103,107,111,118,119,121–123,130 and (3) minimise bias and error/increase rigour in data analysis (n = 10).96,98,103,107,111,119,121–123,130 Nearly all process evaluations reported findings that were grounded in/supported by the data (n = 11).96,98,103,107,108,111,118,119,121,123,130 Seven reports privileged the perspectives of MSM,96,103,107,111,119,122,130 but only four were assessed as having achieved good breadth and depth in their findings.96,118,119,130
Themes emerging from synthesis of process evaluation reports
Appendix 14 shows an example of coding structures developed independently by each of the two reviewers for the process evaluation report on the online mindfulness-based cognitive therapy intervention. Appendix 15 shows the relationship between primary, secondary and tertiary codes developed through our analysis and synthesis of process data.
Intervention characteristics affecting intervention receipt
Nearly all process evaluations explored ways in which intervention characteristics affected the receipt of interventions, although the included reports tended to lack breadth of the areas explored and in-depth exploration of the findings that they did report.96,103,107,108,111,118,119,121–123,130 However, several subthemes emerged in our analysis across studies.
Ease of use
Across health domains, acceptability was reported as being enhanced when interventions were easy to use and free from technical problems. Few technical problems were reported in studies of open-ended interventions. In studies assessed as being of medium reliability, 10% or fewer smartphone self-monitoring users reported technical difficulties119 and participants reported that the HealthMindr app was easy to use without needing technical assistance.118 However, evidence from studies assessed as being of medium96 and high98,103,107,121,122,130 reliability assessing online modular96,98,103,121,122,130 and computer game interventions107 suggested that, when users did encounter technical issues, such as freezing103 or incompatibility with mobile devices,96,130 this was linked with a lower level of acceptability in participant accounts. In a 2007 study of the Hope Project sexual health intervention targeting rural MSM,98 features such as sound, animation or graphics were reported as sometimes causing the programme to load too slowly for participants with slower internet speeds, which authors suggested might undermine participation. From studies assessed as being of medium96 and high107,108 reliability, accompanying materials outside the electronic environment were reported to potentially enhance acceptability, with users appreciating materials that could be printed96 and Rainbow SPARX users (sexual minority youth) giving positive feedback on an accompanying notebook that they could keep.107,108 Participants were reported to dislike exercises that required using materials that they might not have had readily to hand (e.g. raisins for an online mindfulness-based cognitive therapy intervention exercise).96
Intervention content
Clear and comprehensive content
From medium-reliability studies of online modular96,103,111 and open-ended (general content) interventions119 across health domains, it was apparent that intervention content that involved clear and comprehensive information was associated with increased acceptability in participant accounts. For example, Queer Sex Ed participants appreciated that this sexual health programme provided comprehensive information on a range of sexual health and relationship topics, rather than focusing narrowly on STIs.111 In studies of other interventions, acceptability was reported to be greater when content was clear and up to date,96 whereas content that users found confusing appeared to detract from acceptability.119
Some study participants recommended that specific content be added, including ‘[taking] into consideration the new PrEP [HIV pre-exposure prophylaxis] medicine and also [giving] realistic happenings without condom use’,103 and providing information on sexual health for trans people.111
Engaging intervention content
Fun111 and enjoyable96 content was associated with increased acceptability in participant accounts, and the use of different types of content arose as a common theme influencing acceptability in online modular and computer game interventions. For example, in studies assessed as being of medium96 and high103,107,122,130 reliability that, between them, focused on interventions addressing all three health outcomes, participants tended to give positive feedback on the use of diverse content,96,103,130 including animations, videos graphics and games,122 as well as on interventions’ visual appearances.96,107
In a high-reliability study of Rainbow SPARX, users were reportedly particularly positive about the computer game format and the intervention’s ‘look and feel’,107 as expressed by one user aged 13 years: ‘I liked, like, how it looked really shiny on my computer, and it looked like a completely different world.’ 107
Rainbow SPARX users were reported as also liking particular characters who appeared in the game,107 a theme echoed in a high-reliability study of the online modular Keep it Up! intervention (addressing sexual health and substance use) in which participants reported liking the scenarios and examples presented.122 Factors detracting from acceptability included content that participants found boring,96,119 repetitive,96,119 too easy,107 too difficult or draining,96 ‘not soothing’,96 ‘cheesy’103 or generally unenjoyable96 and videos that users judged to be too long or that featured low-quality sound or dialogue.130
Language and tone
Language and tone emerged as important aspects of acceptability across interventions addressing all three health domains and in studies assessed as being of medium96,118 and high103,107,108,122,130 reliability. Evidence for this theme came from online modular96,103,122,130, computer game107,108 and open-ended (general content) interventions.118 Keep it Up! participants liked what authors described as a ‘frank, candid, and sex-positive tone’,103 colloquial language and what one participant described as its ‘up-beat manner’.122 Queer Sex Ed users appreciated that the intervention did not rely on ‘scare tactics’ and that its content was easy to understand without making them feel ‘talked down to’.111 A Keep it Up! user echoed this sentiment, describing the intervention as ‘realistic and not condescending or out of touch’.130
There were also some challenges in getting the language right for MSM-specific interventions. For example, some users of Rainbow SPARX reportedly suggested that the intervention’s sexuality-related terminology could be improved,107 and some users of the online mindfulness-based cognitive therapy intervention were reported as voicing concerns about the intervention’s approach to sexual minorities and a feeling of ‘anti-gay sentiment’.96 In regard to the online mindfulness-based cognitive therapy intervention, the author’s findings suggested that some content might have been overly clinical and miscommunicated the aim of improving overall well-being,96 although it was not clear whether participant concerns stemmed primarily from the intervention’s content itself or from content about participating in a research study.
Interaction and personalisation
Participants in studies assessed as being of medium111 and high107,122,130 reliability, assessing online modular111,122,130 and computer game107 interventions, reportedly valued interactive aspects of interventions spanning all three health outcomes. For example, one Keep it Up! participant described the following:130
I liked that this program was very interactive. You were required to click on things and drag them places in order to get an answer correct. I feel like this allowed for more enhanced learning and retention of crucial information.
Studies assessed as being of medium118,119 and high103 reliability reported that individual-level tailoring based on participant assessments could enhance acceptability. For example, 81% of HealthMindr users were reported as finding recommendations based on their responses useful or very useful,118 and smartphone self-monitoring users reportedly recommended adding, what the authors summarised as, ‘more in-depth questions to better reflect their experiences’.119
Privacy and intrusiveness
In studies assessed as being of medium reliability, privacy and intrusiveness emerged as important themes influencing acceptability across two open-ended (general content) interventions that, between them, addressed all three health outcomes.118,119 Some smartphone self-monitoring users reportedly felt that the intervention’s use of daily surveys on substance use, sexual behaviours and medication adherence, and four-times daily surveys on physical and mental health-related quality of life, were too long and/or too frequent, and therefore intrusive.119 Users expressed concerns about privacy regarding questions about sexual behaviour, including experiences with individual partners.119 The vast majority of HealthMindr app users (86%) reported feeling confident in the app’s security, including its personal identification number/password features and the fact that the app’s name and icon did obviously relate to HIV prevention.118 At least one smartphone self-monitoring user was reported as being uncomfortable with geolocation tagging of phone survey responses, although the authors noted that participants were instructed on how to disable this feature.119
Pacing and structuring
The pacing and structuring of content influenced acceptability across health domains. In studies assessed as being of medium96 and high108,122,130 reliability, there was some evidence that a modular, as opposed to single-session, approach to an intervention could reportedly help users absorb content,122 although users were reported as tending to like setting their own pace,108 and one suggested that they would have preferred to complete all modules in one sitting.130 Requiring a full week between sessions for the online mindfulness-based cognitive therapy intervention was reported as too long, detracting from acceptability.96
Users were reported as liking intervention content that progressed in a cumulative way.96 Module order and how far a participant had progressed could also affect acceptability. Findings from a high-reliability study of the three-module Hope Project (targeting knowledge, motivation and behaviour to address sexual health), which randomised the order in which modules were delivered, suggested that participants were more likely to find the ‘knowledge’ module interesting when they encountered it last, rather than first.121 Assessing level of interest after each module among participants completing all modules, the study also found that participants were more likely to report finding modules very interesting after completing all three, compared with completing only one.121
Programme length arose as a common theme affecting the acceptability of some online modular interventions, with users of the eight-session online mindfulness-based cognitive therapy intervention,96 the seven-module Keep it Up! intervention103,122,130 and the five-module Queer Sex Ed intervention111 suggesting that these programmes were too long or too time-consuming. Some cited other commitments or being too busy as barriers to completing the interventions.96,103 According to one Queer Sex Ed user:111
This program was [way] too long. Like really long. My suggestion would be to either break it up into more sections or cut out some videos that only introduced a topic . . . The information was all very necessary, especially for queer kids, but keep attention spans in mind.
Content designed to be relevant to participants’ lives and experiences
Across intervention types and health outcomes, participants valued that interventions were designed for people like them. From studies of high reliability, it was apparent that participants valued interventions that presented realistic scenarios and examples and that addressed issues relevant to their own lives.103,107,122,130 A Keep it Up! user appreciated that the intervention ‘was geared towards gay men and it understood how we operate and how dating works in the contemporary moment’.130
Similarly, another participant thought the realistic scenarios presented by Keep it Up! would be especially helpful for less experienced MSM:103
I found the program extremely helpful because it encounters real situation[s] within the community such as hooking up online and or bars. I think it can be of great help to a young crowd that has not much experience into the gay scene.
Users of the Rainbow SPARX and Queer Sex Ed interventions for sexual minority youth reportedly liked that these programmes were ‘LGBT-specific’,111 designed for young people108 and included “‘rainbow’ content” tailored to this group.107 Some reportedly suggested that there was room to go further,107 for example by removing content on female sexual anatomy for MSM users and adding more trans-specific content.111
Online mindfulness-based cognitive therapy intervention users were reported as having mixed views on how effectively this intervention was tailored for people like them.96 Some reported appreciating that the programme was designed for men who were attracted to men, whereas others felt that the intervention ‘did not have much value in the context of their lives’.96 Some users of Rainbow SPARX reported that tailoring could be further enhanced by including more sexuality-specific content.107
Perceived usefulness of the intervention
Gaining knowledge and skills
In studies assessed as being of medium96 and high103,107,108,111,121,123,130 reliability, across several online modular and computer game interventions that, between them, addressed all three health outcomes, participants frequently indicated the importance of perceiving that the intervention was useful in terms of its aiming to increase knowledge and skills.95,96,102,103,106–108,110,111,120,121 Where users were reported as highlighting particular topics as useful to address, these included information about depression,108 drugs, alcohol, STI transmission and HIV;130 skills and knowledge for improving relationships;103,111 ‘mindfulness or other psychological skills or knowledge’;96 and relaxation and CBT techniques.107 Queer Sex Ed users were reported as liking that the intervention aimed to support communication and closeness with their partners, helping (as one participant described) to ‘. . . open up doors to healthy communication’.111
Opportunities for self-monitoring and self-reflection
Findings from the evaluation of the open-ended smartphone self-monitoring intervention (targeting sexual health, substance use and mental health outcomes) suggest that some participants valued its daily, mobile-based self-monitoring, compared with the comparison group’s biweekly web-based approach. One user described the benefits this way:119
Helps me keep a ‘log’, like therapy – but can do it every day instead of waiting for a week to see your therapist . . . Nice to do it throughout the day, multiple times a day, on a daily basis. Life happens daily – not weekly like when you see a therapist.
Similarly, users of the online modular Keep it Up! and online mindfulness-based cognitive therapy interventions, and of the open-ended smartphone self-monitoring intervention, which between them addressed all three health outcomes, reportedly highlighted the opportunities for introspection and self-reflection that these interventions presented.96,103,119,122,130 As a smartphone self-monitoring intervention user said:119
I started changing my behavior once I started taking the surveys – I have been thinking about it for a while but the surveys make me concentrate on certain areas of my life that I wasn’t focusing on.
A few also reported that engaging in self-monitoring across multiple domains enhanced their awareness of the relationships between their substance use, sexual behaviours and other triggers for drug use.119 A Keep it Up! user described how observing the characters in the intervention helped him reflect on his own behaviours:122
I was able to see mistakes that I make in the actions of the characters. I wasn’t completely aware of my behavior until I judged a character’s behavior and then compared the same behavior to my own.
Opportunity for self-expression
Participants in the smartphone self-monitoring intervention, which addressed all three health outcomes, were reported as valuing the opportunity for self-expression that the intervention offered, as described by one participant: ‘I feel free to vent to the phone about things that I can’t talk to my partner about – I can really express how I feel’.119
Specific intervention features
The two studies of interventions of the ‘open-ended’ type, which between them addressed all three health outcomes considered in this review, reported on the acceptability of particular programme features.
Smartphone self-monitoring users reportedly highlighted the role of both daily surveys and the programme’s reminder function in supporting their need to take medication at the right times.119 Regarding the features of the HealthMindr app, more than 80% of participants used the ‘ordering’ feature, with nearly two-thirds ordering condoms and more than half ordering at-home HIV test kits, and 70% used a tool to create a HIV testing plan.118 User data also showed engagement with PrEP contents, with approximately 40% viewing PrEP information and smaller proportions screening themselves for eligibility (approximately 25%) and using a map of PrEP providers (approximately 15%). Participants were less likely to use non-occupational post-exposure prophylaxis (nPEP) features, with around 25% viewing information on nPEP and less than 10% using a screening tool to assess their eligibility for nPEP.
Participant characteristics affecting intervention engagement and receipt
Evaluations of four online modular96,121,130 and open-ended (general content)118 interventions (two targeting sexual health only,118,121 one targeting mental health only96 and one targeting sexual health and substance use130) quantitatively explored the relationship between participant characteristics and intervention engagement.
A medium-reliability study of the HealthMindr sexual health mobile phone app found no differences in the time spent on the app by participant location (comparing different cities in the USA), age, race/ethnicity or knowledge of local HIV testing,118 while a high-reliability study of the Keep it Up! intervention targeting young ethnically and racially diverse MSM found that, among black users, those with graduate degrees spent more time on the intervention than those with high school or lower levels of education.130 A study assessed as being of medium reliability found no significant variation in retention for the eight-session modular mental health online mindfulness-based cognitive therapy intervention by age, SES, ethnicity, internalised homonegativity or experience of homophobic bullying.96 A study assessed as being of high reliability found no differences between participants completing one versus all three modules of the Hope Project (an extension of the WRAPP sexual health intervention, targeting rural MSM) by age, ethnicity, marital status, sexual orientation, education or student status, but did find higher completion rates among higher-earning participants.121
Madkins et al.130 conducted a high-reliability assessment of intervention receipt among different users of the Keep it Up! intervention, which was developed with the engagement of a diverse group of young MSM and designed for young MSM of all racial groups.122 Researchers found several differences in the acceptability of the Keep it Up! intervention by race/ethnicity, education level, age and city in the USA.130 Black, Latino and other non-white users reported higher acceptability in a range of domains than did white users, and Latino users rated content more highly than other non-white users. In the overall sample, users with high school-level education or lower rated the intervention more highly than those with a higher level of education. Exploring the interaction of race/ethnicity and education level, the study found that white users with higher levels of education reported lower acceptability, while no such differences were found among black, Latino or other non-white users. Older users and those in Atlanta tended to rate modules more highly than those in New York.
Exploring intervention receipt qualitatively, a study with high reliability found that, for Rainbow SPARX, a computer game intervention addressing mental health among sexual minority youth aged 13–19 years, some older users reported that some aspects were too easy and the programme ‘babied’ them.107 Acknowledging the challenge of designing a programme appropriate for a range of young people, one participant, aged 19 years, said:107
[S]ome things were a little easy . . . Overall it wasn’t difficult to figure out what you needed to do. Those . . . puzzles were quite easy to do. I guess it would be hard to make them more difficult though because you would have to be careful that everyone could actually get it.
Qualitative research suggested that how users experience e-health interventions could also vary by individual circumstances beyond sociodemographic factors. For young people who were also receiving external support (typically in-person therapy), Lucassen et al.108 found, in a high-reliability study, that Rainbow SPARX could play a unique role by complementing that support. In a medium-reliability study, Swendeman et al.119 found that the smartphone self-monitoring intervention, addressing all three health outcomes, could either support users’ antiretroviral therapy maintenance or promote recognition of patterns of non-adherence, depending on a user’s current level of adherence.
Contextual factors affecting intervention engagement
Few studies explored how the context for using the intervention was associated with the experience of its use. Those that did focused on internet speed in high-reliability 200798 and 2010121 studies of two iterations of the WRAPP sexual health intervention, which targeted rural MSM in the USA. Bowen et al.98 found that users with dial-up, compared with high-speed, internet connections were more likely to report taking too long to load programme graphics, while Williams et al.121 found no differences among participants completing one versus all three modules by type of internet connection.
Feedback from stakeholder consultation on synthesis of process evaluations
Stakeholders’ comments regarding process evaluation focused on four topics: context; usability/acceptability; content, language and tone; and privacy/intrusiveness.
Context
E-health interventions were felt by stakeholders to be broadly acceptable among MSM across geographical contexts. This varied by region, with stakeholders in London and Cardiff identifying larger groups of MSM whom they thought could benefit than stakeholders in Scotland and the north of England. This was largely due to regional variations in perceived health service availability compared with the level of need, alongside preferences for service types. Stakeholders from London identified the region as one with a high level of needs and greater demand than could be met by existing health services, thus increasing the potential utility of e-health interventions. Services in Cardiff were similarly felt to be insufficiently developed to meet existing needs.
There were concerns that poor access to broadband/the internet were issues that would negatively affect uptake in Scotland and the north of England.
It was felt that these interventions had significant potential in the UK, given funding constraints in sexual health and increasing imperatives around self-management.
Usability/acceptability
In line with process evaluation findings, stakeholders felt that it was critical to the success of e-health interventions for MSM that interventions were well-designed with minimal technical problems. There was a strong feeling that, should even minimal problems arise, it would have a profound impact on intervention engagement and effectiveness. Significant concern was expressed around the level of funding required to develop a suitable platform to deliver these interventions.
Content, language and tone
Stakeholders recognised that, in line with the process evaluation findings, appropriate content was central to the success of e-health interventions. The tone of the interventions was felt to be important in engaging a diverse group of MSM; matching intervention language to colloquial language used by MSM was felt to be essential. Furthermore, in line with the process evaluation findings, a candid tone and sex positivity were valued. Involvement of potential users in setting the tone and direction of these interventions was highlighted as an important component of formative development.
In addition, all stakeholders stressed the need for e-health interventions to be available in a range of languages to meet the needs of a broad range of MSM from migrant communities. This was regarded as particularly important given the higher engagement in sex work of MSM from Latin American and eastern European countries, recognising significant crossover between issues pertaining to sex work, mental health, substance use and sexual health.
Privacy and intrusiveness
Significant concerns regarding privacy and intrusiveness did not emerge among stakeholders. With regard to evaluations, as opposed to interventions, there was some concern about the burden of data collection for participants and concerns that it could negatively affect acceptability of similar interventions in the UK. Stakeholders felt that the number of data required to conduct rigorous evaluation might lead to a disjointed experience for service users, and that some may have concerns about confidentiality. In addition, one participant raised as a concern the number of data that might be collected and not used.