Recruitment, evaluation acceptability and survey sample characteristics

This chapter describes the recruitment of schools into the study, retention in the study, the study sample, and the feasibility and acceptability of the data collection methods.

Recruitment of schools into the study

All schools in West Lothian and south-west Edinburgh (n = 17) were initially contacted. Two schools did not respond to the e-mail introduction (and two follow-up e-mails). Five schools declined the invitation. Of these schools, two said that they did not have capacity, one expressed concern that the PS role would be too burdensome for students and two did not provide a reason (both were Roman Catholic). Of the 10 schools that expressed initial interest in participation, three ultimately decided that they did not have the capacity to take part and the other seven were recruited into the study. One school agreed to be the pilot school (reported in Chapter 2) and the other six schools took part in the feasibility trial.

In pre-study recruitment meetings, teachers were generally positive about the idea of the STASH study. The teachers recognised a need for high-quality and contemporary SRE, and saw that the STASH study could provide this. The teachers also saw how participation in the study could assist them in meeting nationally set health and well-being targets,116 and valued the potential leadership opportunities for PSs (see Chapter 5). Concerns raised focused primarily on the potential time commitment and on the use of social media. Regarding the time commitment, concern was raised about teacher time involved in delivering the survey and about students in a public examination year taking 2 days out of school. Regarding social media, several teachers expressed concern about the potential for online bullying. The teachers described how recent cases of social media bullying in school had led them to view social media negatively. Some teachers therefore felt uneasy about endorsing social media use. The research team explained that the STASH study could potentially provide young people (and schools) with an alternative view of social media, with social media potentially playing a positive role in health education and behaviour change. The team also sought to reassure teachers that the Facebook chats would be monitored by a trainer, and this reassurance seemed important in the decision to participate. Questions raised at this stage also concerned the nomination of PSs by other students, and the research team emphasised that PSs should be invited to the recruitment session regardless of teachers’ perceptions of their suitability to take on the role.

The school sample

All schools were mixed-sex, state-run schools in the Lothian region of Scotland. They were varied in terms of the geographic location, area-level deprivation (using the SIMD) and educational attainment. Table 4 shows the key characteristics of the sample.

Retention of schools in the study

All six schools in the feasibility trial remained in the study until the final data collection point and all schools undertook the key components of the intervention (see Chapter 2, The STASH study intervention design for the feasibility trial). All planned evaluation activities were carried out in each of the six schools. In one school (school 4), the contact teacher went on long-term sick leave and the role was taken up by a senior staff member who had not been part of the decision to participate in the STASH study but who was committed to the intervention. The impact of this is described in Chapters 5 and 6.

Acceptability of the outcome evaluation

Undertaking the control, baseline and follow-up surveys

We opted to investigate the feasibility and acceptability of web-based computer-assisted personal interview over paper-and-pencil interview. Computer-assisted self-interviewing enables more complex skip patterns, improves internal consistency and reduces missing items,117 but they are rarely used in school surveys.118

The control, baseline and follow-up web-based computer-assisted personal interview questionnaires were completed by students using school computers, with paper versions available in case of technological difficulties. Dates for data collection were arranged 4 months in advance and contact teachers were asked to book computing suites, apprise relevant staff and provide class lists. Reminders were sent 3 weeks prior to data collection. At the data collection sessions, a teacher was usually present to take the register and help with queries. Two fieldworkers were allocated to each computer room, with two or three rooms used at each session. A lead fieldworker moved between rooms, to ensure that participants had arrived in the correct room and to support the fieldworkers in answering any questions. Once the class was seated and the teacher had completed the register, the fieldworker would read a short script to the class. The script provided information about the study and emphasised that data would be confidential. Participants were given an envelope containing their username, password and a web slip with a link to the questionnaire website. Participants were asked to complete the questionnaire as if they were doing an examination, with two fieldworkers per classroom ‘invigilating’ this process. Most schools completed their data collection with three visits per wave of data collection. Occasionally, additional mop-up sessions were scheduled if absence was particularly high at one of the data collection sessions.

Timetabling fieldwork

Adverse weather in March 2018 had an impact on fieldwork at follow-up in schools 1 and 4; both schools were closed as a result of snow on the scheduled week for data collection, and rescheduling of fieldwork at short notice proved challenging because of increasing proximity to public examinations (April/May). In one school, maths revision classes were scheduled after the data collection date had been set and were given priority over the survey; the research team were keen to respect these decisions. Fieldwork was also occasionally disrupted by school activities, such as field trips, work experience or special events. Sometimes these were arranged after the data collection date had been set or the STASH study contact teachers were not made aware of them. Extracurricular activities were generally held during less-pressured times in the academic calendar, such as post examinations (June). There was therefore no ‘ideal time’ to collect data, as periods away from examinations were also those in which more extracurricular activities were scheduled.

Workload for schools

The STASH study contact teacher was required to organise booking of computer suites, arrange movement of students from class to the suites to complete the questionnaire and herd any stragglers or students who had not received the communication about the survey. According to fieldworker reports, school information technology infrastructure was a limiting factor, with the number of available personal computers and computer rooms dictating the number of data collection sessions required. Fieldwork staff reported that network connectivity was acceptable in five schools, but problematic in school 6. There was also a small number of technical issues associated with the web survey, including recurrent login issues.

Ease of data collection varied between schools, but was always smoother if the STASH study contact teacher was present or available. When the contact teacher was absent or called to deal with another issue, fieldworkers were reliant on other teachers to access booked rooms and herd students. Schools varied in their capacity to cope with the absence of the STASH study contact teacher, with those assigning a clear stand-in or depute contact teacher managing more easily. In two schools, the STASH study teacher was rarely available: one through absence and one through lack of capacity. Schools also varied in the extent to which the contact teacher was able to secure ‘buy-in’ for the survey from other teaching staff, and this became increasingly difficult in the run up to examinations.

In interviews (see Chapter 3, Process evaluation), teachers highlighted logistical issues with the evaluation, referring to it as ‘a lot of work’ and ‘a bind’, particularly around organising the computer rooms. This was particularly so when schools felt that they were ‘quite limited in [. . .] resources when it comes to computers’. For this reason, teachers in two schools (schools 3 and 6) suggested that paper questionnaires would be more manageable. Another teacher (school 5) agreed that ‘a lot of organisation was needed’ to co-ordinate questionnaire sessions, but that they had all gone ‘according to plan’.

We used 53 paper questionnaires due to various logistical or technological difficulties. It was notable that data quality in the paper versions was lower. There were 62 instances of non-adherence to filtering or question instructions (skip patterns and prompts prevented these in the web survey).

Confidentiality

Adherence to examination conditions for survey completion varied between schools and classes. Fieldworkers tended to allow some general chat, unless they picked up that students were influencing or observing the responses of others. Fieldworkers reported that, owing to the layout of the computing suites, there was often little space between students. In addition, fieldworkers were required to walk around the room to assist students with queries or technical difficulties. This meant that privacy was not as easy to ensure as with a paper questionnaire (in which students can cover their answers with an arm).

Group interviews with friends of PSs (see Chapter 3) touched on the acceptability of questionnaire completion. Some students raised concerns about privacy arising from the potential visibility to fieldworkers, teachers and other students:

These students were not deterred from answering questions, which may have been in part because of confidence in the confidentiality of the survey:

As long as your name’s not gonna go anywhere [on it] it doesn’t really matter.

At the end of all the questionnaires, participants were asked ‘Is there anything at all you would like to tell us about the STASH project or this questionnaire?’ and the responses were coded post hoc. Of 1979 completed questionnaires, six participants made comments on confidentiality in the open-ended ‘additional comments’ section, including ‘Nothing is stopping anyone from looking at my screen’ and ‘let the pupils complete this survey in a more private manner so they don’t feel embarrassed or lie about their answers because others can see’.

Questionnaire comprehension and sensitivity

Most students interviewed for the process evaluation said that they thought the questionnaire was ‘pretty straightforward’ and ‘easy to understand’. Teachers were able to comment across a broader range of students, with one teacher (school 4) highlighting that the questionnaire was difficult for students with literacy issues:

There was a couple that we had to sit with ’cause their English isn’t great, which probably wasn’t the best environment to do it with, sitting in front of the class, ‘cause you were having to read out things that they possibly didn’t want to share in front of everybody.

Fieldworkers also noted that issues of literacy, English as a second language and learning support limited participation for a few students (schools 2–4).

The nature of the ‘quite personal questions’ raised concerns for some students, primarily focused on sexual behaviour questions. Concerns were also raised in relation to the SNA friendship module, in particular the fact that names of students appeared from a pre-populated class list from which respondents could select their friends. This was done to minimise errors in naming students but was perceived by students as ‘weird’:

This sense of the unexpected was reiterated in interviews with teachers. One teacher said that some students ‘were a bit taken aback by some of the questions’, but attributed this to the fact that ‘they’ve never had that question put to them before’ and that they, nevertheless, ‘got their head down and got on with answering’ them (school 5, DHT). Another teacher commented that some students may have felt ‘timid, or a bit shy’ about the questions, but that the boys they supervised were ‘quite confident and wouldn’t be shy about those kind of things. So they filled it in . . . fine without any issues’ [school 6, contact teacher (CT)].

In the open-ended ‘additional comments’ question that concluded the questionnaire, there were 15 positive responses, mostly brief comments on having enjoyed the experience, such as ‘I enjoyed this’, ‘it was good’ and ‘it was interesting’. Negative comments were more numerous. Eleven comments were general in nature (e.g. ‘pointless’ and ‘weird’). Thirty comments expressed discomfort with the personal nature of the questions, for example ‘This is none of your business’, ‘It’s very personal’ and ‘. . . some questions are uncomfortable to answer’. Five comments were made relating to the questionnaire being insufficiently inclusive, noting for example the equivalence of ‘penis in a vagina’ with intercourse and a lack of options for those identifying as asexual. Five comments expressed concern about the inputting of postcodes (for calculation of the SIMD category) and friend names, querying whether or not the questionnaire could really be anonymous. Six students commented that the questionnaire was too long or that they needed more time to complete it. Ten queries or suggestions were made that were specific to questions (e.g. more ‘not relevant’ options or use of simpler language).

Fieldworkers reported some of the common queries they received. These included requests for meaning of some words (optimistic, self-esteem), queries about why they were being asked to give their postcode and some confusion about the question on data linkage.

Response rates at control, baseline and follow-up

Table 5 outlines the response rate for the control group, intervention group baseline and intervention group follow-up.

The target response rate was met with ease in all but one school (school 4). This school differed from the other schools in being much smaller, with the highest proportion of students eligible for FSM.

It became apparent that class lists (which provided the denominator) were not always up to date, as they were often received a few weeks into the school term. This meant that a school leaver might appear on the class list and a new student might be missing. Contact teachers explained that there are students who appear on class lists, but do not attend school on a regular basis, either because of truancy or because they are engaged with other local authority education or social services. Comparing Tables 4 and 5, it is notable that the three schools with indices indicative of higher socioeconomic deprivation tended to have response rates below the average across schools. This pattern is possibly explained by a higher proportion of non-attenders in these schools.

Loss to follow-up in intervention group: differences in intervention students who did and did not complete the follow-up questionnaire

Across all schools, 81.2% of students completing the baseline survey also completed the follow-up. Six parents opted for their child to leave the study (three in the control group and three in the intervention group) and three students declined to complete the questionnaire.

Table 6 shows differences in sociodemographic and educational profile, and sexual experience between students who did and did not complete the follow-up survey. As only three students opted out of the survey, completion at both time points primarily reflects the extent to which students were in attendance and available to complete the questionnaire. Young women were more likely to complete the follow-up questionnaire than young men. Students from lower socioeconomic backgrounds were less likely to complete the follow-up questionnaire and this was evident for all measures of socioeconomic status: SIMD, FSM eligibility and home ownership. Educational attainment also differed between the groups. Those followed up were more likely to be doing National 5 qualifications (Scottish equivalent of English General Certificate of Secondary Education) than a mix of National 5 and lower-level qualifications, or just lower-level qualifications. They were also less likely to be leaving school early [in Scotland students have the option to leave in S4 after they reach 16 years, or at the end of secondary 5 (S5) or secondary 6 (S6)].

There were also differences in sexual experience between those who did and did not complete follow-up. Students who did not complete follow-up were more likely to have experienced both oral sex and vaginal sex at baseline than those who were followed up.

Sample characteristics of intervention, control and follow-up groups

Table 7 shows the baseline characteristics of the sample, split by study group. The primary comparison of interest is between follow-up and control, as these are the study groups that we mainly wish to compare. There was an imbalance in gender: a higher proportion of female students in the follow-up group than in the control group. There were no marked differences between the follow-up group and the control group for any other baseline characteristics.