Process evaluation

Introduction

Why is process evaluation necessary in the PACE-UP trial?

The PACE-UP RCT is a complex intervention comprising multiple interacting components (pedometer, handbook, diary, practice nurse PA consultations and BCTs – as part of both written materials and consultations). Although the RCT design is able to establish the effectiveness of the intervention (see Chapter 3), it does not provide information on how it works, or whether or not there are contextual factors that could be associated with variation in outcome in different settings.169 Conducting a process evaluation of the PACE-UP trial enables a detailed examination of the mechanisms of change by gaining an understanding of how the intervention was delivered and received and how this may have affected the variation in outcomes. The process evaluation investigates the relationship between the fidelity and quality of implementation, the context of the intervention and the main trial outcomes. The evaluation also helps to draw conclusions on the replicability and generalisability of the intervention. The main findings from the process evaluation are published170 and are reproduced here under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0).

Medical Research Council’s guidance on process evaluations in complex interventions

In 2014, the Medical Research Council (MRC) published new guidance for process evaluations of complex interventions.169 The guidance draws on the experiences of researchers and wider stakeholders who have conducted process evaluations within trials of complex public health interventions. We have used the guidance to provide the framework for the process evaluation of the PACE-UP trial. Process evaluation is accomplished through investigating aspects such as implementation, mechanisms of impact and context, and the relations between these, as described in Figure 11.

FIGURE 11 The key functions of process evaluation and the relationships between these. Green boxes represent the components of process evaluation, which are informed by the causal assumptions of the intervention, and inform the interpretation of outcomes. Adapted from Moore et al.169 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)

Implementation refers to the structures, resources and methods through which delivery is realised, and comprises the following factors: implementation process, reach, fidelity, dose and adaptations. The implementation process describes how the delivery is achieved, through training, support, resources, etc. Reach refers to coverage and the degree to which the intervention is delivered to those for whom it was intended, that is, who receives the intervention. The other aspects of implementation are related to what is delivered. Fidelity is the degree to which the intervention was delivered as intended (content) and includes assessment of the quality of the intervention. Dose denotes the quantity of the intervention implemented. Adaptations are participant and implementer adjustments, which may impede or strengthen the intervention and which arise in response to the intervention itself.

Mechanisms of impact refer to how the intervention activities and participants’ responses to them cause change and adaptations.

Context refers to external factors which may influence, and be influenced by, implementation mechanisms and outcomes.

The first stage of designing the process evaluation was to describe the intervention and to clarify casual assumptions, which was accomplished through the use of a logic model, shown in Figure 12.

FIGURE 12 Logic model for the PACE-UP (Pedometers and Consultation Evaluation – UP) PA trial. (Figure not reproduced here; see the original article.)

Methods and results

The PACE–UP trial process evaluation was conducted alongside the effectiveness evaluation, included both qualitative and quantitative components and was undertaken by the same team that carried out the effectiveness evaluation.

In accordance with the MRC guidance, the methods were selected through following the key functions model (see Figure 11) and are summarised in Table 16, which details the process evaluation components, the data sources, the trial group to which they refer and the measures used. The nurse-supported intervention was the most complex to deliver as it involved eight nurses from seven practices delivering three consultations over a 3-month period. Most of the process evaluation was therefore designed to evaluate the nurse-supported intervention group. When process evaluation occurred for other groups, this is clearly described.

To reduce duplication, and for ease of reading, the methods and results for each aspect of the process evaluation are presented together. The main results are summarised in a further key functions model after all of the results have been presented (Figure 13). Several aspects of the process evaluation are dealt with appropriately in other chapters of this report, such as reach in Chapter 5 and participant responsiveness in Chapter 7. They are referred to in Table 16 for completeness and the relevant chapter in which the methods and results for this aspect are presented are clearly shown.

FIGURE 13 The key functions of process evaluation and the relationships between them for the PACE-UP trial. Green boxes represent components of process evaluation, which are informed by the causal assumptions of the intervention and inform the interpretation of outcomes. Adapted from Moore et al.169 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. (Figure not reproduced here; see the original article.)

We have selected three quantitative aspects of the process evaluation to relate directly to PA outcome measurements at 12 months (change in step counts and change in time in MVPA in bouts): the number of nurse appointments attended whether or not completed step count diaries were returned by participants at 3 months in the nurse-support and postal delivery groups, and the use of pedometers in the nurse-support and postal delivery groups during the 12-week intervention. These analyses relating process to outcome measures are described at the end of Methods and results.

Implementation

Implementation process

Training

Nurses delivering the intervention were provided with training in PA guidance (1 hour and 25 minutes), trial protocols (4 hours), safety reporting (1 hour and 10 minutes) and BCTs (9 hours and 25 minutes) across the duration of the trial (see Appendix 5, Table 38). Data on nurse training were obtained from training-day agendas and BCT trainer telephone feedback records and trial administrative records. The total training time was approximately 16 hours; most of this was allocated to delivering BCTs as the active ingredient in the intervention.

Resources

Resources for the trial include the trial materials [patient handbooks and diaries, see the NIHR Journals Library website: www.journalslibrary.nihr.ac.uk/programmes/hta/103202/#/ (accessed 25 May 2018)], trial equipment (pedometers and accelerometers) and payment of nurse time and room hire. These are all fully costed in Chapter 4 and are not further commented on in this chapter.

Reach

The overall trial recruitment rate was 10% (1023/10,467). Details on how practices and participants were selected and recruited is described in Chapter 2. The methods for assessing trial representativeness and generalisability, by comparison of non-responders, non-participants and participants, and by interviews of non-participants, are described in Chapter 5.

Fidelity (content and quality)

Content

Nurse sessions

Nurse session attendance and session content delivered was recorded by the nurses after each session (see Appendix 5). There were 11 compulsory items to be covered in session 1 and six items to be covered in sessions 2 and 3. The level of nurse session attendance was high; approximately three-quarters of participants attended all three sessions (n = 255/346; 74%). Adherence to content delivered was high in all sessions; the mean number of items delivered in each session was 11 (range 10–11) and six (range 5–6) in sessions 2 and 3, respectively. Of those participants who attended session 3, most reported still using the pedometer and diary (n = 258/263; 98%) (Appendix 5, Table 39).

Physical activity diaries

Physical activity diaries (see Appendix 5, Table 40) returned by participants after the intervention provided data on the achievement of weekly walking targets for the intervention groups. Eighty per cent of participants (n = 549) returned completed diaries; there was similar return across both groups. One-third of participants in the nurse-support group altered their step count targets (89/346; 32%) and the majority were decreased (n = 80). In comparison, just four participants in the postal delivery group altered their step count targets and all were decreased. The relationship between diary return at 3 months and the trial outcome measures was explored (see the association between the process evaluation measures and the trial outcomes at the end of Methods and results).

Pedometer use

All participants were asked about their pedometer use during the 12-week intervention period (see Appendix 5, Table 41). During the 12-week intervention, a high proportion of both the postal delivery and nurse-support groups reported using their pedometer either every day or most days: 238 out of 294 (81%) in the postal delivery group and 269 out of 303 (89%) in the nurse-support group. The relationships between pedometer use during the intervention and the trial PA outcomes at 3 and 12 months were explored for both intervention groups (see the association between the process evaluation measures and the trial outcomes at the end of Methods and results).

Quality

Nurse and patient alliance questionnaire

Following the intervention, both the nurse and the participant independently completed a 12-item nurse and patient alliance questionnaire (see Appendix 5) covering different intervention aspects (e.g. working together and goal-setting, number of appointments). The questionnaires were developed by BCT trainers and trial investigators, and questions 1–11 were adapted from the Working Alliance Inventory,171,172 which is a validated alliance measure used in cognitive–behavioural therapy-based studies, and the Outcome Rating Scale.173 Item 12 was added to specifically ask about the number of PACE-UP trial appointments. The questionnaire was posted to the participant and returned to the researcher, so that the nurse was blind to participant responses. Three directly comparable items (questions 1, 3 and 4) from both the patient questionnaire and the nurse questionnaire provided data on the quality of intervention delivery (Table 17). Seven directly comparable items directly relate to participant responsiveness (questions 5, 6, 8, 9, 10, 11 and 12). Two items (questions 2 and 7) were discounted as they did not relate to quality of participant responsiveness.

The questionnaires were completed by 295 out of 346 participants (85%) in the nurse-supported intervention group and by the nurses for 251 out of 346 nurse-support group participants (73%).

There was strong agreement between the participant and nurse results for all of the items relating to quality and 80% or more of both nurses and participants agreed or strongly agreed with all of these statements, suggesting high quality of delivery.

The following are some examples of participant and nurse comments relating to quality from the questionnaires:

Nurse was encouraging, supportive. Encouraged me to set goals that were achievable for me and not to put too much pressure on myself.

My nurse was lovely and encouraged me all the way through, even when some days I couldn’t do what I needed, we discussed alternatives. My nurse was a diamond. Thanks to PACE-UP and the nurse my walking has really improved.

Client pleased with programme. Learning curve. Would recommend to others.

Enjoyed patient, good discussions and understanding around increasing exercise.

Audio-recordings from nurse sessions

Nurses were asked to audio-record a sample of their sessions so that these could be listened to by the BCT trainer and rated according to their skill in six different communication skill competencies. Ratings were made by the BCT trainer against a primary care consultation rating scale (range 0–6) in six domains (Figure 14) and used for both fidelity (quality) evaluation and supervision purposes. The rating scale used was developed from the Cognitive Behavioural Therapy Techniques for Palliative Care Practitioners Rating Scale174 and the Department of Health and Social Care’s The Ten Essential Shared Capabilities – A Framework for the Whole of the Mental Health Workforce.175 The nurses were each asked to provide three audio-recorded sessions, one each for sessions 1, 2 and 3. They were asked to try to ensure that one of the recorded sessions was from a session where a couple were seen together. The mean scores and ranges for all nurses are shown across all domains in Table 18.

FIGURE 14 Behaviour change technique competency level. (Figure not reproduced here; see the original article.)

The range of scores illustrates that even the lowest ratings were competent, and the highest scores were expert, across all six competencies. The lowest scoring competency was given for empowering explanations (mean score 4.7), whereas all the other competencies had mean scores above 5, demonstrating proficiency, with very good features.

Qualitative perspective

Semistructured individual interviews with participants and focus groups with nurses provided a qualitative perspective of the intervention, including the quality of delivery; this is presented in detail in Chapter 7. Overall, the nurses and participants described the intervention in a positive manner, as highlighted by the following quotations:

. . . they kept saying how well I was doing, and all this sort of thing, so it made me want to continue. I think it was . . . a part motivation, yes, because I knew I had to face somebody and I didn’t want to fail.

. . . if you had, in your drawer, you had like a set . . . a package, programme, you could do, and if through the NHS Health Check you identified someone who was suitable, you could then discuss it with them and say, ‘Would this be something you’d be wanting to look at? . . . and go from there.

Dose

For the purpose of the PACE-UP trial, the dose delivered to the postal delivery group was fixed, as they all received the same handbook, diary and pedometer. The dose could vary for the nurse group according to the number of sessions attended and the length of each of the sessions. Nurses were required to complete checklists at the end of each session (see Appendix 5, Table 39), providing details on attendance and the duration of sessions. The duration of sessions was also captured from the audio-recorded sample of intervention sessions; this allowed a comparison with session durations calculated from nurse checklists. Overall, three-quarters of participants in the nurse-support group attended all three sessions. Ninety-five per cent of participants (330/346) attended session 1, 86% (296/346) attended session 2 and 76% (263/346) attended session 3. The relationship between the number of sessions attended and trial PA outcomes was also explored (see the association between process evaluation measures and trial outcomes at the end of Methods and results).

Trial protocols detailed the following approximate duration of each nurse intervention session: 30 minutes for session 1 and 20 minutes each for sessions 2 and 3. A summary of nurse intervention session durations from nurse self-report and audio-recordings is available in Appendix 5, Table 42 (19 recordings, relating to 22 participants, as some were couples). There was good agreement between the planned protocol session length and the nurse self-report durations: mean of 30 minutes (SD 4 minutes) for session 1, mean of 24 minutes (SD 3 minutes) for session 2 and mean of 22 minutes (SD 4 minutes) for session 3. The duration of consultations from audio-recordings was based on much smaller numbers (n = 22 participants) but had a shorter mean duration: mean of 21 minutes (SD 6 minutes) for session 1, mean of 21 minutes (SD 7 minutes) for session 2 and mean of 14 minutes (SD 5 minutes) for session 3.

Adaptations

Details of nurse and participant adaptations made to the intervention were provided from nurse training session records. There were many examples presented relating to step count target adaptation and tailoring the intervention to individual circumstances. Adjustments were made to the intervention to accommodate religious observances, such as Ramadan and Christmas. Step count targets were adapted to be more achievable to reflect participants’ reduced energy/activity levels in advance of holidays, when there were expected to be reductions or increases in PA, during periods of participant illness and pain and in response to changing weather conditions. Nurses also explained the need for flexibility with participants who experienced difficulties with equipment use; for example, a small number of participants who did not like using the pedometer were advised to use time to measure their walking, rather than measuring step count (e.g. extra walks of 10–15 minutes per day, rather than an extra 1000–1500 steps per day). At the second training session, it became clear that the nurses did not find the optional handouts provided for use in consultations to supplement the patient handbook helpful and, as a consequence, were not using them. From these discussions, it was decided that these materials would be discontinued. Another adaptation revealed by the nurses was adapting targets and advice for participants taking part as a couple, particularly if they had very different PA levels and targets; this sometimes related to one individual of the couple having health problems that had an impact on mobility. Nurses adapted the intervention to encourage both participants to meet their targets, often encouraging a mixture of walking together and walking apart to achieve this.

Physical activity diaries also provided data on adaptions to the prescribed intervention through alterations of participants’ walking targets (see Appendix 5, Table 40). Few participants in the postal intervention group altered their targets in the diary (1%; 4/339), whereas 32% (89/346) of the nurse-supported intervention group altered their walking targets, mainly by decreasing the target [29% (80/346)].

Additional details of nurse and participant adaptations to the intervention by both intervention groups, obtained as part of the qualitative evaluations of the trial, are presented in Chapter 7. Some examples are given here.

The nurse quotations below illustrate the flexibility in intervention delivery during Ramadan and also during bad weather:

. . . they couldn’t walk or increase on their walking at that time because they hadn’t eaten and then they weren’t feeling too good, and all that, so we did it a different way then, and what I did with them was we relaxed it and then I said to them, when Ramadan’s over, we made the appointment so that their actual trial went on a bit longer.

. . . if the weather was bad, or it was cold, or there were obstacles that got in the way . . . they would do things like, you know . . . activities indoors where they could not always go outside.

Mechanisms of impact

Participant responsiveness

Seven items reflecting participant responsiveness were identified in both the patient alliance questionnaire and the nurse alliance questionnaire (see Table 17). There was a good degree of consistency between participant and nurse responses for all of the items relating to participant responsiveness and high levels of agreement with all of the statements. For example, 90% of participants said that the pedometer was helpful and 83% said that the number of nurse appointments was just right, suggesting that there was a good level of participant responsiveness to the intervention.

Some examples of both participant and nurse comments relating to participant responsiveness from the alliance questionnaires are shown below:

PACE-UP has changed my life. I use the car less when I go about. Although I drive to work I park about 1 km away from work, then walk all the way to and from.

Wearing the pedometer really raised my awareness of how far I walked each day. I will continue to use it.

Positive, liked study book. Was a very reflective/honest person regarding his exercise. Positives/negatives easily identified by patient.

Patient has own excel monitor of readings/pedometer count. Also converts to miles daily.

Information about participant responsiveness also came from the qualitative evaluations of participants from both the postal delivery and the nurse groups (from individual interviews) and from practice nurses (from a focus group and individual interviews), and is presented in Chapter 7.

The following quotations from the qualitative evaluation illustrate participant responsiveness and engagement from a nurse and participant perspective:

. . . the only other thing I’d say about the diary is that the people that really liked filling it in found it a really good motivator. When they came to the last appointment, they wanted another one.

There’s nothing like the fact that you know you’re going to meeting someone and talk about it to make you do it, you know, . . . It’s basically the routine of being checked up on by someone else . . .

. . . well having something which counts the steps makes one conscious of it and filling out a little booklet every day, likewise, it just creates some personal pressure.

Context

Comments made by nurses during training sessions relevant to contextual factors were noted down. There was overlap with the factors mentioned in the section on adaptation of the intervention, as contextual factors often required the nurses to consider adapting the intervention or targets after discussion with participants. Examples of contextual factors mentioned are as follows: the difficulty of walking in bad weather, the effect of taking part in the intervention as a couple, health issues that required a slower, more gradual approach and undertaking the intervention during Ramadan or Christmas.

How contextual factors may have affected (and been affected by) the implementation, such as season, environment, health status, employment and social and religious factors, were explored as part of the qualitative evaluation of participants (from individual interviews) and nurse perspectives (from focus group and individual interviews), and are described more fully in Chapter 7. The following factors were described: season/weather (problems with rain or snow and ice), environment (ease of walking in parks, more difficult in built-up areas), health issues (examples both of pain getting worse with walking and of walking improving pain), employment (retired people saying they have increased time for walking, some occupations providing the opportunity for walking or at least walking to work), religious factors (difficulty with walking during Ramadan when fasting and having low energy levels) and social factors (walking with family/grandchildren, etc. or not having anyone to walk with). This last factor is illustrated by the following quotation taken from the qualitative evaluation:

. . . it’s something I want to keep up, because I just felt that it was such a benefit, and even the kids would come out with me sometimes.

Association between process evaluation measures and trial outcome measures

Although the trial was powered only for analysis of the difference in outcome measures between the three groups, and not for exploration of the effect of process evaluation measures, we felt that it was interesting to explore if there was any relationship between adherence to the intervention and the change in outcomes. We have focused on three quantitative measures of process evaluation in relation to the PA outcome measures at 3 months and 12 months (changes in average daily step count and weekly time in MVPA in bouts). The three measures were all to do with the implementation of the intervention:

1. Dose – nurse session attendance (0, 1, 2 or 3 sessions attended; nurse-support group).
2. Fidelity – return of completed diaries after the 3-month intervention (yes/no; postal delivery and nurse-support groups).
3. Fidelity – pedometer use – how often did you wear the pedometer? Every day or most days (yes/no; postal delivery and nurse-support groups) during the 12-week intervention (0–3 months).

All measures described were considered as independent variables in the models, with (1) change in average daily step count and (2) change in total weekly MVPA in bouts as outcomes. All analyses were adjusted for age, sex, practice, month of baseline accelerometry, household identifier (to account for clustering by household) and trial group (as in the main trial analyses; see Chapter 2). Table 19 shows the results of the models.

Nurse session attendance and physical activity outcomes

In the nurse group at 3 months and 12 months there was a positive association between the number of sessions attended and the PA outcomes. Participants attending all three sessions increased their step count and their time in MVPA in bouts at 3 and 12 months by significantly more than those attending between 0 and 2 sessions.

Diary return and physical activity outcomes

In the postal delivery group, there was a strong positive association between returning a diary and on both change in steps and MVPA in bouts at both 3 and 12 months compared with those in the postal delivery group, who did not return a diary. In the nurse-support group, there was a positive association between returning the diary and on change in step count and MVPA at 3 months, compared with those in the nurse-support group who did not return a diary. However, by 12 months, there was no significant association for either PA outcome of returning a diary within the nurse-support group.

Pedometer use and physical activity outcomes

In the postal delivery group, reported use of a pedometer every day or most days during the 12-week intervention period was associated with a significant change in step count at 3 months and 12 months, and with change in MVPA at 3 months (borderline effect at 12 months).

Within the nurse-support group, there were no significant associations between regular pedometer use during the 12-week intervention and change in step count or MVPA at 3 months and 12 months. This lack of significant effect could be explained by the very small numbers of participants in the nurse-support group who reported not having used a pedometer regularly during the 12-week intervention (n = 34; 11%).

Overall, the analysis of the association between process measures and PA trial outcomes exhibits a clear pattern of positive associations (i.e. increased nurse appointments, diary return and pedometer use were associated with increased objective PA levels). This provides clear evidence of the engagement with the trial process and outcomes, but cannot be interpreted as causality.

Discussion

Main findings

Figure 13 summarises the key findings from the PACE-UP trial process evaluation, which followed the MRC guidance for process evaluation of complex interventions.169 We gathered a number of positive data on implementation, suggesting good-quality intervention delivery and adherence to the protocol, despite the low reach of the trial. Nurse training was an important element of the trial, with nurses receiving approximately 16 hours of training, predominantly around delivery of BCTs. We demonstrated good coverage of the proposed session content by the nurses and also good-quality delivery, with the audio-recording of nurse sessions demonstrating high levels of competency in communication skills. High-quality delivery was also reflected in comments from participants who felt heard, understood and respected. Three-quarters of the nurse group attended all three PA consultations, and around 80% of both the nurse-support and the postal delivery groups engaged with the self-monitoring aspects of the trial and returned completed step count diaries at 3 months. In terms of the mechanisms of impact, we demonstrated high levels of participant responsiveness. Context was important and factors affecting the implementation of the walking intervention were suggested by nurses and participants from both intervention arms, and included the effects of weather, the environment, health issues, pain, employment, observance of religious events and social factors. The nurses worked with participants to help them to make adaptations, either to the intervention or their individual targets, or to encourage participants to come up with solutions when possible, when there were contextual challenges. Several process evaluation measures (number of nurse sessions attended, return of a completed diary and regular pedometer use) showed significant associations with PA outcomes at 3 and 12 months.

Strengths and limitations

Strengths

Despite the PACE-UP trial process evaluation having been designed before the publication of the MRC guidance,169 we were able to use the data that we collected and fit them into the framework, which has provided a useful structure for reporting process evaluation findings. We have a number of different data sources that reflect three different perspectives in the trial: participant, practitioner and observer. This provided a broader picture of the process than many studies have reported and allowed comparison of results from different methods (e.g. for consultation duration). We tried to reduce the burden of participant measurement, trial costs and duplication of effort, by collecting as many routine trial data as we could for the evaluation (from administrative records, nurse training records, nurse checklists, nurse audio-recordings from supervision sessions, etc.), but we supplemented this with data collected specifically for the evaluation (e.g. the nurse and patient alliance questionnaires or the 12-month pedometer use questionnaire). We have used a mixed-methods approach to the process evaluation, as recommended, combining quantitative data from key process variables from all participants with in-depth qualitative data from purposively selected samples.169 The qualitative element is described in full in Chapter 7; however, we have used relevant quotations in this chapter to illustrate the quality of delivery, adaptations, participant responsiveness and context, and these have provided a voice to participants and nurses and added a richness and depth to the evaluation. The data were collected longitudinally and contemporaneously throughout the trial, which is seen to be the most complete and accurate method of data collection, and also allows any change in intervention delivery over the course of the trial to be detected.169 The data are comprehensive, with a high response rate and completeness of data sources, strengthening the robustness of the findings. The process evaluation analysis was conducted before the outcome analysis to avoid a biased interpretation of the process data. Only the final analyses examining the effects of process evaluation variables on outcome data were carried out following the main outcome analyses.

Limitations

The process evaluation was conducted by the trial team while the trial was ongoing. This allowed efficient data collection in a contemporaneous manner, but could have led to bias in evaluation. We tried to minimise the bias by using objective instruments when possible (e.g. nurse and patient alliance questionnaires, 12-month pedometer use questionnaires, return of patient diaries). In addition, the qualitative evaluation was led by Christina Victor, who was not involved in the day-to-day trial conduct. Some process measures were not filled out by everyone (patient alliance questionnaire: 85% completion; nurse alliance questionnaire: 73% completion); this could have led to a more positive assessment of statements, if those who felt negatively about the programme did not reply. Nurses could have selected more positive consultations to audio-record, inflating the BCT competency levels, although they were encouraged to record cases as they occurred. Not all of the nurses submitted audio-recorded consultations with couples, meaning that we were unable to look separately at the quality of these consultations. The content delivery for the nurse consultations was evaluated from checklists filled out by the nurses because they may have overestimated what they had achieved. We tried to compensate for this by collecting additional data from both the participant and observer perspective to help to corroborate these data; both participant and observer data suggested a high degree of quality in the consultations. The consultation durations were shorter from observer data than from self-report, but data on observed consultations were based on a much smaller sample. The study was not powered to look at the effects of adherence to different aspects of the protocol on trial outcomes; we therefore have reduced power for these analyses, which limits the interpretation of the findings, which cannot be taken to be casual.

Comparison with other complex intervention process evaluations

A number of studies have examined intervention implementation fidelity, with a large variety of process structures and methods; therefore, it is difficult to draw direct comparisons. Process evaluations have become increasingly important, but the purposes and design of studies have been mixed. Many process evaluations are completed independently of trial data collection and are observations of a random subsample of participants or practitioners.176,177 The process evaluation of the PACE-UP intervention provides both participant and nurse perspectives and identifies a link between contextual factors and adaptations in intervention delivery and acceptance. The study allowed us to look at both perceived and observed behaviour change in the nurse intervention delivery and participant responsiveness; this is unlike many other studies, which have tended to focus on only one perspective, which is most often the person delivering the intervention.176,178–181 Nurse comments collected at training, individual session checklists and nurse and participant comments from the qualitative work illustrate the intervention delivery and adaptations in context. The intervention was designed to bring about change at an individual level when delivered in a primary care environment; we have observed that context influences the delivery and implementation of the intervention through adaptation. This is similar to findings from other PA and dietary complex intervention studies with process evaluation.182,183 Specifically, Fitzgerald et al.182 identified that negotiation and flexibility play an integral role in overcoming the barriers and resistance to change in a dietary intervention. Previous studies have collected data at an organisational or practice level;176,181 there are few studies that have captured the evaluation of behaviour change at an individual level and from two perspectives. A previous study that did look at both patient and practitioner perspectives, however, reported much greater variations in dose and adherence to protocols than seen in the PACE-UP trial, therefore making it difficult to establish which elements of the intervention were effective.184 Berendsen et al.184 reported that many health-care professionals deviated from the protocol of a lifestyle intervention to accommodate individuals and reduce fallout, which was associated with increased patient satisfaction for the intervention sessions. This perhaps suggests that adaptations and tailoring of an intervention have a strong influence on retention, adherence and, possibly, effectiveness in lifestyle and behaviour change interventions. The PACE-UP intervention allowed nurses to adapt the sessions as necessary to each individual, while maintaining the key deliverables in each nurse-led session; although we have not looked at the effect of adaptation on retention, we have seen that dose (nurse session attendance) was associated with effectiveness of the intervention. This, in turn, promotes the consideration of building adaptations and flexibility into intervention design. Our finding of an association between the return of a completed step count diary and a change in PA outcomes is consistent with the findings of a systematic review,31 which suggested that the use of a step count diary was common to many successful pedometer interventions.

Implications of the process evaluation for the interpretation of the PACE-UP trial

We have demonstrated that the PACE-UP trial had good adherence to the protocol, the intervention was acceptable and was rated positively by both the nurse-support group participants and the postal delivery group participants, and both groups engaged in self-monitoring using the pedometer and step count diary. It is not possible to infer causality directly from the process evaluation data, but the high level of engagement with pedometers and diaries by both intervention groups suggests that these were important factors in helping people to make the PA changes observed. This is supported by the associations demonstrated between increased PA levels and the following process measures: number of consultations attended, return of a completed step count diary and pedometer use. The careful description and documentation of the trial processes, the collection of additional data for the process evaluation and the publication of the resources used as appendices mean that our intervention and process evaluation would be easy for others to replicate, from training through delivery, to follow-up and evaluation. Use of the MRC framework gave a logical and coherent structure for reporting,169 which is also easy for others to follow. The PACE-UP trial had a positive and significant effect on PA outcomes, but had this not been the case, the positive process evaluation with high levels of fidelity would have enabled us to have confidence that any negative trial effect would not have been because of poor trial implementation. The trial demonstrated a stronger effect on the main PA outcomes and on exercise self-efficacy at 3 months in the nurse-support group than in the postal delivery group, although the effects on PA outcomes were similar between the groups at 12 months (self-efficacy remained higher at 12 months in the nurse-support group). The process evaluation demonstrated that the nurses were delivering BCTs in their PA consultations in accordance with the protocol and with high levels of competence (in addition to the BCTs provided in the handbook and diary for both groups). This suggests that the nurse-delivered BCT elements of the intervention have strong short-term effects on PA levels (and, possibly, longer-term effects on self-efficacy). The possible effects on longer-term maintenance are examined in Chapter 8. The implications of the low reach of the trial for generalisability and the public health impact have been discussed in Chapter 5 and will be considered further in the main discussion in Chapter 9. The process evaluation demonstrated important contextual factors that had an impact on participants’ ability to engage with a walking intervention; these should be considered in any future roll-out of the programme, particularly regarding how the programme may need adaptations to be made in these circumstances.

Conclusion

The PACE-UP trial process evaluation demonstrated that the trial was well delivered by the trial team and well received by participants. The MRC framework was a useful vehicle for reporting the evaluation. An association between adherence to the trial protocol and main trial PA outcomes has been demonstrated. Important contextual factors were shown that may need adaptations to be considered in any roll-out of the intervention.