Description of individual participant data, data sets and risk of bias

Overview

In this chapter, we report the systematic review-informed development of the RELEASE database, the data sources and the data sets identified that were unavailable to us. We consider the design of the contributed data sets, the participant demographics, the language measures used, the risk of bias in the primary data sets, the data transformation and synthesis approaches, and risk of meta-biases.

Results of the systematic search

Our search strategy identified 11,314 records; once duplicates were removed, we screened 5256 records for eligibility and excluded 2935 (Figure 3). After reviewing 2341 abstracts, 1210 were excluded and 1131 full texts were retrieved. The primary research teams that were linked to 698 references identified (including trial registrations), which appeared eligible for inclusion, were invited to confirm the eligibility and availability of the data set described in the identified report. We received 125 expressions of interest (see Figure 3).

FIGURE 3 The PRISMA flow diagram indicating the search, identification, screening, exclusions, invitations to contribute and final data set. Reproduced with permission from Williams et al.112 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original figure. (Figure not reproduced here; see the original article.)

We created a RELEASE study database of 174 data sets comprising 5928 IPD. Electronic data set contributions included 3940 out of 5928 IPD (66.5%) gathered in 75 out of 174 data sets (43.1%). A total of 1988 out of 5928 IPD (33.5%) appeared in 99 out of 174 (56.9%) eligible data sets reported in the public domain, and these were also included in our database (see Figure 3). Ineligible IPD (e.g. people with aphasia following traumatic brain injury) from mixed study population groups were excluded.

Excluded data sets

We excluded 433, out of 1131, full texts that failed to meet our eligibility criteria; 77 data sets had fewer than 10 IPD; four data sets had qualitative data only; 272 data sets were not about aphasia, stroke or language impairment; and 52 were reviews, secondary data analysis or commentary pieces (see Figure 3).

Unavailable data sets

We were unable to confirm whether or not the 267 remaining data sets identified as potentially eligible (reported across 399 records) met the RELEASE study inclusion criteria. Following careful review, it is possible that 53 were RCTs and 214 were a non-RCT design (see Figure 3). Of the 53 RCTs, 46 included, but did not necessarily evaluate, a SLT intervention.

Included data sets

Source

The date of participant recruitment was available for 79 out of 174 (45.4%) primary research studies (Table 2). These data sets reflected participants recruited from 1973 to 2016, although most were recruited since 2000 [67/79 (84.8%)]. Recruitment dates were unavailable for 95 out of 174 data sets (54.6%), particularly those from the public domain. We considered date of publication as a surrogate indicator of data set age [161/174 data sets (92.5%); 4627/5928 IPD (78%)]. Thirteen data sets did not have an associated publication, including recently completed research or clinical registries. Of the 161 data sets that were published between 1973 and 2017, most [135/161 (77.6%)] were published since 2000.

Data set design

One-quarter of the primary research data sets included were RCTs [47/174 data sets (27.0%); 1778/5928 IPD (30.0%)]. The remainder were case series or cohort studies [104/174 data sets (59.8%); 2886/5928 IPD (48.7%)] and non-RCTs [18/174 data sets (10.3%); 411/5928 IPD (6.9%)]. Five data sets were derived from registries [5/174 data sets (2.9%); 853/5928 IPD (1.4%)]. Approximately half the data sets included a SLT intervention [91/174 data sets (52.3%); 2834/5928 IPD (47.8%)], whereas the remainder were non-interventional in nature.

In 20 data sets (11.5%; 311 IPD), participants crossed over from one intervention to another. An additional five data sets (2.9%; 240 IPD) delayed the start of treatment for a group. Seven data sets (4%; 128 IPD) allocated participants across three arms, and three data sets (1.7%; 84 IPD) allocated across four arms.

Some participants contributed to more than one data set, in some instances, participating in a follow-on study conducted by the same research group. In such cases, their demographic details were counted only once in our data set (e.g. Table 3). For other participants their language performance data had been duplicated across data sets. We identified these duplications and removed them from our database.

People with aphasia

Of the 5928 people with aphasia following stroke represented in the RELEASE study database, 61.4% were male (3407/5928 IPD) and 38.6% (2143/5928) were female. They had a median age of 63 years (IQR 53–72 years). Most were right-handed [3719/3879 IPD (95.9%)]; the remainder were left-handed [133/3879 (3.4%)] or ambidextrous [27/3879 (0.7%)]. The median number of years of formal education, for those participants who had such information recorded [84/174 data sets (48.3%); 3125/5928 IPD (52.7%)], was 12 (IQR 10–16 years). Ethnicity was recorded for one-quarter of participants [1475/5928 IPD (24.9%)] across 24 out of 174 data sets (13.8%). Three-quarters of participants were from a White ethnic background [1116/1475 IPD (75.7%)]; the remainder were recorded as being of Asian [248/1475 (16.8%)], Black [67/1475 (4.5%)], Latino/Hispanic [9/1475 (0.6%)], Mixed [2/1475 (0.1%)] or Other [33/1475 (2.2%)] ethnic backgrounds (see Table 3).

Living context

At the time of data collection, two-thirds of participants were living at home with other people [473/701 IPD (67.5%)], and one-fifth lived alone [146/701 (20.8%)], with a smaller number [70/701 (10.%)] living in a formal care environment. This is based on information from 21 out of 174 data sets (12.1%) and 701 out of 5928 IPD (11.8%) (see Table 3).

Language

The database was predominantly based on English-speaking participants [3162/5928 IPD (53.3%)]: the largest data set contributions were from the USA [56/174 data sets (32.2%); 1317/5928 IPD (22.2%)] and the UK [30/174 data sets (17.2%); 1529/5928 IPD (25.8%)]. The remaining data sets were collected across 22 other languages. One data set included bilingual data from 12 out of 5928 IPD (0.2%) (Table 4).

Stroke

Among those for whom we had information on time since stroke [5841/5928 IPD (98.5%)] the median time since stroke was 321 (IQR 30–1156) days. When reported, this was mostly in the left hemisphere [3965/4130 (96.0%)], ischaemic [2795/3416 (81.8%)] and, in most cases, their first stroke [1164/1275 (91.4%)] (see Table 3).

Stroke severity at study entry was moderate to severe on the National Institutes of Health Stroke Scale (NIHSS) [median score 11 (IQR 5–17); 8/174 data sets (4.6%); 716/5928 IPD (12.1%)], the modified Rankin Scale (mRS) [median score 4 [IQR 3–4]; 6/174 data sets (3.4%); 489/5928 IPD (8.2%)] and the Barthel Index [the 0–100 scale: median score 60 (IQR 15–95); the 0–20 scale: median score 16 (IQR 9–20); 5/174 data sets (2.9%); 442/5928 IPD (7.5%)]. Interestingly, despite all included participants having aphasia, all stroke and disability scales included participants who had no stroke symptoms [NIHSS = 0; 15/716 IPD (2.1%)], no symptoms at all [mRS = 0; 6/489 IPD (1.2%)] or scored as ‘independent’ [Barthel Index = 100; 48/442 IPD (10.9%)] (see Table 3).

Concomitant symptoms

Cognitive impairment was an inclusion or exclusion criteria for 100 out of 174 data sets (57.5%) involving 3945 out of 5928 IPD (66.5%), of whom 7 IPD participants were recorded as having a cognitive impairment. Cognitive scores were available for 591 out of 3945 IPD (15.0%), and an additional 3347 IPD (84.8%) participants did not have a cognitive impairment. Similarly, depression was listed in the inclusion or exclusion criteria for 62 out of 174 (35.6%) data sets relating to 2075 out of 5928 IPD (35.0%). Depression was reported for 352 out of 2075 IPD (17.0%) participants; most [1723/2075 IPD (83.0%)] were not depressed.

Measures of language

All data sets included IPD on one or more of the following language outcomes: overall language ability [2699/5928 IPD (45.5%); 80/174 data sets (46.0%)], auditory comprehension [(2750/5928 IPD (46.4%); 71/174 data sets (40.8%)], naming [2886/5928 IPD (48.7%); 75/174 data sets (43%)], other spoken language [380/5928 IPD (6.4%); 9/174 data sets (5.2%)], reading comprehension [770/5928 IPD (13.0%); 12/174 data sets (6.9%)], writing [724/5928 IPD (12.2%); 13/174 data sets (7.5%)], functional communication observer-rated [1591/5928 IPD (26.8%); 29/174 data sets (16.7%)], functional communication self-rated [68/5928 IPD (1.1%); 3/174 data sets (1.7%)] and discourse analysis [213/5928 IPD (3.9%); 5/174 data sets (2.9%)] (Table 5).

Speech and language therapy interventions

We identified 91 out of 174 data sets (52.3%) [2746/5928 IPD (46.3%)] that delivered a SLT intervention. Two registry data sets included some participants in receipt of SLT, but the interventions were not standard across the data set and were inconsistently recorded, so these were excluded from our analysis of the impact of SLT components on outcomes. Language outcome data availability at baseline and at the end of intervention limited the number of data sets that could inform our planned intervention analysis to 67 out of 174 (38.5%) data sets [2330/5928 IPD (39.3%)] (Table 6).

Interventions included conventional therapy [8/67 data sets (11.9%); 529/2330 IPD (22.7%)], social support [2/67 data sets (3.0%); 169/2330 IPD (7.3%)] or a defined SLT intervention that could be categorised by impairment target or theoretical approach [62/67 data sets (92.5%); 1456/2330 IPD (62.5%)] (see Table 6). Twenty of these defined SLT interventions included a pharmacological or electrical brain stimulation co-intervention, which may have contributed to language outcomes [20/67 data sets (29.8%); 423/2330 IPD (18.1%)]. Eleven data sets included a comparison group that had no access to therapy [11/67 data sets (16.4%); 167/2330 IPD (7.2%)]. One intervention targeted change in communication behaviours among a health-care professional group and so did not contribute to our analysis of SLT interventions delivered to people with aphasia.

Intervention context

Most participants receiving an intervention lived at home with others [195/2330 IPD (8.4%)]. Interventions were typically delivered in a formal health-care setting [1866/2330 IPD (80.1%)] on a one-to-one [1613/2330 IPD (69.2%)], face-to-face [1957/2330 IPD (84.0%)] basis, where therapy was provided by a professional [1871/2330 IPD (80.3%)] (see Table 6).

Intervention description

Therapy targeted a range of language impairments including spoken language and/or naming [734/2330 IPD (31.5%)], auditory comprehension [68/2330 IPD (2.9%)], mixed spoken language and auditory comprehension [583/2330 IPD (25%)] and reading [10/2330 IPD (0.4%)]. No intervention specifically targeted the rehabilitation of writing. Most of the spoken-language interventions targeted improvements on naming (also known as word-finding) therapy [489/2330 IPD (21%)].

A range of theoretical approaches were identified, including semantic [34/2330 IPD (1.5%)], phonological [124/2330 IPD (5.3%)], semantic and phonological [260/2330 IPD (11.2%)], constraint-induced aphasia therapy [113/2330 IPD (4.8%)], melodic intonation therapy [61/2330 IPD (2.6%)] and functional–pragmatic therapy approaches [246/2330 IPD (10.6%)].

Intervention regime

We gathered data on the frequency [2057/2330 IPD (88.3%); 66/67 data sets (98.5%)], duration [1960/2330 IPD (84.1%); 64/67 data sets (95.5%)], intensity [1882/2330 IPD (80.8%); 60/67 data sets (89.6%)] and dosage [1978/2330 IPD (84.9%); 62/67 data sets (92.5%)] of the SLT interventions. Information on whether or not the SLT intervention had been tailored by difficulty of the tasks was available for 1145 out of 1373 IPD (83.4%) [34/67 data sets (50.7%)]; information on whether or not the SLT intervention had been tailored by functional relevance to the individual participant was available for 649 out of 928 (69.9%) IPD [22/67 data sets (32.8%)]. Whether or not participants had been prescribed home-based practice tasks was available for 1306 out of 2330 IPD (56.1%) [31/67 data sets (46.3%)] (see Table 6).

Risk of bias

The sample size of the primary research data sets ranged from 10 IPD of people with aphasia after stroke [25/174 data sets (14.4%)] to one data set with 420 IPD. An overview of each contributing data set can be found in Report Supplementary Material 1, but a summary is provided in Quality of primary research data sets.

Quality of primary research data sets

Selection bias

A total of 47 RCTs were included in the RELEASE study. Of these, 27 employed an adequate approach to the generation of the random number sequence (such as a random numbers table). Concealment of allocation was confirmed in 22 RCTs. We also included 18 non-RCTs that did not employ randomisation.

When data availability permitted, we checked individual RCT comparison groups at baseline for similarity. We identified six out of 47 RCTs for which there was a significant between-group difference at baseline by reported age (3/45 RCTs), sex (1/45 RCTs), time since stroke (2/45 RCTs) and language impairment (1/45 RCTs) (see Appendix 4, Table 12). Only two out of nine non-RCT group comparison data sets were found to differ at baseline in relation to reported time since stroke (see Appendix 5, Table 13) and one dataset differed with regard to sex (see Appendix 5, Table 13).

Detection bias

Although participant blinding in SLT studies is often unachievable, the blinding of outcome assessors is important when considering risk of bias and can usually be achieved. Blinding of outcome assessors was reported in 48 out of 174 data sets (27.5%), and an additional 9 out of 174 (5.2%) data sets described partial blinding (blinding for a subset of data) for participants. The remaining 117 data sets either did not report on blinding (92/117) or clearly indicated that blinding was not present (25/117). These figures reflect the use of blinding across all data sets in the RELEASE study database. In many cases, blinding was simply irrelevant to the primary purpose of that data set (e.g. a clinical registry or observational study). Risk of detection bias will be considered more specifically in relation to our analysis in Chapters 6 and 8.

Attrition bias

One-third of data sets [58/174 (33.3%)] reported participants who failed to complete the study (i.e. who dropped out). Such withdrawals are to be expected in a participant population after stroke. However, although there is a risk of incomplete outcome data, most researchers provided details of dropouts (for any reason) and of those who did not adhere to the intervention. By contrast, 116 out of 174 data sets did not report any loss of participants over time.

Reporting bias

We identified a risk of reporting bias among data sets that were only partially represented in the RELEASE study database. One-third of included IPD [1988/5928 IPD (33.5%); 99/174 data sets (56.6%)] were extracted from the public domain. Although some data sets were fully reported in the public domain, most were only partially available (e.g. baseline data only).

Similarly, some primary research data set electronic contributions were also incomplete. Cross-referencing with the published reports indicated that additional data items were available at the point of publication. It is possible that some data items (e.g. IPD on therapy dosage) had not been stored electronically as part of the main data set or had been lost over time.

Other biases

Inclusion and exclusion criteria were available for 170 out of 174 (97.7%) and 120 out of 174 (69.0%) of the data sets, respectively. Both inclusion and exclusion criteria were reported by 115 out of 174 (66.1%) data sets. Eight data sets were based on retrospective data collection [8/174 (4.6%)]. Eleven out of 174 (6.3%) data sets were collected in the context of primary research studies that aimed to develop or validate an outcome measure or assessment. One intervention study targeted arm movement recovery, whereas the SLT intervention functioned as an attention control intervention (32 IPD).

Other quality data items included reporting on the context of data collection (2441 IPD) and the presence of co-interventions (423 IPD). We planned to consider these items in our analysis and interpretation of findings. Adherence to a SLT intervention was captured for only 27 out of 91 (29.7%) intervention data sets.

Data synthesis

Transformations

A total of 64 assessments were used across data sets to capture participants’ language performance. Some assessments comprised several subtests, which also contributed to the data synthesis for other language domains [e.g. Western Aphasia Battery (WAB) data contributed to overall language ability, auditory comprehension, naming and reading outcomes]. Language-specific adaptations of an original assessment were profiled individually to accommodate differences between languages.

We collated and categorised each assessment and their relevant subtests by language domain. For each data set, we noted the scoring format. Data from some tools [e.g. the Token Test (TT)] were reported in raw scores, percentiles, t-scores and error scores and were either positively or negatively scored. All scores were converted to raw scores and positive scoring.

For each language domain, we identified the assessment used by the most data sets as the ‘anchor measure’ (see Appendix 7, Table 14). In most cases, this choice also reflected the assessment with the greatest availability of IPD. The availability of IPD on an assessment contributed to the choice of anchor measure only in situations in which the greatest number of data sets using an assessment was shared across an equal number of data sets.

We examined the contribution that each assessment made to the overall data for that language domain and plotted the distribution of each score at baseline across increasing time since stroke time points and stratified by the original assessment (see Appendix 6, Figure 27). We concluded that the data points generated by each of the minority and anchor measures were valid and within the expected ranges.

The maximum, minimum, median, lower quartile (LQs) and upper quartile (UQs) were calculated for each of the identified anchor measures (see Appendix 7, Table 14). The values for the maximum, minimum, median, LQ and UQ were calculated for each of the remaining assessment tools that captured that language domain (minority measures). These values were then transformed to the anchor measure scale, ensuring that the values remained within the minimum and maximum values of the anchor measure scale (see Appendix 7, Table 14).

For each analysis, we collated the data with other required data items (e.g. participant demographic data items or intervention data), which resulted in a reduction of overall data to inform our analysis. We profile this drop-off in data availability in Table 5.

Risk of bias in our individual participant data meta-analyses

Selection bias in data set recruitment

The systematic review-based approach to the identification of eligible data sets recruited an additional 122 data sets to those originally identified through our research network alone, yielding important demographic and language IPD. For example, our final database has 5928 IPD on time since stroke (our original recruitment estimate was 3181 IPD) and 2922 IPD on naming outcomes (originally estimated as 2007 IPD). Methodologically, employing a review-informed approach to building the database was essential. It reduced the risk of bias from an analysis based only on data sets contributed through pre-existing networks and increased the feasibility of analysis on a sparse database with limited overlap in outcomes and demographic data across data sets.

Publication bias in data set recruitment

Overall, the database included data sets collected across 28 countries and 23 international languages (see Tables 2 and 4). We included 13 unpublished data sets. We considered the risk of publication bias by examining the proportion of significantly reported results by decade or other suitable period grouping. Any suggested imbalance was tested with the chi-squared test (and is reported in Chapter 8).

Availability bias in data set recruitment

In the systematic review, we identified 399 out of 698 (57.2%) records reporting 267 potentially eligible data sets, which, despite best efforts, were not included (see Report Supplementary Material 2). Of these, 53 data sets appeared to be RCTs, with 46 out of the 53 RCTs including a SLT intervention that was part of the study protocol or standard care. Of the of 47 RCTs that contributed group summary level data to the Cochrane review meta-analysis of SLT for aphasia after stroke,41 we secured IPD for 17. We also secured IPD from five RCTs that were ‘ongoing’ at the time of the Cochrane review41 and from a further four RCTs that were identified but unavailable to the Cochrane review.41

Summary

We created an international database of 5928 IPD on people with aphasia after stroke, gathered from 174 data sets from across 28 countries. Half of the data sets were based on an English-speaking population, one-quarter were RCTs and 13 were unpublished. All data sets included participants for whom data were available on time since the index stroke and a measure of language impairment on overall language ability, auditory comprehension, spoken language (including naming), reading comprehension, writing or functional communication. Within this, approximately half of the dataset had a language measurement as well as corresponding data on baseline demography. Half of the included data sets referred to a SLT intervention.

Interventions were wide-ranging and classification of SLT approach by impairment target and theoretical approach required a consensus approach. We considered risk of bias both at the primary data set level and at the meta-analysis level and, when possible, took methodological approaches to address any risk of bias or to consider its possible impact when interpreting the findings. We transformed all language measures to a single anchor measure for each language domain and filtered IPD based on the overlapping availability of the required variables for each planned analysis.