Wysocki classification validation exercise

Background

Clinically, classification systems may have a prognostic function and could ultimately allow stratified treatment. Such systems also ensure that more precise research comparisons can be carried out. While there are some existing classification systems for PSD,128–135 they are not used in routine practice or for research comparisons. Few studies evaluate their use to inform choice of treatment130,133–135 and no study has analysed the reliability or predictive validity of such a tool.110 Given the huge variation in treatment that we have shown both with the PITSTOP surgeon survey and the cohort study, and the existing issues with the current comparative literature, there is a need for a classification tool that is both reliable and valid. A suitably pragmatic classification system could be integrated into clinical practice to support treatment decisions and the counselling of patients on likely outcomes. Such a system should be simple to use, reflect clinical practice and be meaningful in terms of prognostication.

Rationale

There is no commonly used classification tool to characterise PSD. The absence of such a tool represents an important knowledge gap, since surgeons faced with PSD typically have little exposure to the disease and there exists little guidance on how to treat it.

At the inaugural meeting of the Pilonidal Society in Berlin in 2017, a panel of 13 surgeons gathered to establish a simple classification system for PSD.136 Each member of the panel had a special interest in PSD and had either written other classifications132,134 or had published widely on PSD. Components of a classification system were longlisted and refined by online consensus involving a large group of PSD surgeons137 to form a final configuration that was felt to be easy to use, clinically meaningful and had potential statistical validity.

The tool classifies PSD into one of four categories:

• type 1: only midline pit or sinuses
• type 2: any midline disease with secondary sinus/es or abscess scar/s
• type 3: any midline or secondary disease extending below tip of coccyx
• type 4: any disease after treatment with definitive intent.

While there may be agreement among the panel for the eventual classification (referred to as the ‘Wysocki classification’), none of the parameters of ease of use, reliability and validity have been tested. We aimed to do so by assessing the level of agreement between surgeons when used in clinical practice.

Aims

The aims of this substudy were to:

• Quantify how well different assessors agree in their classification.
• Quantify how classification relates to surgeon’s experience.

Additional exploratory aims were to:

• Identify specific patients with low agreement, which may in turn help clarify or even modify the classification system.
• To present, within each subtype, the frequency with which each treatment option is chosen.

Methods

Participating surgeons

Sampling was purposive and sought to recruit 15 colorectal surgeons. This sampling method aimed for maximum variation based on experience: five expert surgeons who registered an interest in pilonidal disease and who offered a specialised service, five surgeons who carried out pilonidal sinus surgery as part of a general surgical service and five final-year colorectal trainees. Initial contact with surgeons was made by e-mail, and interested surgeons were e-mailed the participant information sheet.

Participating patients

Stimuli required for the validation exercise were obtained from the main cohort study.

All participants referred for elective surgery and participating in the PITSTOP cohort study were asked if they agreed to the surgical site being photographed before surgery (an optional item on the consent form).

Participant photographs of the PSD surgical site were usable for the exercise if they satisfied the following criteria:

• The participant was eligible for the main cohort study (aged 16 years and above with symptomatic PSD, referred for elective surgery).
• The participant consented to a preoperative photograph to be taken of the surgical site.
• The photograph was usable (i.e. unblurred).
• The participant associated with the photograph had been classified using the Wysocki classification system at the time of procedure.

Assessment schedule

Each participant was asked to independently rate 36 cases using the Wysocki classification, with allocation of surgeons to cases selected to ensure overlap with other assessors (Figure 9). A total of 90 photographs were each assessed by two specialist surgeons, two general surgeons and two trainee surgeons. Surgeons were sent photographs accompanied by the medical history (previous PSD history, including number of elective procedures and emergency drains) electronically. Surgeons were not told the classification as recorded by the original surgeon at the time of procedure.

FIGURE 9 Allocation of patients with PSD to surgical assessors. (Figure not reproduced here; see the original article.)

The substudy was also used to provide an exploratory assessment of surgical opinion. Participating surgeons were asked to record their preferred treatment for each patient they assessed, with the aim being to quantify variation in practice among practitioners. Surgeons recorded their assessments in Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) spreadsheets which were returned to CTRU and combined into an analysis data set.

Statistical methods

Agreement was quantified as both raw and chance-corrected agreement. Raw agreement is the percentage of patients for whom the assessments agree, while chance-corrected agreement is the ratio of observed to expected agreement. Both raw and chance-corrected agreement are essentially proportions in which one means complete agreement while zero reflects complete disagreement. As the four categories are not ordinal, agreement is a simple yes/no construct in which any difference is considered ‘disagreement’.

Raw agreement was defined as (100 × number in which all raters agree / number rated) and was accompanied by a 95% Wilson score interval. Chance-corrected agreement was defined using the unweighted kappa statistic and the unweighted Gwet AC1 statistics.138 Agreement among surgeons was reported overall and within for expert, general and trainee surgeons. Finally, the agreement was calculated for each surgeon in relation to the original assessment made at the time of procedure. Analyses were conducted using Stata version 17.139

Sample size

The sample size was based on: (1) a hypothesis test to rule out a minimal kappa statistic;140 (2) the standard error of raw agreement; and (3) the number of patients expected to consent and provide usable photographs. For (1), an internal pilot was undertaken in which study surgeons were asked to assess photographs obtained either online or via published articles. A total of 41 pictures were assessed by seven surgeons (five specialists and two trainees) and yielded an overall kappa statistic of 0.42 (0.55 if trainees were excluded). Since the assessments were based on pictures alone and did not include prior history, these may be an underestimate. Based on this, a target of κ = 0.45 was used. The expected lowest limit for chance-corrected agreement was set at κ = 0.3, which represents the lowest acceptable agreement if this classification were to be introduced into practice. The sample size calculation also depends on the expected prevalence of each class, which was estimated from the cohort study (approximately 25% type 1, 50% type 2, 10% type 3 and 15% type 4 at the point of data review). A sample size of 90 was adequate to rule out differences of 15% between expected and minimum kappa with 90% power and 5% significance (1); to estimate raw agreement to within a CI half-width of ±10% (2); and to be accommodated by the number of photographs available (expected around 150) (3).

Results

Participants

Fifteen surgeons participated in the classification exercise as described in Chapter 7, Assessment schedule. A total of 166 patients consented to and provided a photograph of the diseased area, of which three were too unclear to use and were removed. Ninety participants were randomly selected from these for assessment, all of whom had been assigned a classification of 1–4 by the original treating clinician. The majority were classified as type 2 disease at the time of procedure, and 60% underwent a major procedure, with the most common treatments being a Karydakis asymmetric closure (n = 42) or glue (n = 29) (Table 23).

Agreement among participating surgeons

The agreement among surgeons is summarised in Table 24. Of the 540 assessments (90 patient photographs and case histories each having 6 assessments), 14 (3%) of assessments were classified ‘none of the above’, affecting 12 (13%) of the patients.

Overall, the six assessors all reached the same consensus in 38% of participants, with a chance-corrected kappa statistic of 0.52 (95% CI 0.42 to 0.61) and a Gwet AC1 statistic of 0.63 (95% CI 0.56 to 0.69). Agreement between pairs of surgeons was higher, with specialist surgeons agreeing in 72% of patients, general surgeons agreeing in 70% of patients and trainee surgeons agreeing in 71% of patients. The overall agreement is lower since this measure required all six assessors to agree. All six surgeons agreed in 34 (38%) of patients, and five of the six surgeons agreed for 19 (21%). The chance-corrected kappa agreement was above 0.5 (conventionally considered ‘moderate’) and the Gwet AC1 measure over 0.6 for all subgroups of surgical expertise.

Agreement between participating surgeons and original classification

Each assessor’s agreement with the original classification is described in Figure 10 and 11. Surgeons were less likely to agree with the original classification than with other surgeons given the same photograph. Raw agreement ranged between 47% (17/36) and 75% (27/36) with chance-corrected kappa statistics ranging from 0.11 to 0.59, and chance-corrected Gwet AC1 agreement statistics from 0.35 to 0.71.

FIGURE 10 Raw agreement between assessment and original. (Figure not reproduced here; see the original article.)

FIGURE 11 Kappa chance-corrected agreement between assessment and original. (Figure not reproduced here; see the original article.)

Treatment choice

Surgeons differed markedly when asked how they would treat the individual participants. Overall, surgeons in the substudy recommended a minimally invasive procedure in 46% of cases, but the figure ranged from 0% to 94% (34/36) of patients. There was substantial variation in practice among all levels of expertise, and the variation in recommendation did not appear related to the actual patients assessed (Appendix 3, Figure 16).

The surgeons surveyed were slightly more likely to recommend minimally invasive surgery (46%) than those that actually received this approach (40%). Although the percentage favouring asymmetric closure (45%) was similar to the treatment actually received (49%), the specific procedure types differed: the surgeons surveyed were more likely to use a Bascom’s cleft lift (22%) than a Karydakis (16%). The use of midline closure (5%) and leave open (4%) approaches were uncommon, most notably among the specialist surgeons surveyed.

Discussion

There are three potential roles of a classification system. Two are clinical – predicting prognosis and guiding treatment – and the third is primarily for research purposes, allowing more precise comparative studies to be carried out and reducing the potential for selection bias. While there have been eight previously proposed classification systems for PSD,110 each used judgemental methodology to develop their systems and identified homogeneous categories based on the experience of the investigators. The classifications were mainly used to select patients for different procedures. However, none provided analyses to demonstrate ease of use, reliability or predictive criterion validity. We have shown that the Wysocki classification, developed from components of these other systems, demonstrates moderate but acceptable agreement among the surgeons participating in this classification exercise. Agreement was similar among specialist, general and trainee colorectal surgeons, which offers reassurance that this system could be used across a range of surgeons.

While the level of agreement between substudy assessors and the original classification was lower, this is likely to be attributable to a mixture of picture quality and other features that were not available to the assessor in this exercise. A minority of cases (2.5%) were considered not to fall into any of the four categories, which was similar to the incidence seen in the cohort study (Chapter 3). This itself is relevant and indicates that this classification system incorporates reliably definable disease characteristics for almost all presentations of disease.

The findings from this substudy inform the main study, and vice versa. The general agreement seen in this substudy among surgeons lends weight to the Wysocki classification as an objective measure when conducting the risk adjustment. In turn, the cohort study found this classification to be an important feature in predicting response to treatment, with class 1 disease, in particular, being associated with more favourable outcomes. If the Wysocki classification is used to prognosticate and to inform the appropriate treatment, it is therefore important to demonstrate its reproducibility. While this substudy demonstrated only moderate agreement, its magnitude exceeded previously reported inter-rater reliability of other surgical classifications such as grading of haemorrhoids (κ = 0.38141) and dysplastic colorectal adenomas (κ = 0.38142). These findings support the use of the Wysocki classification in accurately defining subtypes of PSD.

Finally, the opportunistic question ‘what treatment would you recommend’ – while not central to the validation exercise – triangulates the findings of the consultant surgeon survey (see Chapter 2) in demonstrating that the choice of procedure is highly surgeon-dependent rather than evidence-based. This is, perhaps surprisingly, even the case in the specialist group where, for instance, the variation in minimally invasive procedures is immense. This reiterates that the disease characteristics are not the sole driver of treatment choice. This matters for the risk-adjusted comparisons in the cohort study, since non-randomised comparisons are known to be biased in situations when (1) treatment is defined by severity and (2) severity cannot easily be quantified and modelled.