Jeff T Flinn, David Wood, Caroline G Cao
Wright State University
The manual skills component of the FLS Test requires scoring by a trained evaluator. The scoring methodologies being used, though accepted by the surgical community, are resource intensive, and require subjective judgment on the part of the evaluator. A particularly cumbersome and error-prone scoring method is the one used for the pattern-cutting task. One part of the scoring procedure requires the evaluator to manually cut out from the mesh sample all deviations from the drawn circle. These bits are then placed onto a grid to determine how many squares they cover. This procedure has several inherent limitations. It is labor- and time-intensive. Since the cut bits are irregularly shaped they cannot be fit together seamlessly on the grid, thus introducing errors in the area estimation. Additionally, there are many possible ways to arrange the cut bits of deviation so the method is vulnerable to problems with inter- and intra-rater reliability. This source of variability, along with built-in measurement error due to the relative coarseness of the measuring scale (grids are 1 mm squares), is in effect a loss of measurement precision. We have developed a technology-based approach that algorithmically computes the precise area of deviation for the pattern cutting task. This automatic scoring method can objectively provide a quick, accurate, and precise error measurement for the purpose of scoring the FLS pattern cutting task.
The procedure consists of two simple steps and requires minimal instruction to perform. First, the cut pieces of gauze are each digitally scanned. Second, the scanned images are analyzed by custom-developed image analysis software to produce the area of error, both inside and outside the prescribed circle.
Performance evaluations were conducted to compare both the accuracy and the efficiency of the automated and manual methods. In order to provide a standard for comparison, a test sample was prepared for which the total area and the area of error were known quantities. This test sample was then repeatedly measured 5 times per method by an untrained evaluator. The measure of accuracy indicates how much an error estimate deviates from the true error. It was calculated as the difference between the error estimate of the method and the actual error, expressed as a percentage of the actual error. Therefore, 0% deviation represents an exact estimate of the actual error. Results show that the mean accuracy of the automated method was -0.094% deviation with a SD of 0.44, whereas the manual method had a mean of 26.19% deviation with an SD of 3.22. The automated method took an average of 2.0 minutes to scan and calculate the area of error, whereas the manual method took 10.1 minutes to photocopy, cut, assemble, and estimate the area of error.
Based on these results we conclude that our objective technology-based approach to error measurement provides advantages in speed, accuracy, and precision over the manual scoring method currently in use. Consequently, this approach could result in lower administrative costs and increased confidence in testing results.
Session: Poster Presentation
Program Number: ETP066
