Sami Chadi, MD1, Biniam Kidane, MD2, Patrick H Colquhoun, MD3, Nawar AlKhamesi, MD3, Christopher M Schlachta3, Steven D Wexner, MD, PhD, Hon1. 1Digestive Disease Center, Cleveland Clinic Florida, 2Division of Thoracic Surgery, University of Toronto, Toronto, Canada, 3Division of General Surgery, Western University, London, Canada
Introduction: Survey research is an important step in current quantitative and qualitative research methods. Reviews of survey methodology in critical care and nephrology have demonstrated a need for standards. The objective of this study was to review current standards of survey methods and reporting in the surgical literature.
Methods: All issues of 8 surgical journals were reviewed to identify surveys of health professionals between 2002-2013, excluding mixed-methods studies and surveys of specific interventions (4 general: Journal of the American College of Surgeons, Surgical Endoscopy, Annals of Surgery and World Journal of Surgery; 4 subspecialty: Journal of Pediatric Surgery, Journal of Vascular Surgery, Diseases of the Colon and Rectum, Annals of Surgical Oncology). Four other journals were initially included a priori that were subsequently excluded due to (1) no surveys being published, (2) publication of articles for only a part of the cohort. The main outcome was the overall transparency and quality of surveys (5 point Likert scale).
Results: We included 219 surveys in this review. Overall inter-rater reliability was strong for both publication transparency (weighted κ 0.8; 0.67-0.93) and publication quality (weighted κ 0.83; 0.72-0.93). The median response rate was 62%. The most common method of survey dissemination was email (46%) followed by postal (33%), with 20% using multiple methods. Publication rates were assessed per 1000 citations from each journal per year. No overall trend of increasing publication rate was identified (p=0.43) although there was evidence of differing rates of publication among journals (p<0.001). Although 48% reported sending reminders, 46% did not report whether this was done. Regarding instrument development, item generation and item reduction were mentioned in 11.6% and 2.5% of papers. Regarding instrument testing, 19% of papers piloted their instrument prior to dissemination whereas clinimetric properties of at least one of face, content, construct validity, intra- or inter-rater reliability, or consistency were mentioned in less than 13% of studies. Median quality and transparency scores were 3. Higher survey quality (as determined by reviewers) appeared associated with reporting a previously published instrument (OR 3.0; p<0.001), single survey metric (OR 22.0; p<0.001) and discussing the management of incomplete surveys (OR 2.9; p=0.014).
Conclusion: The quality of survey methods and reporting is suboptimal in the surgical literature. In particular, reporting survey metrics and instrument testing appears to need significant improvement, with even lower rates than the medical literature. Guidelines of reporting and methods are necessary to standardize and improve quality.
« Return to SAGES 2016 abstract archive
