Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) Guidelines Development: Standard Operating Procedure

Find a SAGES Surgeon

Amelia T. Rogers, MD¹, Rebecca Dirks, MD, MS¹, Holly Ann Burt, MLIS², Stephen Haggerty³, Geoffrey P. Kohn, MBBS, MSurg^4,5, Bethany J. Slater MD, MBA⁶, Danielle Walsh, MD⁷, Dimitrios Stefanidis, MD, PhD¹, Aurora Pryor, MD, MBA⁸

¹Department of Surgery, Indiana University School of Medicine, Indianapolis, USA
²Society of American Gastrointestinal and Endoscopic Surgeons (SAGES), Los Angeles, USA
³NorthShore University Health System, Evanston, USA
⁴Department of Surgery, Monash University, Eastern Health Clinical School, Melbourne, VIC, Australia
⁵Melbourne Upper GI Surgical Group, Melbourne, VIC, Australia
⁶Department of Surgery, University of Chicago Medicine, Chicago, USA
⁷Department of Surgery, East Carolina University, Greenville, USA
⁸Department of Surgery, Stony Brook University, Stony Brook, USA

Keywords: guidelines development, surgical practice guidelines, systematic review

ABSTRACT

Introduction: The mission of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) is to innovate, educate, and collaborate to improve patient care. A critical element in meeting this mission is the publishing of trustworthy and current guidelines for the practicing surgeon.
Methods: In this manuscript, we outline the steps of developing high quality practice guidelines using a completely volunteer-based professional organization.
Results: SAGES has developed a standardized approach to train volunteer surgeons and trainees alike to develop clinically pertinent guidelines in a timely manner, without sacrificing quality.
Conclusions: This methodology can be used more widely by volunteer organizations to efficiently develop effective tools
for practicing physicians.

INTRODUCTION

Clinicians rely on systematic reviews and guidelines to be of the highest quality of dependable evidence, as they serve as a reliable foundation for shaping their professional practice. However, few systematic reviews adhere to a standardized protocol and only a third report on the quality of the studies that are included [1]. The Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) is committed to producing high quality, reliable guidelines for the greater surgical community and has developed a standardized process to ensure consistently high standards for these publications.

Historically, SAGES guidelines were based on expert and consensus opinions. The Guidelines Committee is completely comprised of volunteers to generate their product. Guideline development is quite a costly endeavor otherwise. For example, the World Health Organization (WHO) estimates that the cost for one standard guideline is around $100,000 [2]. This is a prohibitive expense when scaled to meet both the scope of practice of the general surgeon and the speed at which new literature is published. In addition, rather than invest this amount in a single guideline, SAGES invested in developing a cadre of surgeons with advanced knowledge of guideline methodology through serial educational events, curriculum development, and enduring educational materials tailored to surgical topics. This forms the foundation for utilizing educated volunteers in creating and maintain multiple guidelines simultaneously.

Over the past few years, SAGES has adopted an evidence-based approach to developing guidelines using volunteer surgeon members, without sacrificing quality. SAGES guidelines provide recommendations based on the highest level of evidence available, ideally randomized controlled trials (RCTs) and meta-analyses based on RCTs. However, it is important to recognize that studies related to surgical management often do not include these high levels of evidence, and studies other than RCTs, such as observational trials, often have to be included in our reviews and analyses. Additionally, surgical outcome studies are often negatively impacted by small sample sizes, inadequate patient follow-up periods, and missing data; and significant heterogeneity often exists among studies. As such, expert and consensus opinions still provide value by informing the distillation and interpretation of evidence, without replacing it. Examination the entire body of scientific literature on a specific topic, with recognition of associated biases, allows development of resources that provides guidance to surgeons in an evidence based, relevant manner.

The following standard operating procedure (SOP) is an overview on the Evidence-Based Medicine methodology and steps involved in creating a SAGES Guideline based on published best practices to facilitate future systematic reviews and guidelines both within and outside of SAGES.

METHODS

The steps presented in this paper are informed by The Institute of Medicine’s recommendations for guideline development, which promotes transparency and multidisciplinary groups to maintain balance [3]. We utilize both the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines for reporting systematic reviews [4] and Grading of Recommendations Assessment, Development, and Evaluation (GRADE) process [5]. The PRISMA statement is a 27-point checklist which authors of systematic reviews utilize to ensure accurate reporting [4]. Whereas GRADE is used to evaluate the quality of the evidence included and the strength of the recommendations of the guideline to provide transparent guidelines [5]. GRADE and PRISMA are widely accepted as essential for quality guidelines.

RESULTS

Step 0: Project Conception and Team Assignments

The development of new SAGES guidelines considers input from the SAGES membership via surveys and the SAGES Executive Committee and Board of Governors [6]. Topics are generally within the realm of gastrointestinal and endoscopic surgery. Guideline projects are often initiated around the biannual committee meetings (typically at the SAGES Annual Meeting in the spring and the American College of Surgeons Annual Meeting in the fall). Once a guideline topic has been chosen, a committee member is assigned by the committee chair to lead the development of the guideline. The Lead and Committee Chair then recruit committee members with content expertise to a working group tasked with gathering evidence upon which the guideline can be based, in the form of a systematic review.

The lead and committee chairs invite experts on the Guideline topic, both within and outside of SAGES Guidelines Committee, to the panel. The working group makes every effort to invite members of other SAGES committees and task forces who are experts on the topic. Furthermore, they often reach out to other societies in order to increase the breadth of expertise and ensure more widespread consensus of eventual guidelines endorsement. These colleagues work on the panel to help develop the final recommendations from the results.

SAGES believes that patient engagement in guidelines development is critical, as they can provide insight into what is of importance to the key end user of surgical care. Every effort is made to involve patient advocates at three points during the process:

1. Deciding on the Key Questions a guideline should answer
2. Determining what outcomes should be collected from research to answer these questions
3. Serving as a voting member of the guideline panel

These steps represent a living SOP that was created and edited by the SAGES Guidelines Committee leadership as the most current version of its methodology. See Table 1 for complete list of resources used in the guideline development process.

Table 1: SAGES Guidelines Committee Resources Description

Step 1: Key Question(s), Exclusion/Inclusion Criteria Generation

The working group, in conjunction with a SAGES guidelines committee fellow and contracted methodologist, first develops the key questions (KQs) which the systematic review and subsequent guideline will address. They format clinical questions pertinent to the Guideline topic into KQs which are amenable to research. This is a critical part of the process. If the questions are well-constructed, the results of the systematic review will be meaningful for recommendations on clinical decision-making.

The PICO (sometimes PICO-TS or PICO-D) format is used to construct the KQs. PICO(TS,D) refers to the key components to be addressed in the systematic review and guideline:

P: Population of interest
I: Intervention
C: Comparator
O: Outcome(s) of interest
T: Timing of intervention / D: Study design/duration
S: Setting

While creating the PICO questions, the outcomes for each key question are defined by the working groups and their importance assessed. Additionally, the study inclusion and exclusion criteria are defined using timing of intervention, study design/duration, and setting as guidance.

Step 2: Literature Search

Once the KQs have been developed, they are sent to the SAGES librarian. The librarian, the methodologist, and the guideline Lead work together to construct the appropriate search criteria, paying particular attention to terminology. The librarian then conducts the search in multiple databases, ideally a minimum of four. See Appendix 1 for complete list of possible databases used. Initial search results are evaluated by the librarian, methodologist, and guideline Lead to ensure that known relevant articles have been captured by the search. Depending on the search criteria, this can be an iterative process.

Step 3: Abstract Review

The goal of title/abstract review is to quickly eliminate any studies identified in the literature search that are off-topic or otherwise irrelevant for the KQs. This may include basic (animal) research studies, studies that are performed in different populations, or studies with a different comparator or intervention. There are two components to this step:

1. Calibration: This step ensures all group members are interpreting the PICO formatted key questions and employing the exclusion criteria in the same manner. The librarian uploads 50-100 abstracts to Abstrackr [7]. Abstrackr is a free, open-source citation screening software. Each group member reviews all uploaded articles. After the group has reviewed this subset of studies, a conference call is held. Discordant responses between reviewers and any remaining uncertainties about the title/abstract review process are discussed at that time.
2. Full title and abstract review: After calibration, all citations found during the literature review are uploaded to Covidence by the librarian [8]. Covidence is also a web-based platform to assist in systematic review creation from literature screening through data extraction. Each abstract/title is reviewed by two independent working group members. There are usually a number of abstracts where reviewers will disagree (i.e. one will say include and the other will say exclude or maybe). These disagreements are resolved by the original reviewers via discussion and, if needed, by involvement of the Lead or additional reviewer(s). In addition, previously published systematic reviews are evaluated and, if they directly address one of the KQs, then it is incorporated into the guideline. However, if it cannot by easily incorporated, then the reference list is reviewed and all constituent papers are included in the review phase.

Step 4: Full Text Review

The goal of the full text review is to finalize which studies will move forward for data extraction, and thus be included as evidence in the Systematic Review. The methodologist works with the working group Lead and members prior to beginning the full text review to ensure that the inclusion/exclusion criteria and reasons for rejection are established. The librarian obtains electronic versions of the articles and uploads them to Covidence. Any article excluded during this step must have a reason for exclusion recorded. Each article is reviewed by two independent members and any disagreements are resolved in a fashion similar to the abstract screening phase.

Steps 5: Data Extraction

While the full-text review of studies is underway, the Lead works with the fellow to create a draft of a data extraction form in Covidence. They meet to give as much specificity ahead of time to the outcomes to be collected, in order to diminish any discrepancies between extractors. The working group then pilots this form by extracting the same 1-2 studies. Using the results of this practice round and feedback collected during a subsequent group call, the final extraction form is created. Each study is then assessed and extracted by two independent reviewers on Covidence. Any discordance in their answers prompts an additional reviewer to manage a joint consensus on the extraction criteria in Covidence.

Step 6: Quality Assessment

Quality assessment is performed simultaneously to data extraction in Covidence. This critical step informs the overall quality of evidence supporting the guideline. For each individual study, the same two group members extracting study data make an independent judgement on the risk of bias (internal quality) based on provided frameworks. The Cochrane Risk of Bias tool 2.0 is used for RCTs [9]. A modified Newcastle-Ottawa Scale is used for non-RCTs (Appendix 2) [10]. These tools help to make a subjective process, judging the risk of bias, as objective as possible. We modified the original Newcastle-Ottawa Scale to allow some room for the reviewer to use their content expertise and critical thinking skills. Each of these frameworks guides the user through a series of questions to help determine whether the article has a high, moderate, or low risk of bias based on a study’s design, execution, and reporting. Given the inherent subjectivity in this step, discussion to resolve any conflicts is particularly crucial as is recording the reason(s) for a final judgement.

Step 7: Data Analysis/Meta-Analysis and Follow up Literature Search

After data extraction and quality assessment are complete, the data is compiled and organized by the lead and fellow based on a priori outcomes. Data is usually imported for analysis into RevMan, a Cochrane supported web-based software used to create systematic reviews and meta-analyses [11]. If data on an outcome is amenable, the methodologist will then perform a meta-analysis to obtain a relative effect (odds ratio/risk ratio) for binary outcomes or absolute effect (mean/standardized mean difference) for continuous outcomes and will create forest plots. Forest plots are a visual representation of the results of meta-analysis for a specific outcome, in which the effect from each included study is shown as well as a pooled effect based on all included studies.

If a meta-analysis is not possible or appropriate, the methodologist will work closely with the working group leader to otherwise analyze the data. If there is significant heterogeneity between the effect estimates from included studies, sources of heterogeneity are explored using subgroup analysis based on study risk of bias and based on any pertinent population differences between the included studies. RCT data is not pooled with observational study data. While the data is being compiled for the methodologist, the Librarian performs an updated literature search for new studies published since the start of systematic review.

Step 8: Evidence Tables and Systematic Review Manuscript

If creating a separate systematic review document, the PRISMA flow chart of study inclusion/exclusion is exported from Covidence for the methods section; see Figure 1 for an example [4]. All systematic reviews use PRISMA standards for reporting to structure the manuscript. Once written by the working group, the manuscript is reviewed by the entire Guidelines Committee and revisions suggested. Once approved by the Guidelines Committee, it is forwarded to the SAGES Board of Governors for comment and final approval before submission to Surgical Endoscopy for publication.

Regardless of whether a separate systematic review is submitted, the results are entered into the evidence tables on the GRADEpro online platform [12]. These tables present evidence for each outcome. The body of evidence for an outcome is appraised based on the overall risk of bias across all included studies, inconsistency between studies, indirectness of the evidence relative to the original desired outcome, and imprecision of the evidence as well as additional considerations such as publication bias, dose effect, large effect, and plausible confounding. These appraisals are made with help from the methodologist(s).

Step 9: Guidelines Development

Once the systematic review component has been completed, with or without a separate manuscript, the guidelines development process begins with meeting of the Guideline panel following the GRADE methodology [5]. The panel first reviews the evidence tables and a narrative of the systematic review results. They work with the methodologist to appraise the systematic review results, and they reprioritize the outcomes based on the evidence presented, deciding which of the outcomes are “critical” or “important” to the panel for influencing a decision about the key question. Outcomes whose evidence would not influence their decision on the key question(s) are deemed “not important” for guideline decisions.

The panel then populates the GRADEpro Evidence to Decision table (EtD) with the critical and important outcomes from the systematic review and formulates the evidence-based recommendations by voting on each section of this EtD table [12]. These sections include: assessment of the magnitude of both the desirable and undesirable effect the intervention has relative to the comparator; the certainty in the evidence (based on the evidence for critical outcomes); the variation that may exist of different outcomes between patient populations; the balance of effects from the preceding sections; and the acceptability and feasibility of a judgement made based on the balance of effects. A final recommendation is made based on these individual sections. Additional conclusions are then discussed, including justification for the recommendation, any subgroup considerations, implementation considerations, monitoring and evaluation, and future research priorities.

Step 10: Guidelines Manuscript

Once the Evidence to Decision tables have been finalized in GRADEpro, the Lead of the panel writes the initial draft of the Guidelines manuscript based on the EtDs. The guidelines are structured in the form of the KQs, from the systematic reivew, followed by the recommendations made by the expert panel. All guidelines strive to follow the same format and variability between guidelines is kept to a minimum [13,14]. See Figure 2 for a process overview. While panelists then edit the document, separate members of the Guidelines Committee assess the guideline using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) Instrument [15]. The AGREE II tool systematically measures the transparency and methodological precision used to create a guideline. It comprises of 23 key items within six domains with rwo additional global rating items.

Step 11. Review by Chairs/Panel and Incorporation of Edits

The Guidelines draft is reviewed by the committee Chair, Co-chairs and panel. Once reviewed and approved by the panel and committee chairs, the manuscript is sent for review and approval by the Guidelines Committee members. The lead incorporates suggested edits from the committee.

Step 12. Committee and Board Vote

After Guidelines Committee approval, manuscripts are then sent to the SAGES Executive Committee for comment and approval. After the Executive approval, manuscripts then go to the full SAGES Board for review and comment. The authors then incorporate all suggested edits and a final Board vote is obtained. Also, any joint guidelines, created in conjunction with other societies, are sent to their comparative boards for co-endorsement.

Step 13. Member and Public Review

Once approved by the Board and edited by the author group, Guidelines are uploaded onto the SAGES website by the SAGES webmaster for review and comments by the entire SAGES membership for a one-month period.

Step 14. End Product Publication

Any public comments are addressed, and the Guideline is finalized and submitted to Surgical Endoscopy by the Lead with help from SAGES staff. After publication, final Guidelines are also published on the SAGES website.

Living Guidelines

SAGES is committed to delivering recommendations that reflect the most up to date research. As such, the committee regularly updates their guidelines with the assistance of a new Living Guidelines task force. At regular intervals, a working group reviews the literature using the same process as described above. If enough high-quality evidence has been published since the original guideline, then a new systematic review will also be developed in addition to the guideline update. The future direction of this task force is a literature search every six months and, contingent on the results, consideration of a guideline update would occur at that time.

DISCUSSION

Guidelines are critical to the practicing physician. Every year the body of clinical publications increases substantially and it has become untenable for physicians to keep pace with all literature, while simultaneously maintaining a clinical practice. Guidelines help integrate a wide array of information into a cohesive and reliable manner. The ideal guideline is of exceptional quality and highly visible. However, the development of guidelines takes an exceptional amount of time, effort, and capital. Societies that are made up of volunteers must figure out a way to combat these issues without compromising their quality. SAGES has developed a standardized approach to train volunteer surgeons and trainees alike to develop clinically pertinent guidelines in a timely manner, without sacrificing quality. This methodology can be used more widely by volunteer organizations to efficiently develop effective tools for practicing physicians.

Figure 1. Example of PRISMA flow chart [3]

Figure 2. SAGES Guidelines Process

Appendix 1:

Databases used for literature search

Primary Resources:

1. MEDLINE
   • MEDLINE consists of biomedical and life sciences citations indexed by the National Library of Medicine (NLM) and includes 27.5 million citations from 5274 regularly indexed journals using MeSH (Medical Subject Headings)
   • PubMed (http://pubmed.gov/): Free search interface created by the NLM searches 31 million indexed and non-indexed citations from over 37,500 journals (1781-present)
   • Ovid MEDLINE (https://www.ovid.com/) (1946-present) and EBSCO MEDLINE (https://www.ebsco.com/products/research-databases/medline) (1809-present) are subscription interfaces using NLM data

2. Embase (https://www.embase.com/)
   • Subscription database of 32 million records of biomedical and pharmacological articles (1947-present) and conference abstracts (2009-present)
   • Search 8500 journals including MEDLINE titles and 2900 uniquely indexed using the EMTREE indexing system

3. Cochrane Library (https://www.cochranelibrary.com/)
   • Focused on informing healthcare decision-making, the free search interface accesses nearly 1.7 million items from six databases including: Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Database of Systematic Reviews (CDSR)
   • Search results include trials (from Clinicaltrials.gov and ICTRP), articles (Embase, PubMed, CINAHL), and Cochrane publications (reviews, protocols, etc.); some items are available only by subscription

4. Clinicaltrials.gov (https://clinicaltrials.gov)
   • Nearly 350,000 freely accessible clinical trials in all phases from 216 countries; search by disease/condition, intervention, or other terms (2000-present)
   • Trials may include related article citations linked to PubMed (either supporting articles or publications of results) and/or raw study results

5. ICTRP: International Clinical Trials Registry Platform (https://www.who.int/ictrp/en/)
   • Over 500,000 freely accessible registered trials with links to individual country clinical trial platforms, but few linked articles or study results (2005-present)
   • Use the trial ID(s) to search Google Scholar and/or PubMed for related articles

Specialty Databases

1. CINAHL: Cumulative Index to Nursing and Allied Health Literature (https://www.ebscohost.com/nursing/products/cinahl-databases)
   • Subscription databases focused on nursing, alternative/complementary medicine, biomedicine, and physical and occupational Search 760–5,500 journals (depending on product) indexed with CINAHL Subject Headings (1937-present)

2. PsycINFO (https://www.apa.org/pubs/databases/psycinfo/)
   • American Psychological Association (APA) subscription database focused on social, psychological and behavioral sciences. Search 4.9 million articles and book chapters indexed with the APA Thesaurus of Psychological Index Terms (1700s-present)

3. LILACS: Latin American and Caribbean Health Sciences Literature (https://lilacs.bvsalud.org/en/)
   • WHO PAHO (World Health Organization Pan-American Health Organization) offers this free search interface of nearly 1 million health science records from 900 journals published in Latin America and Caribbean countries and indexed with Descriptors in Health Sciences (DeCS), an extended translation of MeSH. Languages: Portuguese, Spanish and English (1982-present)

4. ERIC: Education Resources Information Center (https://eric.ed.gov/)
   • This free Institute of Education Services sponsored database includes 1.7 million records focused on education research from 1200 journals and 700 government and non-profit sources. Descriptor and identification terms added. (1966-present). ERIC (EBSCO) and ERIC (ProQuest) are subscription databases accessing ERIC data.

5. Theses and Dissertations
   • Open Access Theses and Dissertations (OATD) (https://oatd.org/): Free search of over 5.1 million publications from 1100 international institutions (1734-present)
   • ProQuest Dissertations and Theses Global (https://proquest.libguides.com/pqdt) Subscription database searching over 5 million international records (1637-present)

6. International HTA (Health Technology Assessment) Database (https://database.inahta.org/)
   • Free interface searching 17,000 HTAs indexed with MeSH and providing links to the published reports (1989-Present)

7. FDA Manufacturer and User Facility Device Experience (MAUDE) database (https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfMAUDE/search.CFM)
   • This Food and Drug Administration (FDA) database contains over 9.2 million reports on medical devices that malfunctioned or caused death or serious injury submitted by manufacturers, voluntary reporters. Related databases are linked (1993-present)

Additional Resources and Portals

1. TRIP (Turning Research Into Practice) Medical Database (https://www.tripdatabase.com/)
   • Free search engine focused on clinical research retrieving over 2.5 million articles from free online resources and sorted by study type. Optional subscription includes additional clinical trials, medical images and systematic reviews

2. VHL (Virtual Health Library) Regional Portal: https://bvsalud.org/en/
   • Sponsored by WHO PAHO, this free interface searches across multiple Pan-American databases including LILACS, Bibliographic Index Spanish in Health Sciences (IBECS) and the National Bibliography of Health Sciences (BINACIS)

3. DOAJ: Directory of Open Access Journals (https://doaj.org/)
   • A free portal searching 15,000 international and interdisciplinary peer reviewed journals and over 5.1 million articles from 133 countries. Subject indexing added to journals which must meet strict guidelines to remain included (1874-present)

4. Google Scholar (https://scholar.google.com/)
   • • Free interface searching freely accessible scholarly literature across the web including online journal citations, citation services (e.g., PubMed), pre-print servers, and patents. No indexing; uses only author explicit terms. Automated citation analysis.

   <

5. Web of Science (https://clarivate.com/webofsciencegroup/)
   • Subscription database searching 171 million records from 34,600 journals in the natural and biomedical sciences. Provides links to items citing a selected article (times cited), the article’s cited references, and author h-indexes (1800-present)

6. Scopus (https://www-elsevier-com.proxy.ulib.uits.iu.edu/solutions/scopus)
   • Subscription database focused on sciences and social sciences. Search over 75 million records from 38,000 journals, book chapters, and conference papers. Provides links to items citing a selected article (cited by) and the article’s references (1788-present)

7. AHRQ: Agency for Healthcare Research and Quality (https://www.ahrq.gov/)
   • AHRQ collects data and publications in multiple topic areas; databases include: Systematic Review Data Repository (SRDR), Medical Expenditure Panel Survey (MEPS) and National Healthcare Quality and Disparities Reports

Note: Medical libraries may provide access to subscription databases.

Appendix 2:

Modified Newcastle-Ottawa Scale

Corresponding Author

Amelia Rogers
Louisville, KY

Author Disclosures:

Amelia T. Rogers, MD – salary support from SAGES for role as Guidelines Fellow; Rebecca Dirks, MD, MS – SAGES paid for travel to the G-I-N (guidelines international network) conference last year; hold stock in Johnson and Johnson, unrelated to current project; Holly Ann Burt, MLIS – Contracted by SAGES as a medical librarian; Stephen Haggerty, MD – No disclosures; Geoffrey P. Kohn, MBBS, MSurg – Paid by Avant Law Australia to provide expert opinion reports in the field of upper GI surgery, unrelated to this manuscript; Bethany J. Slater, MD, MBA – receives consulting fees from Boulder Surgical; Danielle Walsh, MD – unpaid member of Society of Gastrointestinal and Endoscopic Surgeons Board of Governors – Member at large; American College of Surgeons Board of Governors; American Academy of Pediatrics Surgery Committee; Dimitrios Stefanidis, MD, PhD – no disclosures; Aurora Pryor, MD, MBA – receives honoraria for speaking from Ethicon, Gore, Medtronic, Merck and Stryker

References

1. Moher D, Tetzlaff J., Tricco AC, Sampson M, Altman DG (2007) Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4:e78
2. World Health Organization (WHO). (2010) WHO handbook for guideline development. March 2010. WHO, International Repository for Information Sharing, Geneva. Available at: https://www.who.int/hiv/topics/mtct/grc_handbook_mar2010_1.pdf
3. Institute of Medicine (2011) Clinical practice guidelines we can trust: standards for developing trustworthy clinical practice guidelines (CPGs), The National Academies Press, Washington, DC
4. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339:b2535
5. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ, Group GW (2008) GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 336:924-926
6. Dirks RC, Walsh D, Haggerty S, Kohn GP, Pryor A, Stefanidis D. (2021) SAGES guidelines: an appraisal of their quality and value by SAGES members. Surg Endosc. 2021 Feb 2. doi: 10.1007/s00464-021-08323-2
7. Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA (2012) Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, Association for Computing Machinery, Miami, Florida, pp 819–824
8. Covidence systematic review software. Veritas Health Innovation, Melbourne, Australia. Available at: http://covidence.org
9. Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Emberson JR, Hernan MA, Hopewell S, Hrobjartsson A, Junqueira DR, Juni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898
10. Wells GA Shea B, O’Connell D, Peterson J, Welch V, Losos M, Tugwell P (2019) The Newcastle-Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses. Ottawa Hospital, Ottawa, Canada. Available at: http://wwwohrica/programs/clinical_epidemiology/oxford.asp
11. Review Manager (RevMan) (2014), [Software]. The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen
12. GRADEpro GDT: GRADEpro Guideline Development Tool [Software]. McMaster University, 2015 (developed by Evidence Prime, Inc.), Available at: https://org
13. Michael Brunt L, Deziel DJ, Telem DA, Strasberg SM, Aggarwal R, Asbun H, Bonjer J, McDonald M, Alseidi A, Ujiki M, Riall TS, Hammill C, Moulton CA, Pucher PH, Parks RW, Ansari MT, Connor S, Dirks RC, Anderson B, Altieri MS, Tsamalaidze L, Stefanidis D; Prevention of Bile Duct Injury Consensus Work Group (2020) Safe cholecystectomy multi-society practice guideline and state-of-the-art consensus conference on prevention of bile duct injury during cholecystectomy. Surg Endosc. 34:2827-2855.
14. Kohn GP, Dirks RC, Ansari MT, Clay J, Dunst CM, Lundell L, Marks JM, Molena D, Rooker C, Saxena P, Swanstrom L, Wong RK, Pryor AD, Stefanidis D (2021) Guidelines for the use of peroral endoscopic myotomy (POEM) for the treatment of achalasia, Surg Endosc. In press.
15. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008