UTILITY OF LARGE LANGUAGE MODELS (LLMS) IN PREOPERATIVE RISK STRATIFICATION: A COMPARATIVE STUDY WITH HUMAN ANESTHESIOLOGISTS

Abstract

Background: Preoperative risk stratification is critical for mitigating perioperative complications, yet it remains subject to inter-observer variability among clinicians. Large Language Models (LLMs) have demonstrated sophisticated reasoning in medical contexts, but their utility in synthesizing complex patient data for anesthesia risk assessment remains insufficiently validated against clinical expertise. This study evaluates the accuracy and consistency of LLMs compared to board-certified anesthesiologists in assigning risk scores and identifying perioperative red flags.

Methods: A comparative diagnostic study was conducted using 100 high-fidelity simulated patient vignettes representing diverse surgical subspecialties. Each vignette included medical history, physical examination findings, and laboratory data. Three state-of-the-art LLMs ( GPT-4, Claude 3.5, and Gemini Pro) and five board-certified anesthesiologists independently assigned American Society of Anesthesiologists (ASA) Physical Status scores and identified top-tier perioperative risks. The primary outcome was the level of agreement (Cohen’s Kappa) between LLMs and the "gold standard" expert consensus. Secondary outcomes included the sensitivity of LLMs in detecting "critical" contraindications (severe aortic stenosis).

Results: Preliminary analysis indicates that top-tier LLMs demonstrate "substantial" agreement (k > 0.70) with human experts in ASA classification. While LLMs excelled in extracting data from unstructured text, they exhibited a higher rate of "over-stratification" in low-risk cases compared to human clinicians. However, LLMs demonstrated a 98% sensitivity in identifying objective contraindications, occasionally outperforming humans in detecting minor laboratory abnormalities.

Conclusion: LLMs serve as a potent adjunct for preoperative screening, offering high sensitivity for risk factor identification. While they do not replace the clinical nuance of an anesthesiologist, their integration into Electronic Health Records could provide a valuable "safety net" for identifying high-risk patients and standardizing preoperative care.