(1362) Utilizing Large Language Models for Disease Phenotyping in Obstructive Sleep Apnea

Introduction: Obstructive Sleep Apnea (OSA) affects millions in the U.S. and is linked to severe illnesses. However, there lacks understanding on how OSA interacts with common comorbidities and its additive risk in developing severe complications. This study focuses on Large Language Model (LLM) embeddings of clinical reports to model medical phenotypes and measures of health. To develop tools for deep phenotyping OSA, we aim to score clinical LLMs on their ability to cluster documents by 6 OSA-related phenotypes.

Methods: We computed document-level embeddings for 331,793 discharge reports from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database using the Perlmutter supercomputer. Twelve clinical LLMs were evaluated including BioBART, BioBERT, BioGPT, BioMegatron, Gatortron, and RadBERT. Embeddings were clustered using K-Means. Then, the purity of clusters was measured by Shannon entropy. Additionally, we assessed the quality of each model by visualizing its latent space with UMAP and manually reading sampled clinical text from clusters of interest.

Results: Among the evaluated LLMs, Gatortron_base's second to last layer consistently achieved low entropy scores along K-means clusters (k=1024). This performance was observed when discriminating OSA from comorbid phenotypes including atrial fibrillation, coronary artery disease, heart failure, hypertension, stroke (broad definition), and type 2 diabetes mellitus. When ranking clusters by class purity, clinical notes were sampled (n=50) from clusters of high rates of OSA patients with heart failure, describing admissions of patients with a history of OSA, dyspnea as a chief complaint, and no prescription/adherence to Continuous Positive Airway Pressure (CPAP) treatment.

Conclusions: Leveraging LLMs for feature discovery, this work serves as a stepping stone towards understanding how statistical language models organize clinical information and if relations between latent representations capture phenotypes. Identifying sub-cohorts of OSA patients at risk of severe complications is crucial for optimizing the allocation of costly CPAP treatment. Even though LLMs have yet to scale in the narrow domain of clinical medicine, our results show models trained on larger sets of data perform better at organizing corpora by clinically relevant measures.