Evaluating the Accuracy of Language Models in Spine-Related Billing Codes

Introduction: Artificial intelligence (AI) has been increasingly applied in the field of spine surgery, including billing code generation. Automating Current Procedural Terminology (CPT) code generation will streamline the accuracy of medical billing, reducing the load on administrators. This research intends to explore the accuracy of four LLMs (Language Learning Models) in generating CPT billing codes given spine surgery operative notes.

Methods: Operative notes were deidentified (n=80) across eight different spine surgeries and were
collected from a single spine surgeon. Four LLMs—ChatGPT-4, ChatGPT-4o, Google Gemini
2.0 Flash, and SuperBot—were assessed through four different trials: (1) direct CPT code
generation, (2) example-based learning, (3) CPT code banks, and (4) exact CPT codes provided. Model accuracy was evaluated using precision, recall, and F1-scoring. ANOVA followed by Tukey HSD post-hoc analysis and the Kruskal-Wallis test with pairwise Wilcoxon tests were used to compare trials, LLMs, surgeries, and diagnoses. The significance threshold for the analysis was p < 0.05. The statistical analyses were conducted using R version 4.3.2.

Results: GPT-4 and GPT-4o significantly outperformed SuperBot and Gemini in multiple trials (p < 0.05). Furthermore, performance improved progressively across each trial, except Trial 3, where verification-based predictions were less accurate than example-based learning (p = 0.002). However, there were no significant differences in billing accuracy across different surgical procedures.

Conclusion : Artificial intelligence models, especially GPT-4 and GPT-4o, exhibit a strong capability in
generating CPT codes for spine surgery operative notes, utilizing structured learning approaches to improve accuracy further. These results point towards AI serving as an adjunct in medical billing, though further validation on more diverse datasets and real-world implementations is required.