Data from this project has be released as Aʙᴀᴄᴜs-212 (49 calculation tasks, 212 vignettes) and Aʙᴀᴄᴜs-409 (10 calculation tasks, 10,000 vignettes). The cleaned datasets are available in the datasets directory. The other code here is the experimental code for our paper. A clean repository for the web tool component of OpenMedCalc is available in a separate OpenMedCalc repository. Note that some of the data in this repository is not covered by the MIT license of this repository (see "external data" section below).

See our paper in NPJ Digital Medicine, available here: Large Language Model Agents Can Use Tools to Perform Clinical Calculation.

Large language models (LLMs) can answer expert-level questions in medicine but are prone to hallucinations and arithmetic errors. Early evidence suggests LLMs cannot reliably perform clinical calculations, limiting their potential integration into clinical workflows. We evaluated ChatGPT’s performance across 48 medical calculation tasks, finding incorrect responses in one-third of trials (n = 212). We then assessed three forms of agentic augmentation: retrieval-augmented generation, a code interpreter tool, and a set of task-specific calculation tools (OpenMedCalc) across 10,000 trials. Models with access to task-specific tools showed the greatest improvement, with LLaMa and GPT- based models demonstrating a 5.5-fold (88% vs 16%) and 13-fold (64% vs 4.8%) reduction in incorrect responses, respectively, compared to the unimproved model. Our findings suggest that integration of machine-readable, task-specific tools may help overcome LLMs’ limitations in medical calculations.

This project explores the augmentation of large language models (LLMs) like ChatGPT with clinician-informed tools to improve performance on medical calculation tasks. The project addresses the limitations of LLMs in performing basic mathematical operations and their tendency to hallucinate knowledge. By integrating an open-source clinical calculation API, OpenMedCalc, with ChatGPT, the project demonstrates significant improvements in accuracy for clinical calculations. This integration allows for the execution of common clinical calculations with enhanced reliability, aiming to revolutionize medical practice by automating routine tasks and providing accurate, evidence-based calculations to clinicians.

Large language models (LLMs) such as ChatGPT have shown the ability to answer expert-level multiple-choice questions in medicine but are limited by their tendency to hallucinate knowledge and inadequacy in performing basic mathematical operations. This project explores the ability of ChatGPT to perform medical calculations, evaluating its performance across diverse clinical calculation tasks. Initial findings indicated that ChatGPT is an unreliable clinical calculator, delivering inaccurate responses in a significant number of trials.

To address the limitations of LLMs, the project developed an open-source clinical calculation API, OpenMedCalc, which was integrated with ChatGPT. The augmented model was evaluated against standard ChatGPT using clinical vignettes in common clinical calculation tasks. The goal was to enhance the accuracy and reliability of medical calculations performed by LLMs.

• The integration of OpenMedCalc with ChatGPT significantly improved the accuracy of clinical calculations.
• The augmented model demonstrated a marked improvement in accuracy over unimproved ChatGPT.
• The project highlights the potential of integrating machine-usable, clinician-informed tools to alleviate the reliability limitations observed in medical LLMs.

LLM Calc is a comprehensive tool designed to facilitate medical calculations using large language models (LLMs). It integrates various clinical calculators and provides a user-friendly interface for performing complex medical computations. The project aims to enhance the accuracy and efficiency of medical calculations by leveraging the power of LLMs.

In this revision, we have expanded the range of models and calculators used in the project:

• gpt4o
• llama3_1 (provided by OpenRouter)

• psiport
• ariscat
• cci
• caprini
• gad7
• sofa
• meldna
• hasbled
• wellsdvt

• Interactive Menu: Navigate through the application using a simple command-line interface.
• Database Management: Easily rebuild and manage the database of medical calculators.
• Vignette Generation: Automatically generate clinical vignettes for testing and demonstration purposes.
• Configuration Viewing: View and customize the configuration settings of the application.
• Testing Suite: Run comprehensive tests to ensure the reliability and accuracy of the calculations.

To use LLM Calc, you can execute the following commands:

Usage: llmcalc [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion          Install completion for the current shell.
  --show-completion             Show completion for the current shell, to copy it or customize the installation.
  --help                        Show this message and exit.

Commands:
  interpreter        Start an iPython interpreter in the current context.
  rebuild-database   Rebuild the database.
  test               Run tests.
  view-config        View config page
  vignettes          Build the vignettes.
  experiment         Manage and run experiments.

1. Installation: You will need to have uv installed. Clone the repository and install the required dependencies, build package using: uv sync. This will install a command line application called llmcalc in the UV environment.
2. Configuration: Set up the environment variables and configuration files as needed. See sameple_env.txt for reference. You will need an account with OpenAI and OpenRouter. Some functionality will require a LangChain API key as wel.
3. External data: Review the "external data" section below and copy files as needed.
4. Execution: Use the command-line interface to interact with the application and perform calculations. To run the full workflow, you will need to run the following commands:

uv run llmcalc rebuild-database

This takes the vignette data, calculators, arms, etc and places it into a database.

uv run llmcalc experiment rebuild

This prepares the system to run an experiment. After, you can start an experiment with the following line. Note that "number of cases" is on a per-calculator per-arm basis, so depending on configuration, can be many cases.

uv run llmcalc experiment new  --description "Evaluating 10 vignettes" --number-of-cases 1

This builds the experiment, including the actual vignettes/cases. A file called cases.json will be output into the data/build_database folder after this step; this contains the synthetic patients information. It will also start the experiement, running the vignettes against the calculators listed in the llmcalc/lib/config.py file. That file contains the following lines, where one can specify the configuration of the experiement.

# Calculators
self.DEFAULT_SELECTED_CALCULATORS_SLUGS = [
CalculatorSlug.nihss,
CalculatorSlug.hasbled,
CalculatorSlug.meldna,
CalculatorSlug.gad7,
CalculatorSlug.sofa,
CalculatorSlug.psiport,
CalculatorSlug.wellsdvt,
CalculatorSlug.caprini,
CalculatorSlug.cci,
CalculatorSlug.ariscat,
]

# arms
self.DEFAULT_SELECTED_ARM_SLUGS = [
ArmSlug.llama_base,
ArmSlug.llama_ci,
ArmSlug.llama_rag,
ArmSlug.llama_rag_ci,
ArmSlug.llama_omc,
ArmSlug.gpt4_base,
ArmSlug.gpt4_ci,
ArmSlug.gpt4_rag,
ArmSlug.gpt4_rag_ci,
ArmSlug.gpt4_omc,
]

We welcome contributions from the community. Please feel free to submit issues, fork the repository, and make pull requests.

Works citing this study should cite:

@article{goodell2025,
  title         = {Large Language Model Agents Can Use Tools to Perform Clinical Calculations},
  author        = {Goodell, Alex J. and Chu, Simon N and Rouholiman, Dara and Chu, Larry F},
  year          = {2025},
  month         = {Feb},
  journal       = {NPJ Digital Medicine},
  doi           = {10.1038/s41746-025-01475-8},
}

Evaluation of the model (RAG arm) included usage of published written material owned by MDCalc Ltd (New York, NY); used and reproduced here with their permission. MDCalc reserves all rights to said material. For this reason, it is stored in the external_data directory. This data is not included in the MIT license.

To re-run the original experiments, you are required to move the data from the external_data directory, which can be accomplished by running the following from the root directory:

mkdir llm_calc/data/build_database/reference_material
cp -r external_data/* llm_calc/data/build_database/reference_material

If you have any other difficulties getting the code running, please reach out to us (using email in our paper).

Apart from the copyrighted material found in the external_data directory (see "External data" section above) and relevant transitory requirements (in table below), this software is licensed under the MIT license.

This Software contains python dependencies which are not property of the authors. Please see the following table of dependencies and their licenses (built with licensecheck).

If there are any questions about usage or licensing, please reach out over email in the email listed for the corresponding author in our article PDF.