UPDATE: New Model Announcement

We are proud to introduce Asclepius, a more advanced clinical large language model. As this model was trained on synthetic clinical notes, it is publicly accessible via Huggingface. If you are considering using CAMEL, we highly recommend switching to Asclepius instead. For more information, please visit this link.

We present CAMEL, Clinically Adapted Model Enhanced from LLaMA. As LLama for its foundation, CAMEL is further pre-trained on MIMIC-III and MIMIC-IV clinical notes, and finetuned over clinical instructions (Figure 2). Our preliminary evaluation with GPT-4 assessment, demonstrates that CAMEL achieves over 96% of the quality of OpenAI’s GPT-3.5 (Figure 1). In accordance with the data usage policies of our source data, both our instruction dataset and model will be published on PhysioNet with credentialized access. To facilitate replication, we will also release all code, allowing individual healthcare institutions to reproduce our model using their own clinical notes.

Overview

Large language models, such as OpenAI’s GPT-series, have demonstrated promising results in tackling clinical tasks, including entity extraction, de-identification, and medical record summarization. Despite their potential, the API-based nature of these models introduces significant privacy concerns, limiting their widespread adoption in real-world healthcare settings. Healthcare professionals require a model that can support their clinical decision-making while alleviating privacy concerns.

In response to the growing demand for privacy-preserving AI solutions in healthcare, we release CAMEL-7B, a new clinical large language model that has been further pretrained with clinical notes (3.4B tokens) and finetuned on 100K clinical instructions over LLaMa (Figure 2). Our results show that CAMEL-7B can achieve similar performance (96% score measured by GPT4) to GPT-3.5 on 13 tasks over previously unseen clinical notes.

In this study, we utilized both MIMIC-III, MIMIC-IV, and i2b2 notes to pretrain the model and generate instruction-following dataset. In compliance with the usage regulations outlined by PhysioNet, both the dataset employed for model training and the resulting model weights will be uploaded to their platform for credentialized access.

Instruction Generation

In clinical domain, particularly concerning clinical notes, there exists a set of tasks that healthcare professionals frequently ask for. In this study, we introduce 13 pre-determined natural language processing (NLP) tasks specifically designed for application to clinical notes. These tasks were derived from a survey conducted on clinical NLP tasks in the UK between 2007 and 2022, and by examining the tools employed in the healthcare industry for clinical practices. The following table delineates each of the 13 task types.

Drawing upon the inspiration from Alpaca’s approach to instruction generation utilizing Text-davinci-003, we adapted this methodology for the generation of clinical instructions. We constructed a dataset consisting of 9,000 clinical discharge summaries, sourced from the MIMIC-III, MIMIC-IV, and i2b2 repositories. In adherence to the data usage regulations prescribed by Physionet, we utilized Microsoft Azure’s HIPAA-compliant GPT-3.5 language model to produce a single question for each pre-defined task and corresponding answer. The final instruction dataset encompasses 100k instructions, generated at a cost of under $100 via the OpenAI API. To facilitate replication of our work by clinical institutions with their own clinical notes, we have made the requisite code accessible through the following link.

Training

Clinical Notes Pre-training

Recent study has explored the enhancement of the LLaMA model by employing additional casual language model pretraining on PubMed articles with the aim of injecting biomedical knowledge into the model. Similarly, to further familiarize our model with the unique characteristics of clinical notes, such as the prevalent use of abbreviations, non-standard grammar, and unexplained jargon, further pretraining was conducted on the entire MIMIC-III (1.3 billion tokens) and MIMIC-IV (2.1 billion tokens) notes. This process took about 200 hours on eight A6000 48GB GPU. The effectiveness of clinical pretraining will be demonstrated in the evaluation results presented below.

Clinical Instruction Finetuning

After pretraining the model with clinical notes, we further finetune the model with instructions that the healthcare professionals might ask about. We employed the training schema devised by Vicuna, which builds upon Alpaca framework. This adaptation involved extending the maximum context length to 2,048 tokens, while also incorporating gradient checkpointing mechanisms to optimize memory usage. For a comprehensive understanding of the methodology, we kindly direct readers to consult the respective repositories of Alpaca and Vicuna.

Preliminary Evaluation

For the fair evaluation of CAMEL-7B, we utilized MTSamples (a publically available clinical note dataset designated for research and educational purposes) that had not been used during the training process. We constructed an evaluation dataset by extracting 1,000 question-answer pairs from 108 MTSamples discharge summary notes. Informed by the evaluation approach in Vicuna, we asked GPT-4 to rate the responses of models (Alpaca, CAMEL, and GPT-3.5) on a scale of 1 to 10, based on the relevance and accuracy of their responses (which costs less than 50$). Additionally, we present GPT-4’s responses for ease of comparison (refer to the three examples provided below).

Examples

Named Entity Recognition

Instruction : Identify all the medications mentioned in the summary.

Alpaca (Assistant 1) (6/10) : The medications mentioned in the summary are ampicillin, gentamicin, and phototherapy.

CAMEL (Assistant 2) (10/10) : The medications mentioned in the summary are betamethasone, penicillin, ampicillin, and gentamicin.

GPT-4’s Answer : Betamethasone, Penicillin, Ampicillin, Gentamicin

• Alpaca did not detect Betamethasone while CAMEL did
• Phototheray is a type of treatment

Natural Language Generation

Instruction : Can you generate a recommendation for the patient’s postoperative care?

Alpaca (Assistant 1) (7/10) : The patient should be instructed to follow up with her primary care physician for any ongoing medical concerns and to have her staples removed on 6/17/2009. She should also be advised to continue taking her medications as prescribed, including Motrin 600 mg tablets 1 tablet p.o. q.8h. p.r.n. pain, Colace 100 mg tablets 1 tablet p.o. daily p.r.n. constipation and ferrous sulfate 60 mg tablets 1 tablet p.o. daily, and multiple vitamin 1 tablet p.o. daily.

CAMEL (Assistant 2) (9/10) : After the patient’s total abdominal hysterectomy (TAH), it is recommended that she follows regular diet, takes bedrest for one week with slow return to normal activity over the ensuing 4 to 6 weeks, and maintains pelvic rest for 6 weeks. Additionally, she should take Motrin 600 mg tablets 1 tablet p.o. q.8h. p.r.n. pain, Colace 100 mg tablets 1 tablet p.o. daily p.r.n. constipation, and ferrous sulfate 60 mg tablets 1 tablet p.o. daily, and multiple vitamin 1 tablet p.o. daily. The patient should also return on Wednesday 6/17/2009 for removal of staples.

GPT-4’s Comment : Assistant 2’s response was more accurate and complete, and closely reflected the information given in the clinical note while Assistant 1’s answer was similar but slightly less detailed

Question Answering

Instruction : Was there any abnormal finding in the patient’s abdomen on physical examination?

Alpaca (Assistant 1) (2/10) : Yes, there was an abnormal finding in the patient’s abdomen on physical examination, which was a calcified old granulomatous disease.

CAMEL (Assistant 2) (10/10) : No, there were no abnormal findings in the patient’s abdomen on physical examination.

GPT-4’s Answer : No, there were no abnormal findings in the patient’s abdomen on physical examination.

CAMEL v.s. GPT-3.5 v.s. Alpaca

As shown in Figure 3, in 80% of the cases, CAMEL demonstrated similar or superior performance compared to Alpaca. In perspective of average scores, Alpaca exhibited 76% of GPT-3.5’s performance, whereas CAMEL achieved 96%.

Ablation Study

Is clinical note pretraining needed?

To validate the efficacy of clinical note pretraining, we compare CAMEL (note pretrained + instruction finetuned over LLaMA) to CAMEL without note pretraining (only instruction finetuned over LLaMA). As in the table above, we can observe that the score decreased without clinical note pretraining. However, the difference is not substantial, and we believe that the clinical notes from the MTSample notes are written with plain languages compared to MIMIC notes.

Future work

1. Of the various types of clinical notes, such as nursing notes, physician notes, and radiology notes, only discharge summaries were employed in our work. We plan to expand the scope of the data in the near future.
2. We plan to upscale the model size using LLaMa-13B and beyond.
3. Beyond GPT4 evaluation, a more refined evaluation of the model will be conducted by getting feedback from actual medical professionals.
4. As this study demonstrated that the enhanced performance was observed when finetuning with answers provided by GPT-4, we intend to conduct a similar experiment employing a HIPAA-certified GPT-4 model.
5. Rather than merely engaging in a question-and-answer format with the model, our research aims to expand its capabilities by enabling dialog with the model.

Limitations

We plan to update limitations and license after uploading the model on physionet.

Team

Authors

Sunjun Kweon*, Junu Kim*, Seongsu Bae**, Eunbyeol Cho**, Sujeong Im**, Jiyoun Kim**, Gyubok Lee**, JongHak Moon** and JeongWoo Oh**

* equal contribute
** alphabetical order

Advisor

Edward Choi

Citation

@misc{CAMEL,
    title = {CAMEL : Clinically Adapted Model Enhanced from LLaMA},
    author = {Sunjun Kweon and Junu Kim and Seongsu Bae and Eunbyeol Cho and Sujeong Im and Jiyoun Kim and Gyubok Lee and JongHak Moon and JeongWoo Oh and Edward Choi},
    month = {May},
    year = {2023}
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/starmpcc/CAMEL}},
}