|
|
--- |
|
|
license: gpl-3.0 |
|
|
language: |
|
|
- nl |
|
|
base_model: |
|
|
- CLTL/MedRoBERTa.nl |
|
|
tags: |
|
|
- medical |
|
|
- healthcare |
|
|
metrics: |
|
|
- perplexity |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
Continued, off-premise, pre-training of [MedRoBERTa.nl](https://huggingface.co/CLTL/MedRoBERTa.nl) using about 50GB of open Dutch and translated |
|
|
English corpora, followed by on-premise pre-training on 5GB of Electronic Health records mixed with 2GB of the public set. |
|
|
|
|
|
# Data statistics |
|
|
|
|
|
Sources: |
|
|
* Dutch: medical guidelines (FMS, NHG) |
|
|
* Dutch: [NtvG](https://www.ntvg.nl/) papers |
|
|
* Dutch: Cardiovascular Electronic Health Records |
|
|
* English: Pubmed abstracts |
|
|
* English: PMC abstracts translated using DeepL |
|
|
* English: Apollo guidelines, papers and books |
|
|
* English: Meditron guidelines |
|
|
* English: MIMIC3 |
|
|
* English: MIMIC CXR |
|
|
* English: MIMIC4 |
|
|
|
|
|
All translated (if not with DeepL) with a combination of GeminiFlash 1.5/2.0/GPT4o mini, MariaNMT, NLLB200. |
|
|
|
|
|
* Number of tokens: 20B |
|
|
* Number of documents: 32M |
|
|
|
|
|
# Training |
|
|
|
|
|
* Effective batch size: 5120 |
|
|
* Learning rate: 2e-4 |
|
|
* Weight decay: 1e-3 |
|
|
* Learning schedule: linear, with 5_000 warmup steps |
|
|
* Num epochs: ~3 (off-premise) followed by 3 (on-premise) |
|
|
|
|
|
Train perplexity: 2.4 |
|
|
Validation perplexity: 3.3 |
|
|
|
|
|
# Acknowledgement |
|
|
|
|
|
This work was done together with the Amsterdam UMC, in the context of the [DataTools4Heart](https://www.datatools4heart.eu/) project. |
|
|
|
|
|
We were happy to be able to use the [Google TPU research cloud](https://sites.research.google/trc/about/) for training the model. |
|
|
|
|
|
|
