| Description |
Instruction |
Short Instruction |
| Full conversion |
Convert this page to docling. |
- |
| Chart |
Convert chart to table. |
<chart> |
| Formula |
Convert formula to LaTeX. |
<formula> |
| Code |
Convert code to text. |
<code> |
| Table |
Convert table to OTSL. (Lysak et al., 2023) |
<otsl> |
| Actions and Pipelines |
OCR the text in a specific location: <loc_155><loc_233><loc_206><loc_237> |
- |
| Identify element at: <loc_247><loc_482><loc_252><loc_486> |
- |
| Find all 'text' elements on the page, retrieve all section headers. |
- |
| Detect footer elements on the page. |
- |
# Model Architecture:
The architecture of granite-docling-258m consists of the following components:
(1) Vision encoder: [siglip2-base-patch16-512](https://huggingface.co/google/siglip2-base-patch16-512).
(2) Vision-language connector: pixel shuffle projector (as in idefics3)
(3) Large language model: Granite 165M.
We built upon [Idefics3](https://huggingface.co/docs/transformers/en/model_doc/idefics3) to train our model. We incorporated DocTags into our LLMโs supervised fine-tuning (SFT) data to help the model become familiar with the format, enabling faster convergence and mitigating issues previously observed with SmolDocling.
The model was trained using the [nanoVLM](https://github.com/huggingface/nanoVLM) framework, which provides a lightweight and efficient training setup for vision-language models
**Training Data**: Our training corpus consists of two principal sources: (1) publicly available datasets and (2) internally constructed synthetic datasets designed to elicit specific document understanding capabilities.
In particular, we incorporate:
* [**SynthCodeNet**](https://huggingface.co/datasets/ds4sd/SynthCodeNet) โ a large-scale collection of synthetically rendered code snippets spanning over 50 programming languages
* [**SynthFormulaNet**](https://huggingface.co/datasets/ds4sd/SynthFormulaNet) โ a dataset of synthetic mathematical expressions paired with ground-truth LaTeX representations
* [**SynthChartNet**](https://huggingface.co/datasets/ds4sd/SynthChartNet) โ synthetic chart images annotated with structured table outputs
* [**DoclingMatix**](https://huggingface.co/datasets/HuggingFaceM4/DoclingMatix) โ a curated corpus of real-world document pages sampled from diverse domains
**Infrastructure**: We train granite-docling-258m using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable and efficient infrastructure for training our models over thousands of GPUs.
**Responsible Use and Limitations** Some use cases for Vision Language Models can trigger certain risks and ethical considerations, including but not limited to: bias and fairness, misinformation, and autonomous decision-making.
Although our alignment processes include safety considerations, the model may in some cases produce inaccurate, biased, offensive or unwanted responses to user prompts. Additionally, whether smaller models may exhibit increased susceptibility
to hallucination in generation scenarios due to their reduced sizes, which could limit their ability to generate coherent and contextually accurate responses, remains uncertain. This aspect is currently an active area of research,
and we anticipate more rigorous exploration, comprehension, and mitigations in this domain. We urge the community to use granite-docling-258m in a responsible way and avoid any malicious utilization. We recommend using this model only as part of the Docling library.
More general vision tasks may pose higher inherent risks of triggering unwanted output. To enhance safety, we recommend using granite-docling-258m alongside Granite Guardian. Granite Guardian is a fine-tuned instruct model designed to detect and flag risks in prompts and responses across key dimensions outlined in the IBM AI Risk Atlas.
Its training, which includes both human-annotated and synthetic data informed by internal red-teaming, enables it to outperform similar open-source models on standard benchmarks, providing an additional layer of safety.
**Resources**
- โญ๏ธ Learn about the latest updates with Docling: https://docling-project.github.io/docling/#features
- ๐ Get started with Docling concepts, integrations and tutorials: https://docling-project.github.io/docling/getting_started/
- ๐ก Learn about the latest Granite learning resources: https://ibm.biz/granite-learning-resources
- ๐ฅ๏ธ Learn more about how to use Granite-Docling, explore the Docling library, and see whatโs coming next for Docling in the release blog: https://ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
## Troubleshooting
**Running with VLLM**
1. You receive `AttributeError: 'LlamaModel' object has no attribute 'wte'` when launching the model through VLLM.
With current versions of VLLM (including 0.10.2), support for tied weights as used in granite-docling is limited and breaks. We provide a version with untied weights on the `untied` branch of this model repo.
To use the untied version, please pass the `revision` argument to VLLM:
```sh
# Serve the model through VLLM
$> vllm serve ibm-granite/granite-docling-258M --revision untied
```
```python
# If using the VLLM python SDK:
from vllm import LLM
...
llm = LLM(model=MODEL_PATH, revision="untied", limit_mm_per_prompt={"image": 1})
```
2. The model outputs only exclamation marks (i.e. "!!!!!!!!!!!!!!!").
This is seen on older NVIDIA GPUs, such as the T4 GPU available in Google Colab, because it lacks support for `bfloat16` format.
You can work around it by setting the `dtype` to `float32`.
```sh
# Serve the model through VLLM
$> vllm serve ibm-granite/granite-docling-258M --revision untied --dtype float32
```
```python
# If using the VLLM python SDK:
from vllm import LLM
...
llm = LLM(model=MODEL_PATH, revision="untied", limit_mm_per_prompt={"image": 1}, dtype="float32")
```
