File size: 13,697 Bytes
03e6f76 967d049 ef3fa61 03e6f76 967d049 03e6f76 967d049 ef3fa61 03e6f76 ef3fa61 03e6f76 ef3fa61 d51517f ef3fa61 03e6f76 ef3fa61 03e6f76 ef3fa61 03e6f76 381249b 967d049 381249b 03e6f76 967d049 21cbb17 03e6f76 967d049 03e6f76 967d049 03e6f76 967d049 03e6f76 967d049 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 |
---
base_model:
- DeepGlint-AI/rice-vit-large-patch14-560
- Qwen/Qwen3-4B-Instruct-2507
datasets:
- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
- lmms-lab/LLaVA-OneVision-1.5-Insturct-Data
- HuggingFaceM4/FineVision
library_name: transformers
license: apache-2.0
pipeline_tag: image-text-to-text
---
<div align="center">
<h1>LLaVA-OneVision-1.5: Fully Open-Source State-of-the-Art VLM Model</h1>
<p>
<a href="https://huggingface.co/papers/2509.23661">
<img alt="Paper" src="https://img.shields.io/badge/Paper-b31b1b?style=for-the-badge&logo=arXiv&logoColor=white">
</a>
<a href="https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5">
<img alt="Code" src="https://img.shields.io/badge/Code-181717?style=for-the-badge&logo=github&logoColor=white">
</a>
<a href="https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M">
<img alt="Mid-Training Dataset" src="https://img.shields.io/badge/Mid--Training%20Dataset-f59e0b?style=for-the-badge&logo=huggingface&logoColor=white">
</a>
<a href="https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Instruct-Data">
<img alt="Instruct Dataset" src="https://img.shields.io/badge/Instruct%20Dataset-3fb950?style=for-the-badge&logo=huggingface&logoColor=white">
</a>
<a href="https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5">
<img alt="Demo" src="https://img.shields.io/badge/Demo-1f6feb?style=for-the-badge&logo=huggingface&logoColor=white">
</a>
<a href="https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/tensorboard">
<img alt="TensorBoard" src="https://img.shields.io/badge/TensorBoard-FF6F00?style=for-the-badge&logo=tensorflow&logoColor=white">
</a>
</p>
</div>
## Introduction
LLaVA-OneVision-1.5 is a fully open-source family of large multimodal models (LMMs) built to democratize multimodal training. Trained on native‑resolution images, it delivers state‑of‑the‑art performance at substantially lower cost. The project also releases high‑quality pretraining and SFT data, a complete and efficient training framework with recipes and configs, and comprehensive logs to support transparent, reproducible research.
#### **Superior Performance**
- The model leads on multiple multimodal benchmarks and generally surpasses Qwen2.5-VL.
- Training on native-resolution images significantly improves its visual understanding.
#### **High-Quality Data at Scale**
- The pretraining corpus comprises large-scale, concept-balanced, diverse, and high-quality captions curated with strict filtering and quality control.
- The instruction-tuning dataset is comprehensive and covers a wide range of tasks.
#### **Ultra-Efficient Training Framework**
- The end-to-end training cost is about $16,000 on A100 GPUs at roughly $0.60 per GPU-hour.
- The system is built on Megatron-LM with support for MoE, FP8, and long-sequence parallelism, and the codebase is optimized for cost-effective scaling.
#### **Fully Open Framework**
- The project releases high-quality pretraining and SFT datasets along with the complete training framework, configurations, and recipes.
- It also provides detailed training logs and metrics to enable reproducibility and community adoption.
## Models
| Model | HF Link | Training Log |
|---|---|---|
| LLaVA-OV-1.5-4B-Instruct | [🤗 HF / 4B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct) | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-4B-Instruct/tensorboard) |
| LLaVA-OV-1.5-8B-Instruct | [🤗 HF / 8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct) | [📈 Tensorboard](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct/tensorboard) |
## Dataset
| Description | Link | Status |
|--------------------|--------------------------------------------------------------------------------------------------------|-------------|
| LLaVA-OneVision-1.5-Mid-Training-85M | [🤗HF / Mid-Training 85M](https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M) | Uploading… |
| LLaVA-OneVision-1.5-Instruct | [🤗HF / Instruct-Data](https://huggingface.co/datasets/mvp-lab/LLaVA-OneVision-1.5-Instruct-Data) | Available |
## Evaluation Results
All evaluations were conducted using [lmms_eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).
## Quick Start with HuggingFace
Here we show a code snippet to show you how to use the chat model with `transformers` and `qwen_vl_utils`:
```python
from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
from qwen_vl_utils import process_vision_info
model_path = "lmms-lab/LLaVA-One-Vision-1.5-8B-Instruct"
# default: Load the model on the available device(s)
model = AutoModelForCausalLM.from_pretrained(
model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
)
# default processer
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
},
{"type": "text", "text": "Describe this image."},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to("cuda")
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
```
## Evaluation
```
# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
accelerate launch --num_processes=8 --main_process_port 12399 -m lmms_eval \
--model=llava_onevision1_5 \
--model_args=pretrained=lmms-lab/LLaVA-OneVision-1.5-8B-Instruct,attn_implementation=flash_attention_2,max_pixels=3240000 \
--tasks=mmmu_val,mmmu_pro_standard,mmbench_en_test,mmerealworld,mmerealworld_cn,ai2d,ai2d_no_mask,vstar_bench,chartqa,charxiv,docvqa_test,mathvista_testmini,mmstar,scienceqa \
--batch_size=1
```
### Mid-Training
To improve model training efficiency, we implement offline sample packing:
1. Download the [**Mid-Training-85M Dataset**](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M)
2. Pack the data into webdataset format, refer to [**Examples offlinepacking**](examples_offline_packing) and [**Offline Padding-Free Data Packing**](examples/llava_ov_1_5/sample_packing/README.md)
### Instruct
1. Download the [**LLaVA-OneVision-1.5-Insturct-Data**](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-1.5-Insturct-Data)
2. Convert the data into webdataset format, refer to [**Conversion for Mixed Instruction Data**](docs/sft_data_preprocessing.md)
## Roadmaps
Q4 2025 Key Deliverables:
1. **Ultra-efficient MoE Training**
2. **Full Video Input LLM**
## Contributors
Thanks so much to all of our amazing contributors!
<!-- readme: collaborators,contributors,jiankangdeng/- -start -->
<table>
<tbody>
<tr>
<td align="center">
<a href="https://github.com/fdcp">
<img src="https://avatars.githubusercontent.com/u/15667917?v=4" width="80;" alt="fdcp"/>
<br />
<sub><b>fdcp</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/anxiangsir">
<img src="https://avatars.githubusercontent.com/u/31175974?v=4" width="80;" alt="anxiangsir"/>
<br />
<sub><b>anxiangsir</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/yiyexy">
<img src="https://avatars.githubusercontent.com/u/35927125?v=4" width="80;" alt="yiyexy"/>
<br />
<sub><b>yiyexy</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/wideyard">
<img src="https://avatars.githubusercontent.com/u/101321826?v=4" width="80;" alt="wideyard"/>
<br />
<sub><b>wideyard</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/chengzheng345">
<img src="https://avatars.githubusercontent.com/u/209475443?v=4" width="80;" alt="chengzheng345"/>
<br />
<sub><b>chengzheng345</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/killTheHostage">
<img src="https://avatars.githubusercontent.com/u/16442720?v=4" width="80;" alt="killTheHostage"/>
<br />
<sub><b>killTheHostage</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/mathCrazyy">
<img src="https://avatars.githubusercontent.com/u/20607153?v=4" width="80;" alt="mathCrazyy"/>
<br />
<sub><b>mathCrazyy</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/yunglechao">
<img src="https://avatars.githubusercontent.com/u/7631185?v=4" width="80;" alt="yunglechao"/>
<br />
<sub><b>yunglechao</b></sub>
</a>
</td>
</tr>
<tr>
<td align="center">
<a href="https://github.com/RobitYadda">
<img src="https://avatars.githubusercontent.com/u/6811311?v=4" width="80;" alt="RobitYadda"/>
<br />
<sub><b>RobitYadda</b></sub>
</a>
</td>
</tr>
<tbody>
</table>
<!-- readme: collaborators,contributors,jiankangdeng/- -end -->
## Citation
If you find *LLaVA-OneVision-1.5* useful in your research, please consider to cite the following related papers:
```
@inproceedings{LLaVA-OneVision-1.5,
title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
booktitle={arxiv},
year={2025}
}
@inproceedings{xie2025region,
title={Region-based Cluster Discrimination for Visual Representation Learning},
author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
booktitle={ICCV},
year={2025}
}
@article{lillava,
title={LLaVA-OneVision: Easy Visual Task Transfer},
author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
journal={Transactions on Machine Learning Research}
year={2024}
}
```
## Acknowledgement
We extend our sincere gratitude to **AIAK team of the** [**Baige AI computing platform**](https://cloud.baidu.com/product/aihc.html) **from Baidu AI Cloud** for providing the exceptional training framework. The outstanding capabilities of AIAK-Training-LLM and AIAK-Megatron have significantly accelerated our training process with remarkable efficiency. These cutting-edge frameworks have been instrumental in achieving our research goals. `To get full AIAK support, you can contact Baidu Cloud.`
We also thank the maintainers and contributors of the following open-source projects, whose work greatly inspired and supported our research:
- LLaVA: Large Language-and-Vision Assistant — [LLaVA](https://github.com/haotian-liu/LLaVA)
- LLaVA-NeXT: Next-generation multi-modal assistant — [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT)
- lmms-eval: A standardized evaluation framework for Large Multimodal Models — [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval)
- Megatron-LM: Efficient, scalable training for large language models — [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
- Qwen2.5-VL: Strong vision-language foundation model — [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)
- InternVL: Open-source large-scale vision-language foundation model — [InternVL](https://github.com/OpenGVLab/InternVL)
- Qwen3: Next-generation Qwen LLM — [Qwen](https://github.com/QwenLM/Qwen)
- MetaCLIP: Scalable contrastive pretraining — [MetaCLIP](https://github.com/facebookresearch/MetaCLIP)
- FineVision: Open Data Is All You Need — [FineVision](https://huggingface.co/spaces/HuggingFaceM4/FineVision) |