LLaDA2.0-mini

LLaDA2.0-mini is a diffusion language model featuring a 16BA1B Mixture-of-Experts (MoE) architecture. As an enhanced, instruction-tuned iteration of the LLaDA series, it is optimized for practical applications.

---

Benchmark	Qwen3-8B (no thinking)	Ling-mini-2.0	LLaDA2.0-mini-preview	LLaDA2.0-mini
Average	70.19	72.13	61.75	71.67
Knowledge
MMLU	80.94	82.15	72.49	80.53
MMLU-Pro	65.48	63.72	49.22	63.22
GPQA	46.59	56.80	31.82	47.98
arc-c	93.35	93.09	89.15	93.56
CMMLU	79.17	80.84	67.53	79.50
C-EVAL	81.36	82.10	66.54	81.38
GAOKAO-Bench	84.94	87.23	74.46	84.30
Reasoning
SQuAD 2.0	85.21	75.56	85.61	86.50
DROP	84.56	78.80	79.49	81.91
KOR-Bench	54.48	62.72	37.26	50.40
HellaSwag	79.56	69.02	74.01	79.01
Coding
CRUXEval-O	74.06	76.12	61.88	71.62
MBPP	78.92	84.07	77.75	81.50
MultiPL-E	61.7	67.09	62.43	67.46
HumanEval	84.76	85.98	80.49	86.59
BigCodeBench-Full	36.05	35.00	30.44	32.89
LiveCodeBench	26.38	34.97	19.93	31.50
Spider	72.80	76.43	75.64	76.76
Math
GSM8K	93.63	94.62	89.01	94.24
MATH	86.28	94.66	73.50	93.22
OlympiadBench	55.33	72.30	36.67	67.70
AIME 2025	22.08	47.66	10.00	36.67
Agent & Alignment
BFCL_Live	70.08	53.98	74.11	70.90
IFEval-strict -prompt	86.9	76.16	62.50	80.78

🚀 Performance Highlights

• Leading MoE Architecture: The open-source Mixture-of-Experts (MoE) diffusion large language model continually trained on the Ling2.0 series with approximately 20 trillion tokens.
• Efficient Inference: With 16 billion total parameters, only 1.4 billion are activated during inference. LLaDA2.0-mini significantly reduces computational costs while outperforming open-source dense models of similar scale.
• Impressive Performance on Code & Complex Reasoning: Excels in tasks such as code generation and advanced mathematical reasoning, demonstrating strong reasoning capabilities.
• Tool Use: Supports tool calling and achieves excellent performance in complex agent-based tasks.
• Open & Extensible: Fully open-source with commitment to transparency. We plan to release a leading inference framework in the future and continue investing in cutting-edge areas like diffusion LLMs (dLLM) to drive disruptive innovation.

🗺️ What's Next

• Supercharged Reasoning with LLaDA 2.0: LLaDA 2.0 series will be fine-tuned with Reinforcement Learning, unlocking a new level of sophisticated reasoning and problem-solving abilities.
• Tools for Innovators: The model was finetuned on the dFactory framework using Fully Sharded Data Parallel (FSDP2). We have begun open-sourcing dFactory and will continuously release our advanced post-training technologies. Whether you want to master the current model or build your own customized versions, you'll have the tools you need. Stay tuned for more updates!

---

📦 Model Variants

Model ID	Description	Hugging Face Link
inclusionAI/LLaDA2.0-mini	Instruction-tuned model, ready for downstream applications.	🤗 Model Card
inclusionAI/LLaDA2.0-flash	Instruction-tuned model, ready for downstream applications.	🤗 Model Card

---

🔍 Model Overview

LLaDA2.0-mini has the following specifications:

• Type: Mixture-of-Experts (MoE) Diffusion Language Model
• Total Parameters (Non-Embedding): 16B
• Number of Layers: 20
• Attention Heads: 16
• Context Length: 32,768 tokens
• Position Embedding: Rotary (RoPE)
• Vocabulary Size: 157,184

---

🤗 Hugging Face Transformers

Make sure you have transformers and its dependencies installed:

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

model_path = "/path/to/LLaDA2.0-mini"
device = "cuda:0"
model = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True, device_map=device
)
model = model.to(torch.bfloat16)
model.eval()
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

prompt = "Why does Camus think that Sisyphus is happy?"
input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
)
generated_tokens = model.generate(
    inputs=input_ids,
    eos_early_stop=True,
    gen_length=512,
    block_length=32,
    steps=32,
    temperature=0.0,
)
generated_answer = tokenizer.decode(
    generated_tokens[0],
    skip_special_tokens=True,
)
print(generated_answer)

Best Practices

To achieve optimal performance, we recommend the following settings:

1. Sampling Parameters: We suggest using Temperature=0.0, block_length=32, and steps=32. Using a higher temperature value may occasionally result in language mixing and a slight decrease in model performance.

2. Adequate Output Length: We recommend using an output length of 32768 tokens for most queries.

---

🌐 License

This project is licensed under the terms of the Apache License 2.0.

---

🤝 Contact & Collaboration

For questions, collaborations, or feedback, please reach out via Hugging Face or open an issue in the repository.

👉 Join us in advancing open, efficient, and intelligent language models!