Enhance model card: Add project page link, overall concept, training, evaluation, case, and acknowledgment sections (#1)

- Enhance model card: Add project page link, overall concept, training, evaluation, case, and acknowledgment sections (17d4f5413338f3666f67f3a4cd695c386b416fb3)


Co-authored-by: Niels Rogge <[email protected]>

README.md

@@ -1,18 +1,6 @@
 ---
-license: apache-2.0
-license_name: qwen
-license_link: https://huggingface.co/Qwen/Qwen2.5-32B/blob/main/LICENSE
-pipeline_tag: text-generation
-library_name: transformers
 base_model:
 - Qwen/Qwen2.5-32B
-base_model_relation: finetune
-language:
-- en
-tags:
-- sdlm
-- diffusion language model
-- custom_code
 datasets:
 - dyyyyyyyy/ScaleQuest-Math
 - OpenCoder-LLM/opc-sft-stage2
@@ -20,34 +8,54 @@ datasets:
 - HuggingFaceTB/smoltalk2
 - LipengCS/Table-GPT
 - allenai/SciRIFF
+language:
+- en
+library_name: transformers
+license: apache-2.0
+license_name: qwen
+license_link: https://huggingface.co/Qwen/Qwen2.5-32B/blob/main/LICENSE
+pipeline_tag: text-generation
+tags:
+- sdlm
+- diffusion language model
+- custom_code
+base_model_relation: finetune
 ---
 
 # SDLM-32B-D4
 
-[\[📂 GitHub\]](https://github.com/OpenGVLab/SDLM)  [\[📜 Tech Report\]](https://arxiv.org/abs/2509.24007)  [\[🤗 HuggingFace\]](https://huggingface.co/collections/OpenGVLab/sdlm-68ac82709d7c343ad36aa552)  
+[\[📂 GitHub\]](https://github.com/OpenGVLab/SDLM) [\[📜 Tech Report\]](https://arxiv.org/abs/2509.24007) [🚀 Project Page](https://internvl.github.io/blog/2025-09-29-SDLM/) [\[🤗 HuggingFace Collection\]](https://huggingface.co/collections/OpenGVLab/sdlm-68ac82709d7c343ad36aa552)
 
 ## Introduction
 
-We propose a <b>S</b>equential <b>D</b>iffusion <b>L</b>anguage <b>M</b>odel (<b>SDLM</b>), to cheaply stimulate the parallel prediction capabilities of diffusion models. Specifically, SDLM reduces distribution shift by limiting the prediction range to a fixed block length and enforces decoding order through the longest prefix decoding method, thereby significantly improving prediction efficiency while ensuring generation quality. Our method can be viewed as a further generalization of the autoregressive (AR) paradigm. Therefore, it is possible to use pre-trained AR weights and quickly migrate to the diffusion framework with only minimal instruction fine-tuning.
+We propose a **S**equential **D**iffusion **L**anguage **M**odel (**SDLM**), to cheaply stimulate the parallel prediction capabilities of diffusion models. Specifically, SDLM reduces distribution shift by limiting the prediction range to a fixed block length and enforces decoding order through the longest prefix decoding method, thereby significantly improving prediction efficiency while ensuring generation quality. Our method can be viewed as a further generalization of the autoregressive (AR) paradigm. Therefore, it is possible to use pre-trained AR weights and quickly migrate to the diffusion framework with only minimal instruction fine-tuning.
 
-![image/png](https://huggingface.co/OpenGVLab/SDLM-32B-D4/resolve/main/assets/three_framework.png)
+### Overall Concept
+
+SDLM delivers strong performance with significantly faster decoding speed. It operates approximately 2x faster than comparable autoregressive models while matching their accuracy, and achieves up to 5x speedup over other diffusion language models, as evidenced by results on the MATH-500 benchmark.
+
+![Overall Framework](https://huggingface.co/OpenGVLab/SDLM-32B-D4/resolve/main/assets/framwork_compare.png)
+
+-   Autoregression: Predicts tokens one by one.
+-   Diffusion: Regenerates all tokens each step.
+-   SDLM (ours): Decodes D tokens per step, then **keeps the longest consecutive n confident tokens** (1 ≤ n ≤ D). Cached tokens are reused, saving computation.
 
 ## SDLM Family
 
 In the following table, we provide an overview of the SDLM series.
 
-| Model Name  | Base Model 🤗                                                 | HF Link 🤗                                    |
-| ----------- | ------------------------------------------------------------ | -------------------------------------------- |
-| SDLM-3B-D4  | <a href="https://huggingface.co/Qwen/Qwen2.5-3B">Qwen2.5-3B</a> | https://huggingface.co/OpenGVLab/SDLM-3B-D4  |
-| SDLM-3B-D8  | <a href="https://huggingface.co/Qwen/Qwen2.5-3B">Qwen2.5-3B</a> | https://huggingface.co/OpenGVLab/SDLM-3B-D8  |
+| Model Name | Base Model 🤗 | HF Link 🤗 |
+| :--------- | :----------------------------------------------------------- | :-------------------------------------------- |
+| SDLM-3B-D4 | <a href="https://huggingface.co/Qwen/Qwen2.5-3B">Qwen2.5-3B</a> | https://huggingface.co/OpenGVLab/SDLM-3B-D4 |
+| SDLM-3B-D8 | <a href="https://huggingface.co/Qwen/Qwen2.5-3B">Qwen2.5-3B</a> | https://huggingface.co/OpenGVLab/SDLM-3B-D8 |
 | SDLM-32B-D4 | <a href="https://huggingface.co/Qwen/Qwen2.5-32B">Qwen2.5-32B</a> | https://huggingface.co/OpenGVLab/SDLM-32B-D4 |
 
 ## Model Architecture
 
 We propose a sequential blockwise masked prediction method that reduces error accumulation in diffusion-based generation. Our method leverages the observation that predictions for tokens at lower positional indices typically benefit from more reliable contextual information, resulting in lower deviation and improved accuracy.
 
-* **(a) Training pipeline.** Reordered input enables structured mask with causal prefix (top-left), visible cross-block prefix (bottom-left), and intra-block bidirectional attention (bottom-right).
-* **(b) Sampling Pipeline.** Confidence-based dynamic block decoding with KV cache reuse. At each step, a block of B tokens is predicted with B-1 padding masks. The longest high-confidence prefix is selected as dynamic output. Cached KV states enable efficient decoding.
+*   **(a) Training pipeline.** Reordered input enables structured mask with causal prefix (top-left), visible cross-block prefix (bottom-left), and intra-block bidirectional attention (bottom-right).
+*   **(b) Sampling Pipeline.** Confidence-based dynamic block decoding with KV cache reuse. At each step, a block of B tokens is predicted with B-1 padding masks. The longest high-confidence prefix is selected as dynamic output. Cached KV states enable efficient decoding.
 
 ![image/png](https://huggingface.co/OpenGVLab/SDLM-3B-D4/resolve/main/assets/framework.png)
 
@@ -75,70 +83,158 @@ Trade-off between performance and speed under different confidence thresholds τ
 
 ## Inference
 
-1. Install Dependencies
-
-   Key package versions:
-
-   ```
-   transformers==4.37.2
-   torch>=2.5.0
-   ```
-   
-2. Download the model generation script [sdlm_inference.py](https://github.com/OpenGVLab/SDLM/blob/main/sdlm_inference.py) to your working directory.
-
-3. We provide an example code to run `SDLM-32B-D4` using `transformers`.
-
-   ```python
-   import torch
-   from transformers import AutoModelForCausalLM, AutoTokenizer
-   from sdlm_inference import SDLM_generate
-   
-   if __name__ == "__main__":
-       ckpt_hf = 'OpenGVLab/SDLM-32B-D4'
-   
-       model = AutoModelForCausalLM.from_pretrained(
-           ckpt_hf, 
-           attn_implementation="eager",
-           trust_remote_code=True
-       ).to(dtype=torch.float16)
-       tokenizer = AutoTokenizer.from_pretrained(ckpt_hf)
-   
-       prompt = 'Write a Fibonacci function in Python.'
-       messages = [
-           {"role": "system", "content": "You are a helpful assistant."},
-           {"role": "user", "content": prompt}
-       ]
-       text = tokenizer.apply_chat_template(
-           messages,
-           tokenize=False,
-           add_generation_prompt=True
-       )
-   
-       model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
-   
-       response, history = SDLM_generate(
-           model,
-           tokenizer,
-           model_inputs,
-           max_gen_len = 1024,
-           temperature = 0,
-           threshold = 0.5,
-           n_future_tokens = 4,
-           alg = 'prob_conf', #  prob_conf | entropy_conf | self_speculative
-           save_history = True,
-           use_cache = True
-       )
-   
-       print('response: ', response[0])
-   
-       print('=======histroy')
-       for item in history:
-           print('cur total token ', item[1])
-           print(item[0][0])
-           print('--------')
-   ```
+1.  Install Dependencies
+
+    Key package versions:
+
+    ```
+    transformers==4.37.2
+    torch>=2.5.0
+    ```
+
+2.  Download the model generation script [sdlm_inference.py](https://github.com/OpenGVLab/SDLM/blob/main/sdlm_inference.py) to your working directory.
+
+3.  We provide an example code to run `SDLM-32B-D4` using `transformers`.
+
+    ```python
+    import torch
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    from sdlm_inference import SDLM_generate
+
+    if __name__ == "__main__":
+        ckpt_hf = 'OpenGVLab/SDLM-32B-D4'
+
+        model = AutoModelForCausalLM.from_pretrained(
+            ckpt_hf,
+            attn_implementation="eager",
+            trust_remote_code=True
+        ).to(dtype=torch.float16)
+        tokenizer = AutoTokenizer.from_pretrained(ckpt_hf)
+
+        prompt = 'Write a Fibonacci function in Python.'
+        messages = [
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": prompt}
+        ]
+        text = tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            add_generation_prompt=True
+        )
+
+        model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+        response, history = SDLM_generate(
+            model,
+            tokenizer,
+            model_inputs,
+            max_gen_len = 1024,
+            temperature = 0,
+            threshold = 0.5,
+            n_future_tokens = 4,
+            alg = 'prob_conf', #  prob_conf | entropy_conf | self_speculative
+            save_history = True,
+            use_cache = True
+        )
+
+        print('response: ', response[0])
+
+        print('=======histroy')
+        for item in history:
+            print('cur total token ', item[1])
+            print(item[0][0])
+            print('--------')
+    ```
+
+## Train
+
+1.  Environment Setup
+
+    ```bash
+    git clone https://github.com/OpenGVLab/SDLM.git
+    cd SDLM
+    ```
+
+2.  Install Dependencies
+
+    Key package versions:
+    ```
+    transformers==4.37.2
+    deepspeed==0.16.5
+    torch>=2.5.0
+    accelerate==0.32.1
+    ```
+    **Note**: Additional setup is required if using Flex Attention.
+
+
+3.  Prepare Training Data
+
+    The training dataset we used is specified in the meta file: [meta.json](https://github.com/OpenGVLab/SDLM/blob/main/shell/playground/data/meta/sft_opc436k_scale_math_1m_smoltalk_1m_tulu_1m.json) and is organized in the ShareGPT style, according to the [InternVL chat data format](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html).
+
+    This dataset is composed of several open-source datasets, with the following structure:
+
+    | Dataset Name | # Sample | Domain |
+    | :----------------------------------------------------------------------------------------- | :--------- | :------- |
+    | <a href="https://huggingface.co/datasets/dyyyyyyyy/ScaleQuest-Math">ScaleQuest-Math</a> | 1,000K | Math |
+    | <a href="https://huggingface.co/datasets/OpenCoder-LLM/opc-sft-stage2">Opc-sft-stage2</a> | 436K | Code |
+    | <a href="https://huggingface.co/datasets/HuggingFaceTB/smoltalk">Smoltalk</a> | 1,100K | General |
+    | <a href="https://huggingface.co/datasets/allenai/tulu-3-sft-mixture">Tulu-3-sft-mixture</a> | 939K | General |
+    | <a href="https://huggingface.co/datasets/allenai/SciRIFF">SciRIFF</a> | 79K | Scienece|
+    | <a href="https://huggingface.co/datasets/LipengCS/Table-GPT">Table-GPT</a> | 13K | Table |
+    | **Total** | **3,506K** | -- |
+
+
+4.  Start Training
+
+    All training scripts are available in the [shell/train](https://github.com/OpenGVLab/SDLM/tree/main/shell/train) directory. Key parameters include:
+    *   `block_size`: The size of the diffusion window. Current settings use `4`, we also try to use `8`; larger sizes are under exploration.
+    *   `attn_implementation`: Attention implementation type. Options include sdpa, eager, or flex_attn. Using Flex Attention requires additional setup. Prefer to use `sdpa` for a quick start.
+    *   `causal_attn`: Whether to use causal attention within the window. Currently set to non-causal (`False`).
+
+    Our training setting is:
+
+    <p align="center">
+        <img src="https://huggingface.co/OpenGVLab/SDLM-32B-D4/resolve/main/assets/hyper-param.png" width="50%"></a>
+    </p>
+
+    The training loss of our 3B model. loss_pos_`i` refers to the loss at the `i`-th position of each block. The loss at `i=0` is close to the SFT loss of AR's NTP.
+
+    Here, we display the loss corresponding to each position within the window during the training process. When bs=8, only the first 4 are shown.
+    The correspondence is as follows:
+
+    bs = 4 (red):
+
+    | x | m | m | m |
+    | :-- | :-- | :-- | :-- |
+    | loss_pos_1 | loss_pos_2 | loss_pos_3 | loss_pos_4 |
+
+    bs = 8 (orange):
+
+    | x | m | m | m | m | m | m | m |
+    | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
+    | loss_pos_1 | loss_pos_2 | loss_pos_3 | loss_pos_4 | -- | -- | -- | -- |
+
+    ![](https://huggingface.co/OpenGVLab/SDLM-32B-D4/resolve/main/assets/train_log_3b.png)
+
+## Evaluation
+
+Currently, we use [Opencompass](https://github.com/open-compass/opencompass) for evaluation. For more details, please refer to the [evaluation guide](https://github.com/OpenGVLab/SDLM/blob/main/eval/with_opencompass/readme.md).
+
+## Case
+
+<p align="center">
+    <img src="https://huggingface.co/OpenGVLab/SDLM-32B-D4/resolve/main/assets/case.gif" width="70%"></a>
+</p>
+
+## Acknowledge
 
+We extend our gratitude to the open-source community for their foundational contributions:
 
+*   [InternVL](https://github.com/OpenGVLab/InternVL/tree/main) The codebase we build upon.
+*   [SMDM](https://github.com/ML-GSAI/SMDM), [LLaDA](https://github.com/ML-GSAI/LLaDA), [Dream](https://github.com/HKUNLP/Dream), [Block Diffusion](https://github.com/kuleshov-group/bd3lms) for insights into diffusion-based generative modeling.
+*   [Qwen2.5](https://qwenlm.github.io/blog/qwen2.5-llm/) as a robust base model for comparative studies.
+*   [Opencompass](https://github.com/open-compass/opencompass) for providing a comprehensive evaluation framework.
+*   The creators of all datasets used in this work, enabling rigorous training and validation.
 
 ## Citation
 
@@ -151,4 +247,4 @@ If you find this project useful in your research, please consider citing:
   journal={arXiv preprint arXiv:2509.24007},
   year={2025}
 }
-```
+```