---
language:
- en
- ko
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
model_id: BCCard/kanana-1.5-8b-instruct-2505-FP8-Dynamic
repo: BCCard/kanana-1.5-8b-instruct-2505-FP8-Dynamic
developers: Kanana LLM
training_regime: bf16 mixed precision
base_model:
- kakaocorp/kanana-1.5-8b-instruct-2505
---
# FP8 Dynamic Quantized kakaocorp/kanana-1.5-8b-instruct-2505
## 1. What FP8‑Dynamic Quantization Is
* **FP8 format**
* 8‑bit floating‑point (1 sign bit + 5 exponent bits + 2 mantissa bits).
* Drastically shrinks weight/activation size while keeping floating‑point behavior.
* **Dynamic scheme (`FP8_DYNAMIC`)**
* **Weights:** *static*, **per‑channel** quantization (each out‑feature channel has its own scale).
* **Activations:** *dynamic*, **per‑token** quantization (scales are recomputed on‑the‑fly for every input token).
* **RTN (Round‑To‑Nearest) PTQ**
* Post‑training; no back‑prop required.
* No calibration dataset needed because:
* Weights use symmetric RTN.
* Activations are quantized dynamically at inference time.
## 2. Serving the FP8 Dynamic Model with vLLM
If your card does not support NCL P2P, such as A40 or RTX4090/5090, you need to add the options below.
For reference, the Dense model can be used after compression with FP8 Dynamic.
```
export NCCL_P2P_DISABLE=1
```
In GPU 2 units, with KV Cache 90%, Max token 32768
```
vllm serve BCCard/kanana-1.5-8b-instruct-2505-FP8-Dynamic \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.9 \
--max-model-len 32768 \
--enforce-eager \
--api-key bccard \
--served-model-name kanana-1.5-8b-instruct
```
## 3. Quantization Code Walk‑Through (Shared Knowledges)
[LLM Compressor](https://github.com/vllm-project/llm-compressor) is an easy-to-use library for optimizing models for deployment with vllm, including:
Comprehensive set of quantization algorithms for weight-only and activation quantization
Seamless integration with Hugging Face models and repositories
safetensors-based file format compatible with vllm
Large model support via accelerate
```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = "kakaocorp/kanana-1.5-8b-instruct-2505"
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
# Configure the quantization algorithm and scheme.
# In this case, we:
# * quantize the weights to fp8 with per channel via ptq
# * quantize the activations to fp8 with dynamic per token
recipe = QuantizationModifier(
targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]
)
# Apply quantization.
oneshot(model=model, recipe=recipe)
# Confirm generations of the quantized model look sane.
print("========== SAMPLE GENERATION ==============")
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
print("==========================================")
# Save to disk in compressed-tensors format.
SAVE_DIR = "BCCard/kanana-1.5-8b-instruct-2505-FP8-Dynamic"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
```
## 4. Kanana 1.5 model card
`Kanana 1.5`, a newly introduced version of the Kanana model family, presents substantial enhancements in **coding, mathematics, and function calling capabilities** over the previous version, enabling broader application to more complex real-world problems. This new version now can handle __up to 32K tokens length natively and up to 128K tokens using YaRN__, allowing the model to maintain coherence when handling extensive documents or engaging in extended conversations. Furthermore, Kanana 1.5 delivers more natural and accurate conversations through a __refined post-training process__.
## Performance
### Base Model Evaluation
| Models |
MMLU |
KMMLU |
HAERAE |
HumanEval |
MBPP |
GSM8K |
| Kanana-1.5-8B |
64.24 |
48.94 |
82.77 |
61.59 |
57.80 |
63.53 |
| Kanana-8B |
64.22 |
48.30 |
83.41 |
40.24 |
51.40 |
57.09 |
### Instruct Model Evaluation
| Models |
MT-Bench |
KoMT-Bench |
IFEval |
HumanEval+ |
MBPP+ |
GSM8K (0-shot) |
MATH |
MMLU (0-shot, CoT) |
KMMLU (0-shot, CoT) |
FunctionChatBench |
| Kanana-1.5-8B* |
7.76 |
7.63 |
80.11 |
76.83 |
67.99 |
87.64 |
67.54 |
68.82 |
48.28 |
58.00 |
| Kanana-8B |
7.13 |
6.92 |
76.91 |
62.20 |
43.92 |
79.23 |
37.68 |
66.50 |
47.43 |
17.37 |
> [!Note]
> \* Models released under Apache 2.0 are trained on the latest versions compared to other models.
## Processing 32K+ Length
Currently, the `config.json` uploaded to HuggingFace is configured for token lengths of 32,768 or less. To process tokens beyond this length, YaRN must be applied. By updating the `config.json` with the following parameters, you can apply YaRN to handle token sequences up to 128K in length:
```json
"rope_scaling": {
"factor": 4.4,
"original_max_position_embeddings": 32768,
"type": "yarn",
"beta_fast": 64,
"beta_slow": 2
},
```
## Contributors
- Language Model Training: Yunju Bak, Doohae Jung, Boseop Kim, Nayeon Kim, Hojin Lee, Jaesun Park, Minho Ryu
- Language Model Alignment: Jiyeon Ham, Seungjae Jung, Hyunho Kim, Hyunwoong Ko, Changmin Lee, Daniel Wontae Nam
- AI Engineering: Youmin Kim, Hyeongju Kim
- Quantizatization: Taeyoung Lee
## Citation
```
@misc{kananallmteam2025kananacomputeefficientbilinguallanguage,
title={Kanana: Compute-efficient Bilingual Language Models},
author={Kanana LLM Team and Yunju Bak and Hojin Lee and Minho Ryu and Jiyeon Ham and Seungjae Jung and Daniel Wontae Nam and Taegyeong Eo and Donghun Lee and Doohae Jung and Boseop Kim and Nayeon Kim and Jaesun Park and Hyunho Kim and Hyunwoong Ko and Changmin Lee and Kyoung-Woon On and Seulye Baeg and Junrae Cho and Sunghee Jung and Jieun Kang and EungGyun Kim and Eunhwa Kim and Byeongil Ko and Daniel Lee and Minchul Lee and Miok Lee and Shinbok Lee and Gaeun Seo},
year={2025},
eprint={2502.18934},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.18934},
}
```
## Contact
- Kanana LLM Team Technical Support: kanana-llm@kakaocorp.com
- Business & Partnership Contact: alpha.k@kakaocorp.com
- BCCard Quantized LLMs Engineering Support : seen@bccard.com