Qwen3-VL-4B-Instruct-heretic-NVFP4

Overview

This repository provides an NVFP4 (FP4 E2M1) mixed-precision quantized build of Qwen3-VL-4B-Instruct-heretic in ComfyUI comfy_quant format, primarily intended for use as a text encoder (e.g. for Krea 2 / Qwen3-VL conditioning).

This model is a decensored derivative of the official Qwen/Qwen3-VL-4B-Instruct, modified using Heretic v1.1.0.

Build note (v2): the vision encoder is now kept in FP16 (previously it was NVFP4-quantized). ComfyUI's qwen3vl text-encoder does run the vision tower in image/multimodal workflows (sd1_clip.process_tokens → Qwen3VL.preprocess_embed → self.visual), and the vision tower is small but image-quality-sensitive, so leaving it in FP16 preserves visual conditioning fidelity at a negligible size cost. This matches the Qwen3-VL-8B NVFP4 baseline, which also ships vision in bf16.

Quantization Details

• Backend: ComfyUI convert_to_quant v1.2.6 (comfy_kitchen CUDA NVFP4 kernels)
• Format: comfy_quant mixed precision
  • NVFP4 (FP4 E2M1, 16-element blocks, learned rounding) — text transformer blocks 2–33 projections only
  • FP8 (float8_e4m3fn, tensorwise, learned rounding) — text blocks 1 & 34
  • FP16 (bfloat16) — embed_tokens, model.norm, all norms/biases, pos_embed, patch_embed, text blocks 0 & 35, and the entire vision encoder (blocks, merger, deepstack_merger_list)
• No input_scale: ComfyUI quantizes activations dynamically at runtime (per-tensor amax via the NVFP4 layout), so a baked-in activation scale is not required. This matches the stock Qwen3-VL NVFP4 baseline.
• Hardware target: NVIDIA Blackwell (RTX 50 series, SM 12.0) — required for native NVFP4 tensor-core execution (see Runtime below).
• Size: 4.03 GB / 1413 tensors (vs 8.88 GB FP16 source — ~55% smaller).

Layer breakdown

Tier	Layers	Count
FP16 (bf16)	embed_tokens, model.norm, all norms/biases, pos_embed, patch_embed, text blocks 0 & 35, entire vision encoder	kept lossless
FP8 (e4m3fn, tensorwise)	text blocks 1 & 34	14 weights
NVFP4 (E2M1, block=16)	text blocks 2–33 projections	224 weights

Runtime / hardware

How the layers execute depends on the GPU (ComfyUI pick_operations gates each format on the device capability):

GPU (SM)	NVFP4 layers	FP8 layers
RTX 5090 / Blackwell (12.0)	native FP4 tensor cores (fast)	native FP8
RTX 4090 / Ada (8.9)	dequantized to bf16 (size only, no speedup)	native FP8

On Blackwell, NVFP4 weights stay packed and run through comfy_kitchen's TensorCoreNVFP4Layout GEMM (FP4 weight × dynamically-quantized FP8 activation). On non-Blackwell GPUs the format is emulated via dequantization, so you get the smaller footprint but not the FP4 speedup.

Usage (ComfyUI)

Place qwen3_vl_4b_nvfp4_full.safetensors in ComfyUI/models/text_encoders/.

• As Krea 2 text encoder: load with a CLIPLoader node, type krea2.
• As generic Qwen3-VL text encoder: load with a CLIPLoader node, type qwen3vl_4b.

ComfyUI auto-detects the quantization metadata (Found quantization metadata version 1) and selects MixedPrecisionOps for the text encoder. The vision weights are included (unquantized) so detect_te_model() identifies the file as QWEN3VL_4B and routes to krea2.te() / qwen3vl.te() correctly.

License

Apache-2.0 (inherited from the base model).