NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMs
Haeun Lee, Omin Kwon, Yeonhong Park, Jae Lee
Large Language Models (LLMs) are increasingly deployed in latency-critical, high-throughput services such as virtual assistants, code generation, and multi-modal applications. To satisfy stringent service-level objectives (SLOs), techniques such as continuous batching, paged attention, and hybrid scheduling of prefill and decode phases have been developed. While weight and activation quantization has been widely explored to accelerate inference, methods for adapting precision dynamically and efficiently at runtime remain largely underexplored. Recent hardware support for FP8 arithmetic—offering up to 2× the throughput of FP16—makes it an attractive option for interactive LLM serving. However, naively co-deploying separate FP8 and FP16 models increases storage overhead and may sacrifice accuracy or performance.We present Dual-FP LLM, a precision-adaptive serving technique that enables both FP8 and FP16 inference from a single 16-bit model representation. By decomposing each FP16 weight into two 8-bit components, our method enables efficient FP8 execution while preserving full FP16 accuracy—without incurring additional memory cost. We implement a custom CUTLASS-based GEMM kernel that reconstructs FP16 operands on-the-fly and integrated it into the vLLM serving framework. Dual-FP LLM delivers up to 1.55x throughput improvement in FP8 mode, with negligible accuracy degradation, and introduces only up to 4.51% average overhead for FP16 inference. This enables dynamic, SLO-aware precision selection and offers a flexible foundation for scalable LLM serving under bursty, heterogeneous workloads.