Early-Signal Detector

method · first seen 5 Oct 2026

KV cache quantization

Also: low-bit state quantization, key-value cache quantization

Storing cached attention keys and values at reduced numeric precision to save memory.

KV cache quantization stores the attention keys and values kept during generation at reduced precision (often 2 to 4 bits) to cut memory and bandwidth in long-context inference. The sources show active refinement rather than a new idea: a paper trading token count against precision for long chain-of-thought decoding, selective rotation before 2-bit quantization (FlagRot), native FP4 cache decoding on Blackwell tensor cores (BitDecoding-FP4, with an accompanying repo, and Jet-QSparse from Song Han's group), fused 2- and 4-bit decode kernels (FOSCAR), sub-1-bit codes for looped transformers, and extensions to multi-agent systems, diffusion LMs and prefill-decode disaggregation.

The technique has been a standard part of LLM serving since roughly 2023-2024 (KIVI, KVQuant, vLLM FP8 cache), so this burst reflects steady research volume. The newer threads worth watching are hardware-native FP4 cache formats and per-workload bit allocation.

Analyst note (established), 17 Nov 2026.

Mentions per week

0919 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 116 on 14 Nov 2026 · 14 mentions in 28 days · 14 independent groups · papers → code · first seen 6 weeks ago · acceleration +1.14

Earliest sources

Latest sources

Mentioned together with

rotation-based quantizationmixed-precision quantizationNVFP4 quantizationchain-of-thoughtdiffusion language modelpost-training quantizationvector quantizationlatent attentionlooped transformerlinear attention