method · first seen 5 Oct 2026
KV cache quantization
Also: low-bit state quantization, key-value cache quantization
Storing cached attention keys and values at reduced numeric precision to save memory.
KV cache quantization stores the attention keys and values kept during generation at reduced precision (often 2 to 4 bits) to cut memory and bandwidth in long-context inference. The sources show active refinement rather than a new idea: a paper trading token count against precision for long chain-of-thought decoding, selective rotation before 2-bit quantization (FlagRot), native FP4 cache decoding on Blackwell tensor cores (BitDecoding-FP4, with an accompanying repo, and Jet-QSparse from Song Han's group), fused 2- and 4-bit decode kernels (FOSCAR), sub-1-bit codes for looped transformers, and extensions to multi-agent systems, diffusion LMs and prefill-decode disaggregation.
The technique has been a standard part of LLM serving since roughly 2023-2024 (KIVI, KVQuant, vLLM FP8 cache), so this burst reflects steady research volume. The newer threads worth watching are hardware-native FP4 cache formats and per-workload bit allocation.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 116 on 14 Nov 2026 · 14 mentions in 28 days · 14 independent groups · papers → code · first seen 6 weeks ago · acceleration +1.14
Earliest sources
- Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought ReasoningPapers · 5 Oct 2026 · Runguo Li
- Shared Stopping Decisions Change Answers in HQQ Cache QuantizationPapers · 5 Oct 2026 · Seunghui Jwa, Minsu Oh, Chanjun Park, Yeo-Chan Yoon
- SoloQ: Calibration-Free Quantization for Diffusion Language ModelsPapers · 5 Oct 2026 · Donghyun Lee, Arkapravo Ghosh, Varun Manjunath, Bumjoon Kyle Rhee, Hyunho Kook,…
- Few Bits, One Law: Toward W2A4KV2Papers · 6 Oct 2026 · Kai Yi, Tarek Elgamal, Sruthikesh Surineni, Vignesh Vivekraja, Soumyadeep Ghosh,…
- Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV CachesPapers · 7 Oct 2026 · Sunjoo Whang, Jungjun Oh, Minsung Kim, Dongho Seo, Jisu Shin, Gregory Kielian, H…
- ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit ResidualsPapers · 7 Oct 2026 · Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han, Insu Han, Sehoon Kim
Latest sources
- MiSTiQ: MicroScaling Trellis Quantization for Extreme KV Cache CompressionPapers · 16 Nov 2026 · Ofir Gordon, Lior Dikstein, Paul Balanca, Idan Achituve, Douglas Orr, Hai Victor…
- Memory-Efficient Speculative Decoding with Quantized Draft KV CachePapers · 16 Nov 2026 · Saksham Gupta, Michael Li
- BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-DesignPapers · 6 Nov 2026 · Dayou Du, Woon Yee Ng, Guangming Sheng, Nick Lau, Jianyi Cheng, Luo Mai
- Coordinated Mixed-Precision KV Cache Quantization for Efficient LLM-based Multi-agent SystemsPapers · 6 Nov 2026 · Weizhong Huang, Jinchao Zhang, Xiawu Zheng
- FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine FoldingPapers · 6 Nov 2026 · Donglin Zhuang, Pragaash Ponnusamy, Xiaoxia Wu, Shuaiwen Leon Song, Zhongzhu Zho…
- ReKV: Overlapping Quantized KV Transfer and Suffix RecomputationPapers · 5 Nov 2026 · Guangjian Dong
- StrataFlux: Memory-Hierarchy-Native Low-Bit LLM InferencePapers · 5 Nov 2026 · Jake Hyun, Ziyu Zhao, Jae-sun Seo, Mohamed S. Abdelfattah
- OpenBitSys/BitDecoding-FP4Code · 3 Nov 2026 · OpenBitSys · 49 stars
- LoopVQ: Sub 1-bit KV Cache Quantization for Looped TransformersPapers · 3 Nov 2026 · Minsoo Cheong, Sungjoo Yoo
- FlagRot: Selective Rotation for Low-Bit KV CachesPapers · 2 Nov 2026 · Kunming Shao, Xijie Huang, Jierun Chen, Jiaqi Shao, Yanli Wang, Jiangnan Yu, Qiw…
- Jet-QSparse: Accelerating Agentic LLM Inference with 4-Bit Sparse AttentionPapers · 31 Oct 2026 · Haozhan Tang, Hao Kang, Meng Lu, Yecheng Wu, Zerui Wang, Seokho Han, Haocheng Xi…
- te-comp: A Complete MLIR Compiler Generating Self-Contained C Inference Engines for MCUsPapers · 31 Oct 2026 · Van Chan Ngo