method · first seen 5 Oct 2026
KV cache compression
Also: KV retrieval, sparse KV cache, KV cache retrieval, compressed KV cache, KV memory reduction, KV cache memory reduction, key-value cache compaction
Reducing the memory footprint of cached attention keys and values while preserving model output quality.
KV cache compression reduces the memory held by cached attention keys and values during LLM inference, using eviction, quantization, low-rank projection, cross-layer sharing or page-level retrieval. The sources are almost entirely arXiv papers refining known approaches. One studies how to split a byte budget between token count and precision for long chain-of-thought decoding. Another delays eviction until early generation queries are available. Others push to 1-bit or FP4 storage for multi-head latent attention, or extend compression to vision-language models and autoregressive video generation. A llama.cpp fork applies compressed-KV flash attention on older Pascal GPUs. This is a steady stream of incremental academic work, not a new idea.
KV cache size is a main limit on long-context and agentic inference cost, so small gains in compression ratio or accuracy carry directly into serving throughput and into which hardware can run long contexts. The newer focus on reasoning traces and multimodal histories reflects where memory pressure is currently growing.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 122 on 14 Nov 2026 · 41 mentions in 28 days · 40 independent groups · papers → code · first seen 6 weeks ago · acceleration +0.23
Earliest sources
- Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought ReasoningPapers · 5 Oct 2026 · Runguo Li
- DeferKV: Rethinking Eviction Timing for One-Shot KV Cache CompressionPapers · 5 Oct 2026 · Zhe Wang, Jiakai Li, Yujia Sun, Rongzheng Wang, Shuang Liang
- Behavior-Preserving KV Cache CompressionPapers · 5 Oct 2026 · Doo Hwan Hwang, Junyoung Jang, Junho Na, Hosung Lim, Kee-Eung Kim
- OVAL: Output-Aware Local Page Bases for KV Cache RetrievalPapers · 5 Oct 2026 · Ashkan Shahbazi, Chayne Thrash, Soheil Kolouri
- Hybrid Latent Attention for Looped Language ModelsPapers · 6 Oct 2026 · Yuhan Chen, Siyuan Zhang, Nan Wang, Feiyang Kang, Ruoxi Jia
- A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal AttentionPapers · 6 Oct 2026 · Davis Wertheimer, Haochen Shen, Ahan Gupta, Derrick Liu, Yu Chin Fabian Lim, Mud…
Latest sources
- MovieRoll: A Systematic Framework for Rolling Out Long-Form VideosPapers · 16 Nov 2026 · Yunyang Ge, Weijie Kong, Ryan Xu, Yue Wu, Qi Tian, Jiangfeng Xiong, Bin Lin, Min…
- GaugeRoPE: RoPE-Compatible Executable Latent AttentionPapers · 16 Nov 2026 · Yuanhang Gao, Xiangrui Yang, Lingyuan Meng, Yuanfeng Chen, Dongsheng Li
- NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent AttentionPapers · 13 Nov 2026 · Chao Ni, Qi Zhang, Xiao Wang, Zongfei Jing, Mingyang Hao, Binghan Chen, Kefeng D…
- KVPath: Connecting Key-Value Representations Across Transformer LayersPapers · 11 Nov 2026 · Xuan Luo, Zhiyuan Wang, Hanzhong Cao, Xifeng Yan
- TraceKV: Policy-Preserving Low-Rank KV Cache Compression for Reasoning Vision-Language ModelsPapers · 11 Nov 2026 · Ke Li, Zishan Shao, Yiran Chen
- Keep Images in Sight: Visual Position Re-Anchoring for Long-Context MLLMsPapers · 11 Nov 2026 · Juntao Xu, Qichai Zhao, Rui Sun, Zhouyang Jiang, Shiwei Wu, Zhi Li, Kaisheng Ma,…
- joris-sta/llama-cpp-pascalCode · 9 Nov 2026 · joris-sta · 22 stars
- Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra OptimizationPapers · 9 Nov 2026 · Zhengyu Hu, Zheyuan Xiao, Yike Wang, Hamish Ivison, Haoyu Wang, Yige Yuan, Rulin…
- Semantically Aligned KV Cache Compression for Efficient Agentic Autoregressive Video GenerationPapers · 8 Nov 2026 · Weizhong Huang, Huaijia Kou, Tianyu Xie, Xiawu Zheng, Liujuan Cao
- Verify Before You Fetch: Reliable KV Cache Compression for LLM InferencePapers · 8 Nov 2026 · Devanshu Jha, Gaurav Kumar, Agniva Chowdhury, Kumar Krishna Agrawal, Aditya Desa…
- CALFA-KV: Joint Calibration and Fisher-Guided Anchors for 1-Bit KV Cache CompressionPapers · 7 Nov 2026 · Xiancai Tian, Leichao Zhang, Shenhui Zhang, Yi Shan, Hailong Li, Feng Lin, Xiaot…
- Maverick720/Focus-KV-Head-wise-KV-Cache-Compression-Focus-on-Reasoning-HeadsCode · 6 Nov 2026 · Maverick720 · 85 stars