Early-Signal Detector

method · first seen 5 Oct 2026

KV cache compression

Also: KV retrieval, sparse KV cache, KV cache retrieval, compressed KV cache, KV memory reduction, KV cache memory reduction, key-value cache compaction

Reducing the memory footprint of cached attention keys and values while preserving model output quality.

KV cache compression reduces the memory held by cached attention keys and values during LLM inference, using eviction, quantization, low-rank projection, cross-layer sharing or page-level retrieval. The sources are almost entirely arXiv papers refining known approaches. One studies how to split a byte budget between token count and precision for long chain-of-thought decoding. Another delays eviction until early generation queries are available. Others push to 1-bit or FP4 storage for multi-head latent attention, or extend compression to vision-language models and autoregressive video generation. A llama.cpp fork applies compressed-KV flash attention on older Pascal GPUs. This is a steady stream of incremental academic work, not a new idea.

KV cache size is a main limit on long-context and agentic inference cost, so small gains in compression ratio or accuracy carry directly into serving throughput and into which hardware can run long contexts. The newer focus on reasoning traces and multimodal histories reflects where memory pressure is currently growing.

Analyst note (established), 17 Nov 2026.

Mentions per week

01519 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 122 on 14 Nov 2026 · 41 mentions in 28 days · 40 independent groups · papers → code · first seen 6 weeks ago · acceleration +0.23

Earliest sources

Latest sources

Mentioned together with

rope data structurelatent attentionsparse attentionautoregressive video diffusionautoregressive video generationspeculative decodingvision-language modellow-rank approximationgrouped-query attentioncross-layer cache sharing