tool · first seen 6 Oct 2026
LLM serving systems
Also: LLM serving, model serving, LLM inference serving, LLM serving inference, large language model inference serving
Infrastructure for efficiently deploying and serving large language model inference requests at scale.
LLM serving systems are the runtimes and schedulers that run large language model inference at scale, handling batching, KV cache memory, parallelism and latency targets. The sources are a steady stream of incremental work on a mature topic: lifetime-aware KV cache placement across HBM and high-bandwidth flash, memory paging for hybrid attention-SSM models in a modified vLLM, decode-jitter-aware scheduling, token-level routing from Tsinghua, agent-oriented infrastructure from MIT Han Lab, and engines for Trainium and unified-memory devices. The recurring sub-themes are agentic workloads and memory hierarchy, but the umbrella concept itself has been mainstream since vLLM and PagedAttention.
The umbrella term is already standard, so the rise mostly reflects ordinary research volume. The sub-trends inside it, such as agent-aware KV cache management and serving on non-GPU or unified-memory hardware, are more likely places for real change.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 105 on 14 Nov 2026 · 17 mentions in 28 days · 17 independent groups · papers → code · first seen 6 weeks ago · acceleration +0.79
Earliest sources
- Lachesis: Lifetime-Aware KV Cache Placement for Agent Serving across HBM and High-Bandwidth FlashPapers · 6 Oct 2026 · Jaehoon Yang, Jeongmin Lee, Haneul Park, Seung Yul Lee, Nam Sung Kim, Jae W. Lee
- thu-nics/TokenRouterCode · 8 Oct 2026 · thu-nics · 79 stars
- Native Dual Pools or Shared Paging? Memory Organization for Concurrent Hybrid LLM ServingPapers · 11 Oct 2026 · Dongwook Kang
- Scaling LLM Inference via Parallelism: Throughput-Latency Trade-offPapers · 14 Oct 2026 · Javad Mirzaei, Jeebak Mitra
- Accelerated Dataflow: scaling inference in streaming data analytics with efficiencyPapers · 15 Oct 2026 · RuiLong Jiang, Dustin Rhodes, Danny McCormick, Yuta Labur, Reza Rokni
- Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference FleetsPapers · 17 Oct 2026 · Mankeerat Sidhu, Gangmuk Lim, Hetarth Chopra, Maximillian Eisenberg, Quan Hao Ng…
Latest sources
- VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated SchedulingPapers · 13 Nov 2026 · Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh
- Scaling VLA Serving Under Robotic Execution ConstraintsPapers · 11 Nov 2026 · Yue Zhang, Zicong Ye, Lufang Chen, Dance Yang, Mingyang Li, Zhenwei Duan, Michae…
- mit-han-lab/EvolvingInfraCode · 10 Nov 2026 · mit-han-lab · 96 stars
- Inference Validation for Regulated AI: A Governed Decision Pipeline for LLM ServingPapers · 9 Nov 2026 · Dev Akre, Roman Bobalo, Manoj Dobbali
- simranjeet97/AI_Engineer_Roadmap_2027Code · 7 Nov 2026 · simranjeet97 · 37 stars
- Hibana: Fast Agentic Inference on Unified-Memory DevicesPapers · 7 Nov 2026 · Yahya Emara, Chi-Chih Chang, Anthony Fei, Ahmed F. AbouElhamayed, Chong-Yan Chen…
- SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems ResearchPapers · 6 Nov 2026 · Jinwoo Hwang, Sukmin Cho, Tao Zhang, Soheil Abbasloo, Jongse Park, Peng Cheng
- The GPU Needs an Operating SystemPapers · 5 Nov 2026 · Reza Nourai, Behnam Bastani
- cacheMon/PrefixSimCode · 4 Nov 2026 · cacheMon · 77 stars
- ServerBench: How can your coding agents design specialized LLM serving systems?Papers · 4 Nov 2026 · Xinrui Zhong, Zhiming Wang, Yongqi Liu, Zhaohui Wang, Shanglin Yang, Zikun Li, Z…
- Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast InterconnectsPapers · 2 Nov 2026 · Youhe Jiang, Pavle Glušac, Binhang Yuan, Eiko Yoneki
- RunanywhereAI/trnrtCode · 30 Oct 2026 · RunanywhereAI · 1,588 stars