method · first seen 6 Oct 2026
continuous batching
Serving technique that dynamically adds and removes requests from a running batch to raise LLM inference throughput.
Mentions per week
PapersCodeDiscussion
Earliest sources
- DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized BatchingPapers · 6 Oct 2026 · Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager
- basecompute/superfluidCode · 8 Oct 2026 · basecompute · 31 stars
- Dissecting the Carbon-SLO Tension in Geo-Distributed LLM ServingPapers · 10 Oct 2026 · Burak Topcu, Musa Cim, Zuhal Ozturk, Mahmut Taylan Kandemir
- Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU ArchitecturesPapers · 11 Oct 2026 · Ashman Mehra, Akshan Agrawal, Aryeh Carmi
- Quota-Based Aging for Prediction-Guided Scheduling in LLM ServingPapers · 13 Oct 2026 · Hanzhi Zhang, Anxuan Che, Hang Qi, Qiyue Li, Jingjin Wu
- priyammaz/tinyserveCode · 20 Oct 2026 · priyammaz · 32 stars
Latest sources
- High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel FusionPapers · 15 Nov 2026 · Aditya Chatterjee, Sanjay Singh Chauhan, Mayank Jain, Arushi Dhingra, Sahin Chow…
- Scaling Shapley Energy Attribution to Large LLM BatchesPapers · 10 Nov 2026 · Qi Luo, Ziwen Wang, Kunlin Li, Changxi Liu, Yun Chen
- eelgaev/LegionCode · 4 Nov 2026 · eelgaev · 21 stars
- DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM ServingPapers · 24 Oct 2026 · YuXuan Lin, KeQiang Wu, Kingsum Chow
- LLMs in Cloud: A Discussion on Architecture, Orchestration, and InferencePapers · 24 Oct 2026 · Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque, Sreezon Das Gupta, A…
- AkiYangDev/mini-deepinferCode · 20 Oct 2026 · AkiYangDev · 31 stars
- priyammaz/tinyserveCode · 20 Oct 2026 · priyammaz · 32 stars
- Quota-Based Aging for Prediction-Guided Scheduling in LLM ServingPapers · 13 Oct 2026 · Hanzhi Zhang, Anxuan Che, Hang Qi, Qiyue Li, Jingjin Wu
- Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU ArchitecturesPapers · 11 Oct 2026 · Ashman Mehra, Akshan Agrawal, Aryeh Carmi
- Dissecting the Carbon-SLO Tension in Geo-Distributed LLM ServingPapers · 10 Oct 2026 · Burak Topcu, Musa Cim, Zuhal Ozturk, Mahmut Taylan Kandemir
- basecompute/superfluidCode · 8 Oct 2026 · basecompute · 31 stars
- DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized BatchingPapers · 6 Oct 2026 · Jingpo Xu, Paul Joe Maliakel, Ivona Brandic, Shashikant Ilager