method · first seen 7 Oct 2026
expert parallelism
Also: EP, wide EP, expert-parallel serving, mixture-of-experts parallelism
Distributing the experts of a mixture-of-experts model across multiple devices for parallel execution.
Expert parallelism shards the experts of a mixture-of-experts layer across GPUs, with all-to-all dispatch and combine collectives moving tokens to the devices that own their experts. The technique itself dates back to GShard and Switch Transformer. What these sources add is a steady stream of systems work around its costs. Papers address load imbalance from hot experts, including in-switch replication in TensorRT-LLM and per-step balancing solvers. Others target communication-compute overlap, one-sided NVLink all-to-all from NVIDIA authors, and attention-FFN disaggregation on GB200 NVL72. Further work covers EP on commodity GPUs and MoE training at AMD-scale clusters and in AI2's Olmo-core. A small repo builds AFD communication on DeepEP.
The concept is mature; the activity reflects how central MoE serving and training efficiency has become, with communication and load balance under wide EP as the main bottlenecks. Watch sub-ideas like wide EP on rack-scale NVLink and attention-FFN disaggregation rather than EP itself.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 51 on 14 Nov 2026 · 13 mentions in 28 days · 13 independent groups · papers → code · first seen 5 weeks ago · acceleration +1.42
Earliest sources
- Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token ShufflingPapers · 7 Oct 2026 · Radha Gulhane, Quentin Anthony, Beren Millidge
- Democratizing MoE inference on commodity GPUs with CoMoEPapers · 7 Oct 2026 · Ruwen Fan, Yuezhi Zu, Minhui Xie, Junru Li, Qingda Hu, Xinjun, Yang, Jiwu Shu, Y…
- Zepp: Accelerating Distributed MoE Serving under Relaxed Balance ConstraintsPapers · 8 Oct 2026 · Chang Chen, Andrew Yang, Tiancheng Chen, Jiangfei Duan, Xinwei Qiang, Zhongkai Y…
- FORGE: Procedural Weight Generation for Sparse MoE InferencePapers · 12 Oct 2026 · Chanyoung Moon, Travis Pence, Daisuke Yamada, Jiaqi Mo, Yanxin Zhang, Vikas Sing…
- FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE ServingPapers · 13 Oct 2026 · Yunfei Feng, Jingkai He, Shiqi Liu, Cheng Zhang, Dong Du, Erhu Feng
- 3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMPPapers · 19 Oct 2026 · Yudong Pan, Lian Liu, Shixin Zhao, Yintao He, Kaiyan Chang, Tianhua Han, Chenyan…
Latest sources
- Supercharging Olmo-core for Efficient and Scalable MoE TrainingPapers · 13 Nov 2026 · Tianhua Tao, Akshita Bhagia, Dirk Groeneveld, Pete Walsh, Tyler Romero, Yashas S…
- MoE Pretraining:Parallelism Layout Changes the Model, Not QualityPapers · 11 Nov 2026 · Esther Xin
- One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLinkPapers · 9 Nov 2026 · Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingF…
- OverlapAdvisor: Modeling and Adapting Computation-Communication Overlap for Distributed MoE ServingPapers · 9 Nov 2026 · Chutong Ding, Kaixuan Zhang, Luping Wang, Shiyou Qian, Jian Cao, Cheng Huang, Gu…
- A Per-Step Load-Balancing Engine for Expert-Parallel MoE ServingPapers · 8 Nov 2026 · Jingyuan Li, Maolin Wang, Hayden So
- ISER-MoE: Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-ParallelPapers · 6 Nov 2026 · Jintao Peng, Ethan Zhang, Kefeng Duan, Bangyu Sheng, Min Li, Itamar Rosenfeld Ra…
- KOSMOS: Self-scheduling Persistent Kernels for Communication-Compute OverlapPapers · 5 Nov 2026 · Alexander Magro, Aristotle Martin, Ye Wang, Wen Chen
- FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?Papers · 4 Nov 2026 · Yichao Fu, Yuxuan Zhang, Minshen Zhang, Ruitian Wang, Junda Chen, Hao Zhang
- AdaFloor: Adaptive Floor Planning and Rank-Time Reuse for Long-Tail MoE RL RolloutsPapers · 4 Nov 2026 · Jia Wei, Ziyi Qiu, Yi Xu, Xuehai Qian
- GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green ContextsPapers · 3 Nov 2026 · Yong Li, Kan Liu, Tao Lan
- LRMoE: Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained NodesPapers · 27 Oct 2026 · Jiaming Hu, Zhida Jiang, Zhaolong Xing, Yu Li, Zhen Chen, Ke Zhang
- afdEP/afdEP-commCode · 22 Oct 2026 · afdEP · 101 stars