Early-Signal Detector

method · first seen 7 Oct 2026

expert parallelism

Also: EP, wide EP, expert-parallel serving, mixture-of-experts parallelism

Distributing the experts of a mixture-of-experts model across multiple devices for parallel execution.

Expert parallelism shards the experts of a mixture-of-experts layer across GPUs, with all-to-all dispatch and combine collectives moving tokens to the devices that own their experts. The technique itself dates back to GShard and Switch Transformer. What these sources add is a steady stream of systems work around its costs. Papers address load imbalance from hot experts, including in-switch replication in TensorRT-LLM and per-step balancing solvers. Others target communication-compute overlap, one-sided NVLink all-to-all from NVIDIA authors, and attention-FFN disaggregation on GB200 NVL72. Further work covers EP on commodity GPUs and MoE training at AMD-scale clusters and in AI2's Olmo-core. A small repo builds AFD communication on DeepEP.

The concept is mature; the activity reflects how central MoE serving and training efficiency has become, with communication and load balance under wide EP as the main bottlenecks. Watch sub-ideas like wide EP on rack-scale NVLink and attention-FFN disaggregation rather than EP itself.

Analyst note (established), 17 Nov 2026.

Mentions per week

0819 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 51 on 14 Nov 2026 · 13 mentions in 28 days · 13 independent groups · papers → code · first seen 5 weeks ago · acceleration +1.42

Earliest sources

Latest sources

Mentioned together with

mixture of expertsattention-FFN disaggregationall-to-all communicationprocedural weight generationelastic scalingnear-memory processingquantizationpersistent kernelcommunication-computation overlapexpert replication