method · first seen 28 Oct 2026
fully sharded data parallelism
Also: FSDP, ZeRO, FSDP2, sharded data parallelism
Distributed training that shards model parameters, gradients, and optimizer states across devices.
Mentions per week
PapersCodeDiscussion
Earliest sources
- The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitchPapers · 28 Oct 2026 · Zejian Wang, Yifan Zhao, Kaihui Ren, Xiaoling Zhou, Yubin Wang, Chenhao Guan, Li…
- Dobb: Shard-Local Preconditioning in Block-Diagonal EigenbasesPapers · 29 Oct 2026 · Shi-Yuk Fong, Shuhua Yu, Jiawei Zhang
- DMuon-2: Scaling Matrix Optimizers in Data Parallelism with Topology-Aware Matrix Ownership and CommunicationPapers · 4 Nov 2026 · Starrick Liu, Ruizhi Chengze, Lufang Chen, Michael Zhao, Ran Yu, Xiaofan Li, Dan…
- SS-HSDP: Efficient and Flexible Sharded Data Parallelism via Runtime-Update DecouplingPapers · 12 Nov 2026 · Chendong Xiang, Pengle Zhang, Jinliang Li, Zichen Liang, Jintao Zhang, Fan Bao,…
Latest sources
- SS-HSDP: Efficient and Flexible Sharded Data Parallelism via Runtime-Update DecouplingPapers · 12 Nov 2026 · Chendong Xiang, Pengle Zhang, Jinliang Li, Zichen Liang, Jintao Zhang, Fan Bao,…
- DMuon-2: Scaling Matrix Optimizers in Data Parallelism with Topology-Aware Matrix Ownership and CommunicationPapers · 4 Nov 2026 · Starrick Liu, Ruizhi Chengze, Lufang Chen, Michael Zhao, Ran Yu, Xiaofan Li, Dan…
- Dobb: Shard-Local Preconditioning in Block-Diagonal EigenbasesPapers · 29 Oct 2026 · Shi-Yuk Fong, Shuhua Yu, Jiawei Zhang
- The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitchPapers · 28 Oct 2026 · Zejian Wang, Yifan Zhao, Kaihui Ren, Xiaoling Zhou, Yubin Wang, Chenhao Guan, Li…