architecture · first seen 4 Oct 2026
softpick attention
Also: Softpick
Attention variant replacing softmax with rectified scores to eliminate attention sinks and massive activations.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Underscoring the Problem: Why Softpick Fails at InitializationPapers · 4 Oct 2026 · Aryan Sood, Jaikaran Singh, Ishaan Bansal
Latest sources
- Underscoring the Problem: Why Softpick Fails at InitializationPapers · 4 Oct 2026 · Aryan Sood, Jaikaran Singh, Ishaan Bansal