Author of "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free"
How this journalist typically writes
Qwen Team, Alibaba Group as author
Applying a head-specific sigmoid gate after Scaled Dot-Product Attention consistently improves LLM performance, training stability, and long-context extrapolation while mitigating attention sink effects.
“Author of "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free"”