Low Latency Attention Module for Streaming Self-Supervised Speech Representation Learning
Efficient low-latency attention module proposed for streaming self-supervised speech representation learning.
SimA: Simple Softmax-free Attention for Vision Transformers
Introducing SimA, a Softmax-free attention block for vision transformers, simplifying computation and achieving on-par results with SOTA models.
Efficient Attention Mechanism with Constant Cost per Token
A simple modification to the conventional attention mechanism enables its linearization as a composition of log-sums of exponentials, with a fixed-size latent space, for sequential application with constant cost per token.
SageAttention: Accelerating Transformer Inference Using 8-Bit Quantized Attention for Improved Efficiency
SageAttention is a novel quantization method that significantly accelerates transformer inference by employing 8-bit quantization for the attention mechanism while preserving accuracy, outperforming existing methods like FlashAttention2 and xformers in both speed and accuracy.
미러 디센트를 이용한 어텐션 최적화: 일반화된 최대 마진 토큰 선택
본 논문에서는 소프트맥스 어텐션 메커니즘을 위한 미러 디센트(MD) 알고리즘 군의 수렴 속성과 내재적 편향을 분석하여, ℓp-AttGD가 일반화된 하드 마진 SVM 솔루션으로 수렴하고, 특히 ℓ1.1-MD가 일반적인 경사 하강법보다 우 뛰어난 일반화 성능과 토큰 선택 능력을 보여준다는 것을 입증합니다.
