The AI PDF translator that handles your scans, too.

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best-performing models also connect the encoder and decoder through an attention mechanism [1].

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train [2].

Scaled dot-product attention is defined as:

where dₖ is the dimensionality of the key vectors; the scaling factor 1/√dₖ is used to prevent the softmax from entering regions where gradients are extremely small at high dimensions.

Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results by over 2 BLEU [3].

Experiments show that multi-head attention, compared to single-head, attends to information from different representation subspaces at different positions — particularly effective for modeling long-range dependencies.

主流的序列转导模型基于复杂的循环神经网络或卷积神经网络，通常包含一个编码器和一个解码器。表现最好的模型还通过一种注意力机制将编码器和解码器连接起来 [1]。

我们提出一种全新的、结构简单的网络架构 —— Transformer，它完全基于注意力机制，不再使用循环或卷积结构。在两项机器翻译任务上的实验表明，本模型在质量上显著优于既有方法，同时更易并行化、训练时间显著缩短 [2]。

标度点积注意力（scaled dot-product attention）的定义如下：

其中 dₖ 为 key 向量的维数；缩放因子 1/√dₖ 用于在维度较大时防止 softmax 进入梯度极小的区域。

我们的模型在 WMT 2014 英德翻译任务上取得了 28.4 BLEU 的成绩，比当时最好的模型高出 2 个 BLEU 以上 [3]。

实验结果表明，多头注意力相比单头能在不同子空间上同时关注不同位置的信息，对长距离依赖的建模尤为有效。