同樣處理掃描件的 AI PDF 翻譯工具。

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best-performing models also connect the encoder and decoder through an attention mechanism [1].

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train [2].

Scaled dot-product attention is defined as:

where dₖ is the dimensionality of the key vectors; the scaling factor 1/√dₖ is used to prevent the softmax from entering regions where gradients are extremely small at high dimensions.

Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results by over 2 BLEU [3].

Experiments show that multi-head attention, compared to single-head, attends to information from different representation subspaces at different positions — particularly effective for modeling long-range dependencies.

主流的序列轉導模型基於複雜的循環神經網路或卷積神經網路，通常包含一個編碼器和一個解碼器。表現最好的模型還透過一種注意力機制將編碼器和解碼器連接起來 [1]。

我們提出一種全新的、結構簡單的網路架構 —— Transformer，它完全基於注意力機制，不再使用循環或卷積結構。在兩項機器翻譯任務上的實驗表明，本模型在品質上顯著優於既有方法，同時更易並行化、訓練時間顯著縮短 [2]。

縮放點積注意力（scaled dot-product attention）的定義如下：

其中 dₖ 為 key 向量的維度；縮放因子 1/√dₖ 用於在維度較大時防止 softmax 進入梯度極小的區域。

我們的模型在 WMT 2014 英德翻譯任務上取得了 28.4 BLEU 的成績，比當時最好的模型高出 2 個 BLEU 以上 [3]。

實驗結果表明，多頭注意力相比單頭能在不同子空間上同時關注不同位置的資訊，對長距離依賴的建模尤為有效。