Core Concepts
Evaluating the performance of 16 open-source Generative Pre-Trained Transformer (GPT) models in translating 50 different non-English languages into English text without any custom fine-tuning.
Abstract
This study examines the capabilities of 16 different open-source Generative Pre-Trained Transformer (GPT) models in performing automated, zero-shot, black-box, sentence-wise translation from 50 non-English languages into English text. The models were evaluated using translated TED Talk transcripts as the reference dataset, with no custom fine-tuning applied.
The key highlights and insights from the study are:
The best overall performing GPT model for translating into English text was ReMM-v2-L2-13B, with mean BLEU, GLEU, chrF, and METEOR scores of 0.152, 0.256, 0.448, and 0.438 respectively across all 50 languages.
The GPT model translations were compared against the Google Translate API, and the GPT models performed comparably or better for some languages like French and Chinese.
Several GPT models, such as the phi models and Llama-2-13b-chat-hf, consistently performed poorly across the different languages.
The languages that the GPT models struggled the most with were Mongolian, Burmese, Kazakh, Kurdish, Armenian, and Georgian.
The slowest GPT models for translation were phi-1, phi-2, phi-1 5, zephyr-7b-beta, and falcon-7b-instruct.
The study demonstrates the potential of using local, offline GPT models for automated multi-language translation, while also highlighting the limitations of the current models.
Stats
This is a photograph from the viking lander on the surface of mars
there is intriguing evidence suggesting that the early history of mars may have had rivers and streams of water
there is no water liquid on the surface of mars today
i want to talk about one of the greatest myths of medicine and that is the idea that all we need are additional medical procedures and then all our problems will be solved
Quotes
This is a photograph from the viking lander on the surface of mars
there is intriguing evidence suggesting that the early history of mars may have had rivers and streams of water
there is no water liquid on the surface of mars today
i want to talk about one of the greatest myths of medicine and that is the idea that all we need are additional medical procedures and then all our problems will be solved