Introducing MILU: A Benchmark for Evaluating Cultural Understanding of Indic Language Models
Current large language models (LLMs) struggle to demonstrate adequate understanding of Indic languages and cultures, highlighting the need for a dedicated benchmark like MILU to drive progress in this area.
MORCELA: A New Linking Theory for Evaluating Language Model Acceptability Judgments
Language models' (LMs) probability scores can be better correlated with human acceptability judgments by using a new linking theory called MORCELA, which accounts for model-specific variations in sensitivity to sentence length and word frequency.
A Statistical Approach to Analyzing Language Model Evaluations and Reducing Variance
This article introduces a statistically rigorous framework for analyzing language model evaluations, advocating for the use of confidence intervals, paired statistical tests, and power analysis to improve the reliability and informativeness of model comparisons.
LINGOLY: A Benchmark Using Linguistics Olympiad Puzzles to Evaluate Reasoning Abilities of Large Language Models in Low-Resource and Extinct Languages
LINGOLY, a novel benchmark using Linguistics Olympiad puzzles, reveals that even state-of-the-art LLMs struggle with multi-step reasoning in low-resource languages, particularly when mitigating memorization.
Measuring Consistency in Human Evaluation of Language Models: Introducing the SEPARABILITY Metric
Inconsistent human evaluations of language models, particularly in pairwise comparisons, can be attributed to the difficulty in distinguishing between model outputs. The SEPARABILITY metric addresses this by quantifying the distinguishability of generations from different models on a given input, offering a measure of evaluation reliability and enabling more robust model comparisons.
MM-Eval: A 18-Language Benchmark for Evaluating Language Models as Judges of Text Quality
Existing benchmarks for evaluating language models as judges of text quality primarily focus on English, hindering the assessment of these models' effectiveness in multilingual contexts. MM-Eval addresses this gap by introducing a multilingual benchmark covering 18 languages and various linguistic challenges, revealing that both proprietary and open-source language models have significant room for improvement in multilingual settings.
DOLOMITES: A Benchmark for Evaluating Language Model Performance on Domain-Specific Methodical Writing Tasks
DOLOMITES is a novel benchmark designed to evaluate the capabilities of language models in assisting experts with complex, domain-specific writing tasks, revealing significant room for improvement in both model performance and automatic evaluation methods.
Evaluating the Sensitivity of Language Models to Argument Roles: A Psycholinguistic Approach
Large language models demonstrate some sensitivity to argument roles in sentence processing, but their performance differs significantly from human behavior, suggesting a reliance on lexical cues rather than a deep understanding of syntactic structure and argument role relationships.
Fisher Susceptibility: An Efficient Alternative to Monte Carlo Estimation for Measuring Context Sensitivity in Language Models
This paper introduces Fisher susceptibility, an efficient method for estimating the sensitivity of language models to input context, offering a faster alternative to the computationally expensive Monte Carlo approximation.
Evaluating the Alignment of Large Language Models with Human Language Acquisition Stages
While large language models (LLMs) have made significant strides in various language tasks, their developmental trajectory does not mirror human language acquisition. LLMs' capabilities are more influenced by training data and architecture than by mimicking the stages of human language development.
© 2024 by Linnk AI