Crosslinguistic Analysis of the Relationship Between Word Predictability and Reading Times Across 11 Languages

Core Concepts

Surprisal and contextual entropy are consistent predictors of reading times across 11 languages from 5 language families, and the relationship between surprisal and reading times is linear.

Abstract

The study investigates the relationship between word predictability and reading times across 11 languages from 5 language families. The key findings are:

1. Surprisal, or negative log probability of a word given its context, is a consistent predictor of reading times across all 11 languages tested. Models that include surprisal as a predictor show significantly better predictive power over baseline models that do not.

2. Contextual entropy, or the expected surprisal of a word, also contributes to predicting reading times in most languages when added as an additional predictor. However, replacing surprisal with contextual entropy tends to hurt predictive power.

3. The relationship between surprisal and reading times is found to be linear across languages, contradicting some recent studies that have suggested non-linear relationships.

4. The magnitude of the surprisal effect, around 2-4 milliseconds per bit of surprisal, is remarkably consistent across languages, suggesting a stable crosslinguistic preference for the rate of information processing during reading.

5. The predictive power of surprisal shows some variation across languages, but this does not seem to be primarily driven by differences in language model quality across languages.

Overall, the results provide robust crosslinguistic evidence for the role of information-theoretic measures of word predictability in shaping reading behavior, and suggest that the core principles of surprisal theory generalize well beyond English.

Stats

"The time required to successfully comprehend a word is based on its predictability."
"Surprisal is strongly correlated with psychometric measurements in large naturalistic reading corpora."
"Contextual entropy, or the expected surprisal of a word, also correlates with reading times."
"The relationship between surprisal and reading time is linear."

Quotes

"Surprisal theory (Hale, 2001; Levy, 2008) posits that less predictable words should take more time to process, with word predictability quantified as surprisal, i.e., negative log probability in context."
"Pimentel et al. (2023) and Cevoli et al. (2022) have argued for what may be considered an expanded version of surprisal theory where processing difficulty is still determined by surprisal, but where people's reading behavior is additionally sensitive to expected surprisal (contextual entropy)."
"Smith and Levy (2013), Wilcox et al. (2020), and Shain et al. (2022) have found evidence that the linking function between reading times and surprisal is linear."

Key Insights Distilled From

Testing the Predictions of Surprisal Theory in 11 Languages

by Ethan Gotlie... at arxiv.org 09-12-2024

https://arxiv.org/pdf/2307.03667.pdf

Deeper Inquiries

How do the crosslinguistic differences in the predictive power of surprisal relate to other linguistic features of the languages, beyond just the language family?

The crosslinguistic differences in the predictive power of surprisal can be influenced by a variety of linguistic features beyond mere language family classification. For instance, syntactic structure plays a crucial role; languages with different word orders (e.g., SVO in English versus SOV in Turkish) may exhibit varying degrees of predictability based on their syntactic configurations. This can affect how readers anticipate upcoming words, thereby influencing reading times. Additionally, morphological complexity, such as the presence of extensive case marking in Finnish versus the relatively impoverished case system in English, can impact the predictability of words in context. Languages that are agglutinative, like Turkish and Korean, may have longer words with more complex structures, which could lead to higher surprisal values and longer reading times. Furthermore, lexical richness and frequency distributions of words within a language can also affect how surprisal correlates with reading times. For example, languages with a high frequency of function words may show different patterns in reading times compared to those with a more varied lexical inventory. Overall, these linguistic features contribute to the nuanced relationship between surprisal and reading times, highlighting the importance of considering a broader range of linguistic characteristics when analyzing crosslinguistic differences.

What other information-theoretic measures, beyond surprisal and contextual entropy, might also contribute to predicting reading times across languages?

In addition to surprisal and contextual entropy, several other information-theoretic measures could enhance the prediction of reading times across languages. One such measure is successor entropy, which quantifies the uncertainty regarding the next word given the current context. This measure can provide insights into how much information a reader anticipates needing to process as they encounter new words. Another relevant measure is information density, which assesses how information is distributed across a sentence or discourse. High information density may lead to increased cognitive load, thereby affecting reading times. Additionally, predictive coding frameworks, which consider the brain's predictions about incoming sensory information, could be integrated into reading time models. This approach posits that discrepancies between expected and actual input (prediction errors) can influence processing times. Lastly, contextual predictability, which evaluates how predictable a word is based on its surrounding context, could also serve as a valuable predictor. By incorporating these additional information-theoretic measures, researchers can develop a more comprehensive understanding of the cognitive processes underlying reading across different languages.

How do the findings from this study on reading times relate to information processing during other language modalities, such as speech production and comprehension?

The findings from this study on reading times have significant implications for understanding information processing in other language modalities, particularly speech production and comprehension. The consistent relationship between surprisal and reading times suggests that similar cognitive mechanisms may govern both reading and listening. In speech production, for instance, speakers often plan their utterances based on the predictability of upcoming words, which aligns with the principles of surprisal theory. Just as readers allocate processing time based on the predictability of words in a text, speakers may adjust their speech rates and pauses according to the anticipated difficulty of upcoming words. Furthermore, the concept of contextual entropy, which reflects the expected processing difficulty, is equally relevant in spoken language comprehension. Listeners rely on contextual cues to anticipate the flow of conversation, and higher contextual entropy may lead to longer processing times as they navigate through less predictable speech. Overall, the study's findings reinforce the idea that information-theoretic principles, such as surprisal and contextual entropy, are fundamental to understanding cognitive processes across various language modalities, highlighting the interconnectedness of reading, speech production, and comprehension.