Core Concepts
Comparing effect sizes and variability in on-line vs. lab studies reveals lower power in on-line experiments.
Abstract
The content discusses the feasibility of conducting language production studies online compared to traditional lab settings. It explores the impact of effect sizes, variability, and power in on-line experiments through a comparison of data collected in both settings. The study focuses on response times in a picture-word interference task and analyzes various factors affecting statistical power.
Author Contributions: Audrey Bürki focused on conceptualization, statistical analyses, and writing while Shravan Vasishth contributed to writing and editing.
Abstract: With the shift to online data collection due to the pandemic, assessing the feasibility of on-line experiments is crucial. This study compares response time data from lab and online settings to determine differences in effect sizes and variability.
Introduction: The pandemic has led to a surge in online data collection, raising concerns about noise levels impacting experimental results. The study aims to evaluate whether psycholinguistic effects can be reliably estimated using online data.
Experiments: Two datasets were collected - one in the lab and one online - for a picture-word interference task involving semantic and phonological manipulations.
Participants: German-speaking participants aged 18-30 were recruited for both lab-based and online studies.
Materials & Procedure: Participants named pictures with distractor words under different conditions across multiple blocks.
Analyses: Mixed-effects models were used to compare effect sizes between lab and online data, assess overall variability, within-participant consistency, between-participant variability, and residual error.
Results & Discussion: Findings indicate smaller effect sizes but higher residual variability in online data compared to lab data. Power simulations suggest lower statistical power for online experiments even with increased sample sizes.
Stats
The authors made the following contributions. Audrey Bürki: Conceptualization, Statistical analyses, Writing - Original Draft Preparation, Writing - Review & Editing; Shravan Vasishth: Writing - Review & Editing.
Quotes
"Data collected over the internet might allow detection of psycholinguistic effects but come with non-negligible costs."
"Our findings suggest that power is lower in an on-line setting than in the lab due to differences in effect sizes."