Tokenization Disparities Contribute to Large Language Model Misgendering of Underrepresented Pronouns

Byte-Pair Encoding (BPE) tokenization, the dominant tokenizer for popular large language models, disproportionately fragments neopronouns compared to binary pronouns due to data scarcity. This tokenization disparity is strongly associated with language models' inability to correctly use neopronouns, leading to higher misgendering rates.