Evaluating Language Model Embeddings: Variance and Invariance to Semantic and Lexical Alterations

Language models face challenges in precisely understanding the semantics of language, exhibiting different behaviors for semantically equivalent sentences with varying syntactic/lexical structures. The VISLA benchmark systematically evaluates the ability of language models to distinguish semantic and lexical variations in text.