P-MMEval: A Comprehensive Benchmark for Evaluating Multilingual Capabilities of Large Language Models Across Diverse Tasks
This paper introduces P-MMEval, a new benchmark designed to comprehensively evaluate the multilingual capabilities of large language models (LLMs) across a variety of tasks, addressing the limitations of existing benchmarks that primarily focus on English or specific aspects of language processing.
Evaluating the Risk-Taking Tendencies and Biases of Large Language Models Using Role-Play and Ethical Scales
This research introduces a novel approach to assess and quantify the risk-taking behaviors and inherent biases present within Large Language Models (LLMs) by employing role-playing scenarios and specialized ethical scales, revealing potential ethical concerns and avenues for improvement in LLM development.
Assessing the Linguistic Understanding of Large Language Models: A Psycholinguistic and Neurolinguistic Approach
Large language models (LLMs) demonstrate a stronger grasp of linguistic form (grammar) over meaning (semantics), suggesting their understanding of language relies heavily on statistical correlations rather than true conceptual understanding.
RoCar: A Relationship Network-Based Method for Evaluating the Reasoning and Memory Abilities of Large Language Models
RoCar is a novel evaluation method for Large Language Models (LLMs) that leverages randomly generated social network graphs to assess reasoning and memory capabilities, ensuring fairness by minimizing the chance of LLMs having pre-existing knowledge of the evaluation tasks.
UTMath: A Challenging Benchmark for Evaluating Mathematical Reasoning of Large Language Models Using Unit Tests and Reasoning-to-Coding Thoughts
Existing mathematical reasoning benchmarks for Large Language Models (LLMs) are limited in their ability to assess true reasoning capabilities, leading to the development of UTMath, a novel benchmark that utilizes unit tests and a Reasoning-to-Coding of Thoughts (RCoT) approach to robustly evaluate LLM reasoning skills.
Chinese SimpleQA: A Benchmark for Evaluating the Factuality of Chinese Large Language Models
This paper introduces Chinese SimpleQA, a new benchmark designed to evaluate the factuality of large language models (LLMs) when answering short questions in Chinese.
A New Benchmark and Evaluation Framework for Assessing Instruction Following and Stability of Large Language Models in Long-Context Scenarios
This paper introduces LIFBench, a novel benchmark designed to evaluate the instruction-following capabilities and stability of Large Language Models (LLMs) in long-context scenarios, along with LIFEval, a rubric-based evaluation framework for accurate and efficient assessment of LLM performance.
OpenAI-o1 AB Testing: An Analysis of the o1 Model's Reasoning Abilities in Math Problem Solving
The OpenAI o1-mini model demonstrates strong intuitive reasoning and problem-solving abilities in mathematics, comparable across public and private datasets, suggesting its capabilities extend beyond memorization, though it often struggles to provide complete, rigorous proofs.
ALI-Agent: An Agent-Based Framework for Evaluating the Alignment of Large Language Models with Human Values
ALI-Agent is an agent-based evaluation framework that leverages the autonomous abilities of LLM-powered agents to automatically generate realistic test scenarios and iteratively refine them to assess and identify misalignment between LLMs and human values in diverse real-world contexts.
SciEval: A Comprehensive Benchmark for Evaluating Large Language Models in Scientific Research
SciEval is a new multi-disciplinary benchmark designed to address limitations of existing LLM evaluation datasets in the context of scientific research by employing a multi-level evaluation system based on Bloom’s taxonomy, incorporating both objective and subjective questions, and utilizing dynamic data generation to minimize data leakage.
© 2024 by Linnk AI