CPSDBench: A Comprehensive Evaluation Benchmark for Chinese Public Security Domain
The authors introduce CPSDBench, a specialized evaluation benchmark tailored for the Chinese public security domain, to assess Large Language Models (LLMs) across various tasks. The study aims to provide insights into the strengths and limitations of existing models in addressing public security issues.
Comprehensive Evaluation of Large Language Models: Metrics, Challenges, and Solutions
This paper provides a comprehensive exploration of evaluation metrics for Large Language Models (LLMs), offering insights into the selection and interpretation of metrics currently in use, and showcasing their application through recently published biomedical LLMs.
Evaluating Large Language Models Using Multiple Instruction Prompts: A Call for Robust and Meaningful Assessment
Single-prompt evaluation of large language models leads to unstable and unreliable results. A multi-prompt evaluation approach is necessary to provide a more robust and meaningful assessment of model capabilities.
Evaluating Large Language Model Outputs: Comparing Criteria Developed by Domain Experts, Lay Users, and the Models Themselves
Domain experts, lay users, and Large Language Models (LLMs) develop distinct sets of evaluation criteria for assessing LLM outputs, with domain experts providing the most detailed and specific criteria, lay users emphasizing formatting and clarity, and LLMs generating more generalized criteria based on prompt keywords.
Evaluating the Sequential Instruction Following Ability of Large Language Models with the SIFo Benchmark
Large language models (LLMs) struggle to follow sequences of instructions, even when those instructions are logically connected, highlighting a critical area for improvement in LLM robustness.
Evaluating Large Language Models: A Critical Review of Challenges and Recommendations for Reproducible, Reliable, and Robust Assessments
Evaluating large language models (LLMs) is complex and often inconsistent, hindering reproducibility, reliability, and robustness; this review identifies key challenges and provides recommendations for improved evaluation practices.
ProcBench: A Benchmark for Evaluating Multi-Step Reasoning and Instruction Followability in Large Language Models
ProcBench is a new benchmark designed to evaluate the ability of large language models (LLMs) to follow explicit multi-step instructions, revealing that while LLMs excel in knowledge-driven tasks, they struggle with complex procedural reasoning.
大型語言模型個人化回應生成評測：PersoBench
大型語言模型 (LLM) 雖然在流暢度和多樣性方面表現出色，但在個人化和連貫性方面，尤其是在考慮對話上下文和指定人物設定時，仍有很大進步空間。
A Novel Psychological Depth Scale for Evaluating Creative Writing in Large Language Models
The Psychological Depth Scale (PDS) is a novel framework for evaluating the psychological impact of stories generated by large language models, demonstrating that models like GPT-4 can achieve comparable or even surpass human-level narrative depth.
A New Benchmark Dataset, MMLU-SR, for Evaluating the True Reasoning Abilities of Large Language Models
Current large language models (LLMs) often rely on memorized terms and struggle to demonstrate true reasoning abilities when presented with unfamiliar symbols or concepts, highlighting the need for more robust evaluation methods like the proposed MMLU-SR benchmark.
© 2024 by Linnk AI