Evaluating Faithfulness and Content Selection in Book-Length Summarization by Large Language Models

Core Concepts

Large language models can technically summarize book-length documents, but the length and complexity of the documents have so far prohibited evaluations of input-dependent aspects like faithfulness. This study conducts the first large-scale human evaluation of faithfulness and content selection on LLM-generated summaries of fictional books.

Abstract

The authors present FABLES, a dataset of human annotations on the faithfulness and content selection of LLM-generated summaries of 26 recently published fictional books. They hired annotators who had fully read each book prior to the annotation task to mitigate the issue of data contamination.
The study finds that CLAUDE-3-OPUS significantly outperforms all closed-source LLMs in terms of faithfulness, while the open-source MIXTRAL is on par with GPT-3.5-TURBO. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
The authors also implement several LLM-based raters of faithfulness, but find that none correlate strongly with human annotations, especially with regard to detecting unfaithful claims. This suggests that detecting unfaithful claims is an important future direction not only for summarization evaluation but also as a testbed for long-context understanding.
Beyond faithfulness, the study explores content selection errors in book-length summarization. The authors develop a typology of omission errors related to crucial narrative elements and identify a systematic over-emphasis on events occurring towards the end of the book.

Stats

The mean length of books in the dataset is 121,467 tokens.
The dataset contains 3,158 claim-level annotations across 26 books.

Quotes

None

Key Insights Distilled From

FABLES

by Yekyung Kim,... at arxiv.org 04-02-2024

https://arxiv.org/pdf/2404.01261.pdf

Deeper Inquiries

How can we improve automatic faithfulness evaluation to better detect unfaithful claims that require indirect reasoning over the narrative?

To enhance automatic faithfulness evaluation for detecting unfaithful claims that necessitate indirect reasoning over the narrative, several strategies can be implemented:

Contextual Understanding: Develop models that can comprehend and reason over long contexts effectively. This may involve incorporating memory mechanisms or attention mechanisms that can retain and utilize information from the entire document.

Multi-hop Reasoning: Implement models capable of multi-hop reasoning to connect disparate pieces of information within the text. This can help in validating claims that require indirect connections or inferences.

Fine-grained Evidence Retrieval: Improve the evidence retrieval process by identifying and extracting specific pieces of evidence from the text that directly support or refute a claim. This can aid in verifying claims that rely on nuanced details.

Semantic Understanding: Enhance the semantic understanding capabilities of the models to grasp the underlying meaning and implications of the text, enabling them to identify subtle inconsistencies or inaccuracies.

Adversarial Training: Train models using adversarial examples that specifically target indirect reasoning errors. This can help in making the models more robust and adept at detecting such challenging cases.

By incorporating these approaches, automatic faithfulness evaluation systems can be refined to better identify unfaithful claims that require complex reasoning over the narrative.

How can we characterize other types of content selection errors, beyond omissions, that might be present in book-length summarization?

In addition to omissions, other types of content selection errors that may occur in book-length summarization include:

Inaccurate Emphasis: Models may disproportionately focus on certain aspects of the text, such as events or characters, while neglecting other crucial elements. This can lead to a skewed representation of the narrative in the summary.

Chronological Discrepancies: Summaries may fail to maintain the correct chronological order of events, resulting in a disjointed or inaccurate portrayal of the story's progression.

Character Confusion: Models might incorrectly attribute actions or characteristics to the wrong characters, leading to confusion and misrepresentation of the relationships between characters.

Theme Misinterpretation: Misunderstanding or misinterpreting the central themes or motifs of the text can result in summaries that miss the essence or message conveyed by the original work.

Characterizing these content selection errors involves analyzing the summaries for patterns of inaccuracies, inconsistencies, and distortions in the representation of the source text. By identifying and categorizing these errors, researchers can gain insights into the limitations of current summarization models and work towards improving their content selection capabilities.

How might the findings from this study on book-length summarization apply to or differ from summarization of other long-form content, such as academic papers or legal documents?

The findings from the study on book-length summarization can have implications for summarization of other long-form content like academic papers or legal documents:

Complexity of Content: Similar to books, academic papers and legal documents contain intricate and detailed information that may require nuanced understanding and faithful representation in summaries.

Faithfulness Evaluation: The challenges in evaluating faithfulness and content selection in book summarization, such as indirect reasoning and multi-hop validation, may also be relevant in academic and legal document summarization.

Specificity of Information: Legal documents and academic papers often contain specialized terminology and domain-specific information, which may require tailored approaches for accurate summarization.

Structural Differences: Academic papers follow a specific structure with sections like abstract, introduction, methodology, results, and conclusion, which may influence the summarization process differently than a narrative book.

Legal Constraints: Legal documents have strict requirements for accuracy and precision, posing unique challenges for summarization in terms of legal terminology and implications.

While there are similarities in the summarization challenges across different types of long-form content, the specific nuances and requirements of academic papers and legal documents may necessitate tailored approaches and considerations in the summarization process.