Core Concepts
Replication study of BASS framework reveals discrepancies in performance and challenges in replicating key components.
Abstract
The study replicates the BASS framework for abstractive summarization based on Unified Semantic Graphs. Challenges in replicating components are highlighted, including discrepancies in performance compared to the original work. The study emphasizes the importance of clear technical details and the impact of model architecture on replication results.
- Introduction to automatic text summarization and abstractive summarization systems.
- State-of-the-art abstractive summarization systems based on Pre-trained Language Models.
- Description of the BASS framework and its unique features.
- Replication study methodology and challenges faced.
- Evaluation of replication results and comparison with original work.
- Recommendations for writing replicable papers and addressing challenges in replication studies.
Stats
"One of the graph-enhanced transformer models is BASS, which is of specific interest because i) it introduces a compressed dependency graph structure based on the idea of semantic units and ii) the authors report competitive performance in abstractive summarization while being only half the size (201M parameters for BASS vs. 406M parameters for BART [19] and PEGASUS)."
"Our model ended up having approx. 205M trainable parameters, which is around 2% larger than reported in the BASS paper [30] (201M parameters), implying architectural differences we could not entirely resolve."
Quotes
"We conduct a replication study of the BASS framework and publish our implementation, including source code for the graph construction component provided by the authors of the original paper."
"Our results indicate the poor performance can be ascribed to the model architecture, which might, in addition, be undertrained."