Uncovering a Pivotal Shift in American History: Analyzing the Language Patterns in State of the Union Addresses

Core Concepts

There is a significant discontinuity in the language and style of State of the Union addresses before and after the late 1920s, suggesting a fundamental shift in the nature of American governance and politics.

Abstract

The authors use natural language processing techniques, including BERT and GPT-2 embeddings combined with dimensionality reduction methods like UMAP, TriMAP, and PaCMAP, to analyze the State of the Union (SOTU) address dataset from Kaggle. Their analysis reveals a surprising finding - there is a sharp break in the language and style of SOTU addresses around 1927-1932, suggesting a major discontinuity in American history.
The authors first observe that addresses delivered by the same president are closely clustered, and those written in chronological proximity are also similar. However, the most striking result is the clear separation between addresses written before 1927 and those written after 1932, as shown in the UMAP and TriMAP visualizations.
The authors hypothesize that this shift may be due to two factors: 1) the increased use of speechwriters by presidents, starting with Franklin Roosevelt, and 2) the transformation of the United States from a remote, provincial country to a global superpower after World War II, leading to changes in the focus and emphasis of presidential addresses.
The authors also experiment with authorship attribution and year prediction tasks using fine-tuned DistilBERT models. They are able to achieve high accuracy (93-95%) in identifying the president who delivered a particular address, and reasonably good performance (RMSE of around 4.5 years) in predicting the year of an address, despite the relatively small amount of training data available for each president.
The authors conclude by acknowledging that they do not have a definitive explanation for the observed discontinuity, but they believe that there must be an underlying reason or reasons for this significant shift in the language and style of SOTU addresses over time.

Stats

The average length of a State of the Union address is around 8,358 words.
The amount of training data for each president varies greatly, from 1,790 words on average for John Adams to 22,614 words on average for William H. Taft.

Quotes

"What was surprising however, was that there is a large break, as demonstrated by the UMAP visualization (Figures 1a – 1b), between addresses written before 1927 and addresses written after 1932."
"We should note here that the amount of training data for each author is relatively small, and also varies greatly: from 1790 words in an average SOTU address for John Adams to 22614 words on average for William H. Taft [5]. The average length of a SOTU is ∼8358 words."

Key Insights Distilled From

A ripple in time: a discontinuity in American history

by Alexander Ko... at arxiv.org 05-07-2024

https://arxiv.org/pdf/2312.01185.pdf

Deeper Inquiries

What other historical or political factors might have contributed to the observed discontinuity in the language and style of State of the Union addresses

The observed discontinuity in the language and style of State of the Union addresses could be influenced by various historical and political factors. One significant factor could be the changing socio-political landscape of the United States during the early 20th century. Events such as the Great Depression, the New Deal policies of Franklin D. Roosevelt, and the aftermath of World War I might have shaped the language and priorities of the presidents delivering these addresses. The societal shifts, economic challenges, and political ideologies of different eras could have led to distinct changes in the rhetoric and themes addressed in the State of the Union speeches.
Additionally, the evolving role of the United States on the global stage could have impacted the content of these addresses. As the country transitioned from a more isolationist stance to a prominent world power post-World War II, the language used in the State of the Union speeches might have reflected this shift. The emphasis on international relations, foreign policy, and global leadership could have influenced the tone and messaging of the presidents during this period.

How might the increased use of speechwriters by presidents have influenced the content and tone of these addresses over time

The increased use of speechwriters by presidents could have had a significant impact on the content and tone of State of the Union addresses over time. Before the time of Franklin Roosevelt, it was uncommon for presidents to heavily rely on speechwriters. The speeches were often more reflective of the personal style and beliefs of the president delivering them. However, with the introduction of speechwriters, there was a shift towards more polished and structured speeches that might have been crafted to appeal to a broader audience or convey specific political messages.
Speechwriters could have brought a professional touch to the addresses, ensuring consistency in messaging and enhancing the rhetorical quality of the speeches. This professionalization of speechwriting could have led to a more strategic approach in crafting the State of the Union addresses, potentially influencing the language used, the themes addressed, and the overall impact on the audience.

Could the shift in the United States' global role and influence after World War II be reflected in other types of political or governmental discourse, beyond just the State of the Union addresses

The shift in the United States' global role and influence after World War II could indeed be reflected in other types of political or governmental discourse beyond just the State of the Union addresses. The changing dynamics of international relations, the emergence of the Cold War, and the country's position as a superpower would have likely influenced various aspects of political communication.
Other forms of political discourse, such as diplomatic communications, official statements, and public speeches by government officials, would have also mirrored the shift in the U.S.'s global role. The language used in treaties, agreements, and diplomatic negotiations may have adapted to reflect the country's new position in the world order. Additionally, public speeches by officials addressing international audiences or discussing foreign policy matters would have likely emphasized the United States' role as a global leader and its responsibilities on the international stage.