How Twitter Revolutionized Real-Time Event Processing: Scaling to 400 Billion Events per Day

Core Concepts

Twitter has revolutionized its real-time event processing architecture to handle over 400 billion events per day, achieving low latency, high accuracy, and reduced operational costs.

Abstract

The article discusses how Twitter has evolved its event processing system to handle the massive scale of data it generates daily.
Key highlights:

Twitter processes approximately 400 billion events in real-time and generates a petabyte of data every day.
Twitter's previous architecture was based on the lambda architecture, with batch and real-time components. This led to challenges like data loss, inaccuracies, and high latency during high-traffic events.
The new architecture leverages both Twitter's data centers and Google Cloud Platform. It uses Pub/Sub to stream events to Google Cloud, where streaming data flow jobs perform real-time aggregation and store data in BigTable.
The new architecture provides lower latency, higher throughput, and better handling of late events compared to the old Heron-based topology. It also simplifies the design and reduces compute costs.
The migration to the hybrid architecture on Twitter's data centers and Google Cloud has enabled Twitter to process billions of events in real-time with low latency, high accuracy, and stability, while also reducing operational costs for engineers.

Stats

Twitter processes approximately 400 billion events in real time and generates a petabyte of data every day.

Quotes

"Twitter ad services are one of the top revenue models for Twitter, and if its performance is impacted, it directly impacts their business model."
"Twitter offers various data product services to retrieve information on impression and engagement metrics; these services would be impacted by inaccurate data."

Key Insights Distilled From

How Twitter improved the processing of 400 billion events.

by Mayank Sharm... at medium.com 03-30-2024

https://medium.com/@mayank.sharma2796/how-twitter-improved-the-processing-of-400-billion-events-621b3456c9dc

Deeper Inquiries

How does Twitter's new architecture compare to other real-time event processing systems used by large tech companies?

Twitter's new architecture stands out in comparison to other real-time event processing systems used by large tech companies due to its hybrid approach, leveraging both Twitter Data Center services and the Google Cloud platform. This combination allows Twitter to achieve high accuracy, low latency, stability, and reduced operational costs. The system can handle millions of events per second with low latency of around 10 ms, showcasing its scalability and efficiency. Additionally, the new architecture eliminates the need for maintaining different real-time event aggregations in multiple data centers, simplifying the design and reducing compute costs.

What were the key technical challenges Twitter faced in migrating from the old lambda architecture to the new hybrid architecture?

Twitter encountered several key technical challenges during the migration from the old lambda architecture to the new hybrid architecture. One significant challenge was related to real-time event processing, where the system faced backpressure issues during high-volume events like the FIFA World Cup. This led to latency within the topology and accumulation of spout lag, impacting system performance. Another challenge was in batch processing, where heavy computation pipelines processing petabytes of data could lead to data loss if sync jobs took longer than expected, causing back pressure and potential loss of data. Additionally, the TSAR query service had to consolidate data from Manhattan and the cache, risking inaccurate metrics due to potential loss of real-time data.

How does Twitter's approach to real-time event processing align with broader industry trends in data engineering and cloud-based architectures?

Twitter's approach to real-time event processing aligns well with broader industry trends in data engineering and cloud-based architectures by focusing on scalability, low latency, and cost efficiency. By leveraging a hybrid architecture combining Twitter Data Center services and the Google Cloud platform, Twitter demonstrates a forward-looking approach to handling large volumes of real-time events. The emphasis on real-time aggregation, high accuracy, stability, and reduced operational costs reflects the industry's shift towards cloud-based solutions for efficient data processing. The system's ability to handle millions of events per second with low latency and high throughput aligns with the industry's demand for scalable and reliable real-time event processing systems.