Investigating and Resolving a Sawtooth Pattern in Renderer Service Response Times

Core Concepts

Proactive monitoring and detailed analysis are crucial for identifying and resolving performance issues in critical services, ensuring high-quality user experiences.

Abstract

The content describes the investigation process undertaken by the Typeform team to address a performance issue in their Renderer service, which is responsible for displaying forms to end users. After deploying a new version of the service, the team noticed a curious "Sawtooth" pattern in the response times, with sudden drops followed by gradual increases.
To tackle the issue, the team leveraged various tools and techniques:

Metrics and Tracing: The team used Datadog Metrics and distributed tracing to detect the problem and narrow down the issue, identifying a discrepancy between the time spent in the Renderer service and the downstream Forms API service.

Runtime Metrics: The team added Node.js runtime metrics to their toolkit, which provided insights into CPU usage, memory usage, and event loop delays, suggesting the presence of expensive object creation in the code.

Profiler: The Datadog Profiler allowed the team to compare flame graphs between different time frames, highlighting the trackPrivateForm function as the potential source of the problem.

After identifying the root cause, the team optimized the code by moving the creation of the RudderstackAnalytics instance outside the trackPrivateForm function, ensuring it is created only once. This optimization resolved the latency issue, and the Sawtooth pattern in the response times was no longer observed.
The key takeaways from this investigation are the importance of proactive monitoring, detailed analysis, and prompt issue resolution to maintain robust and efficient systems, even under heavy traffic.

Stats

The renderer was taking about 100ms to call the service which stores form definitions, forms-api, where it itself only took a few milliseconds to process the request.
During the period that the response time latency accumulated, the team observed an increase in CPU usage and the event loop delay, but no increase in memory usage.

Quotes

"Tools like Datadog Metrics are really great for detecting that there is a problem and even pinpointing the time it first occurred. We could even confirm and narrow down the issue with distributed tracing."
"The Datadog Profiler allows us to choose different time frames and compare their flame graphs. It even conveniently highlights functions in which more time is spent in a given sample compared to another."

Key Insights Distilled From

Latency Whack-a-Mole: Our Renderer’s Sawtooth Mystery Solved

by Arnau Guadal... at medium.com 07-04-2024

https://medium.com/typeforms-engineering-blog/typeforms-sawtooth-mystery-solved-8051550ce780

Deeper Inquiries

How can the team proactively monitor and detect performance issues in their services before they impact the user experience?

To proactively monitor and detect performance issues in their services, the team can implement a comprehensive monitoring strategy. This includes setting up alerts based on key performance metrics such as response time, CPU usage, memory usage, and event loop delays. By establishing thresholds for these metrics, the team can receive real-time notifications when any of them deviate from normal levels, allowing them to address potential issues before they impact the user experience. Additionally, leveraging tools like Datadog Metrics and distributed tracing can help in identifying anomalies and pinpointing the root cause of performance issues.

What other techniques or tools could the team have used to reproduce the issue locally and speed up the investigation process?

To reproduce the issue locally and expedite the investigation process, the team could have utilized techniques such as load testing and chaos engineering. Load testing involves simulating realistic user traffic to the service, which can help in identifying performance bottlenecks under different load conditions. Chaos engineering involves intentionally introducing failures or disruptions to the system to observe how it behaves under stress. By incorporating these techniques into their testing and debugging processes, the team can create more robust and resilient services that are better equipped to handle performance issues.

How can the team ensure that similar performance issues are prevented in the future, as they continue to evolve and update their services?

To prevent similar performance issues in the future, the team can implement best practices such as code reviews, performance testing, and continuous monitoring. Code reviews can help in identifying potential performance bottlenecks early in the development process, while performance testing can ensure that new changes do not degrade the system's performance. Continuous monitoring, using tools like Datadog Runtime Metrics and Profiler, can help in detecting and addressing performance issues as soon as they arise. By incorporating these practices into their development workflow, the team can maintain high-quality service for their users even as they continue to evolve and update their services.