Accurate Performance Modeling for Machine Learning Training on Multi-GPU Platforms

Core Concepts

This paper presents a comprehensive performance modeling pipeline that can accurately predict the per-iteration training time of modern machine learning workloads, such as deep learning recommendation models (DLRM) and transformer-based natural language processing (NLP) models, on multi-GPU platforms. The key innovations include modeling communication collectives, handling inter- and intra-rank synchronizations, and improving embedding lookup performance modeling.

Abstract

The paper addresses the challenges in characterizing and predicting the training performance of modern machine learning (ML) workloads on multi-GPU platforms. The primary challenges include the complexity of synchronization and load balancing between CPUs and GPUs, the variance in input data distribution, and the use of different communication devices and topologies.
The authors make the following key contributions:

Added performance models for communication operations (all-to-all and all-reduce) using sigmoid curve fitting.
Enhanced the critical-path-based end-to-end (E2E) performance modeling algorithm to accurately account for inter-rank and intra-rank synchronizations.
Improved embedding-table-lookup performance modeling to handle flexible lookup numbers and patterns (i.e., input data distribution) using an ML-based approach.
Supported extra minor ops such as layer norm and dropout for NLP models.

The performance model achieves a geomean prediction error of 5.21% on randomly generated industrial-scale DLRM workloads and 3.00% on Transformer-based NLP models across two multi-GPU platforms. The model is also capable of quickly selecting the fastest embedding table sharding configuration for DLRM training without actually running the workloads, with a success rate of 85%.

Stats

"The geomean prediction error of the performance model is 5.21% on randomly generated industrial-scale DLRM workloads and 3.00% on Transformer-based NLP models across two multi-GPU platforms."
"The performance model is capable of quickly selecting the fastest embedding table sharding configuration for DLRM training without actually running the workloads, with a success rate of 85%."

Quotes

"Communication collectives, like all-to-all and all-reduce across various network media (e.g., NVLink, PCIe, network cards) and topologies that connect multiple compute devices, are essential operations in multi-GPU training and commonly the performance hotspot."
"We claim that both inter- and intra-rank synchronizations are the keys to accurately modeling ML workloads training performance on multi-GPU, or even more broadly, all types of workloads running on multi-heterogeneous-device platforms."

Key Insights Distilled From

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms

by Zhongyi Lin,... at arxiv.org 04-22-2024

https://arxiv.org/pdf/2404.12674.pdf

Deeper Inquiries

How can the performance modeling techniques presented in this paper be extended to other types of heterogeneous computing platforms beyond multi-GPU systems, such as CPU-FPGA or CPU-TPU systems

The performance modeling techniques presented in the paper can be extended to other types of heterogeneous computing platforms beyond multi-GPU systems by adapting the models to account for the specific characteristics and communication patterns of the new platforms. For CPU-FPGA systems, the performance models would need to consider the unique architecture of FPGAs and the communication protocols between the CPU and FPGA. This could involve developing new performance models for FPGA-specific operations and optimizing data movement between the CPU and FPGA.
Similarly, for CPU-TPU systems, the performance models would need to incorporate the specialized hardware acceleration provided by TPUs and the efficient data transfer mechanisms between the CPU and TPU. This may require creating performance models for TPU-specific operations and optimizing the workload distribution between the CPU and TPU to maximize performance.
Overall, extending the performance modeling techniques to different heterogeneous computing platforms would involve understanding the architecture and communication patterns of each platform and tailoring the models to accurately predict the performance of machine learning workloads on these systems.

What are the potential limitations of the current performance modeling approach, and how could it be further improved to handle more complex ML workloads or system configurations

The current performance modeling approach may have limitations in handling more complex ML workloads or system configurations due to several factors. One potential limitation is the scalability of the models to larger and more diverse datasets. As the size and complexity of ML models continue to grow, the performance models may struggle to accurately capture the behavior of these workloads.
Another limitation could be the generalization of the models to different types of ML algorithms beyond the ones tested in the paper. More complex algorithms with unique computational patterns may require additional kernel performance models and optimizations to accurately predict their performance.
To further improve the performance modeling approach, researchers could explore incorporating more advanced machine learning techniques, such as deep learning models, to enhance the accuracy of the predictions. Additionally, optimizing the models for specific hardware configurations and workload distributions could improve the overall performance prediction accuracy.
Overall, continuous refinement and adaptation of the performance modeling approach to address these limitations will be crucial in handling more complex ML workloads and system configurations effectively.

Given the insights provided by the performance model, how could ML practitioners leverage this information to design more efficient hardware-software co-optimized systems for training large-scale ML models

ML practitioners can leverage the insights provided by the performance model to design more efficient hardware-software co-optimized systems for training large-scale ML models in several ways:

Hardware Selection: Based on the predicted performance of different hardware configurations, practitioners can choose the most suitable hardware setup for their specific ML workloads. This could involve selecting the optimal GPU configuration, memory allocation, and communication topology to maximize training efficiency.

System Optimization: By analyzing the performance bottlenecks identified by the model, practitioners can optimize the system configuration to reduce idle time and improve resource utilization. This may involve adjusting the workload distribution, data movement strategies, and synchronization mechanisms to enhance overall system performance.

Algorithm Design: The insights from the performance model can guide ML practitioners in designing more efficient algorithms that are tailored to the hardware architecture. By understanding the impact of different operations on performance, practitioners can optimize their algorithms to leverage the strengths of the hardware and minimize computational overhead.

Real-time Optimization: With the ability to quickly evaluate different sharding configurations and predict the impact on training time, practitioners can make real-time decisions to optimize the hardware-software co-design during the training process. This agile approach allows for rapid adjustments to improve training efficiency and reduce time-to-solution.

By leveraging the information provided by the performance model, ML practitioners can create more efficient and effective hardware-software co-optimized systems for training large-scale ML models, ultimately improving performance and reducing training costs.