Core Concepts
Optimized data placement is critical to harnessing the full potential of PIM acceleration for GEMV computations that dominate Generative AI inference. The proposed PIMnast methodology balances multiple factors to identify data placements that deliver up to 6.86x speedup for GEMVs, leading to up to 5x end-to-end speedup for Generative AI per-token latencies.
Abstract
The content discusses the importance of efficient data placement for accelerating general matrix-vector multiplication (GEMV) computations, which dominate Generative AI (GenAI) inference, using processing-in-memory (PIM) technology.
Key highlights:
GEMV computations demand high memory bandwidth, making them a critical target for PIM acceleration.
However, a key challenge in harnessing PIM acceleration is deducing the optimal data placement to map the matrix in memory banks.
The authors identify multiple factors that impact data placement, including PIM architecture, memory configuration, GenAI needs, and GEMV characteristics.
They propose the PIMnast methodology, which balances these factors to identify data placements that maximize GEMV-PIM acceleration.
PIMnast, coupled with additional orchestration knobs, delivers up to 6.86x speedup for GEMVs (of the available 7x roofline speedup), leading to up to 5x speedup for per-token latencies in GenAI models.
The authors also discuss software and system considerations to realize the PIMnast data placement, as well as potential hardware and software optimizations to address deficiencies for certain GenAI models.
Stats
The content does not contain any explicit numerical data or metrics. The key figures and results are presented in the form of speedup numbers compared to baseline SoC performance.
Quotes
The content does not contain any direct quotes.