Optimizing Throughput for Constraint-Aware Execution of Large Language Models
ExeGPT, a distributed system, finds and runs with an optimal execution schedule to maximize inference throughput while satisfying a given latency constraint by leveraging the distribution of input and output sequences to effectively allocate resources and determine optimal execution configurations.
Optimizing Key-Value Cache for Efficient Large Language Model Inference via Layer-Wise Adaptive Budget Allocation
By identifying the importance of attention layers, SQUEEZEATTENTION optimizes the KV-cache jointly from both the sequence and layer dimensions, achieving significant memory and throughput improvements for LLM inference.
Efficient Tree-Attention Algorithm for Accelerating Large Language Model Inference with Tree Search
DEFT, an IO-aware tree attention algorithm, reduces memory access redundancy in tree-based decoding by leveraging the tree topology to minimize KV cache IO and eliminate IO of partial results during attention calculations.
Efficient Quantization of Large Language Models with Outliers Using Block Floating Point Formats
A novel approach to enable usage of low-precision block floating point formats without compromising the resulting model accuracy, by exploiting the common channel-wise patterns exhibited by outliers in weights and activations.
© 2024 by Linnk AI