Optimizing Throughput for Serving Small Language Models on Resource-Constrained Devices

Core Concepts

Small language models can reach Pareto-optimal throughput within the resource capacity of a single accelerator by leveraging large batches, enabling new optimization strategies for multi-model serving.

Abstract

The paper presents a set of experiments designed to benchmark the inference performance and energy consumption of small language models (SLMs) ranging from 125M to 13B parameters. The key findings are:

For SLMs, the Pareto-optimal throughput can be reached within the resource capacity of a single high-end accelerator (e.g., NVIDIA A100 GPU) by leveraging large batches of requests. Beyond this point, further increasing the batch size results in minimal or no improvement in throughput.

The small memory footprint of SLMs allows for larger batch sizes compared to large language models (LLMs), leading to higher arithmetic intensity and better resource utilization. This paves the way for new optimization strategies, such as partitioning GPU resources in multi-model serving.

An initial analysis on model replication shows that running multiple instances of the same SLM on a single device can improve both throughput and latency, by better utilizing the available GPU resources.

The energy consumption when serving SLMs is considerably lower than that of larger models, suggesting potential opportunities for power-efficient serving of small models.

The authors leave as future work the analysis of more realistic serving scenarios with heterogeneous requests and devices, as well as the exploration of model replication techniques like Multi-Process Service (MPS) or Multi-Instance GPU (MIG) for optimal GPU utilization.

Stats

The maximum batch size that can fit in a 40GB A100 GPU ranges from 512 for OPT-125M to 16 for OPT-13B.
The average GPU power usage when serving OPT-125M and OPT-1.3B is lower than when serving larger models like OPT-2.7B and OPT-6.7B.

Quotes

"For OPT 125M, OPT 1.3B and OPT 2.7B, this throughput frontier appears within the resource capacity of a single accelerator. Beyond this point doubling the size of the batch results in minimal or no improvement."
"We observe a considerable increase in GPU utilization in the three models as we increase the number of replicas."

Key Insights Distilled From

Towards Pareto Optimal Throughput in Small Language Model Serving

by Pol G.Recase... at arxiv.org 04-05-2024

https://arxiv.org/pdf/2404.03353.pdf

Deeper Inquiries

How can the insights from this work be applied to optimize serving of heterogeneous language models with varying resource requirements on a shared hardware infrastructure

The insights gained from this study can be instrumental in optimizing the serving of heterogeneous language models with varying resource requirements on a shared hardware infrastructure. By understanding the memory allocation and batch size dynamics that lead to Pareto-optimal throughput for small language models, one can tailor the resource allocation strategy for different models. For models with lower resource requirements, such as OPT-125M, a single high-end accelerator may suffice to reach the optimal throughput frontier. On the other hand, larger models like OPT-13B may benefit from distributing the serving across multiple GPUs to meet the memory demands for larger batch sizes.
To optimize serving for heterogeneous language models, one could implement a dynamic resource allocation system that intelligently assigns resources based on the model's size and memory requirements. By monitoring the memory utilization and throughput performance of each model, the system can allocate resources efficiently to maximize overall throughput while minimizing latency. This adaptive resource allocation approach can ensure that each model receives the necessary resources to operate at peak efficiency without over-provisioning hardware.

What are the potential trade-offs between power efficiency, throughput, and latency when serving small language models on lower-end or embedded devices

When serving small language models on lower-end or embedded devices, there are potential trade-offs between power efficiency, throughput, and latency that need to be carefully considered. Lower-end devices may have limited memory and processing capabilities, which can impact the performance of serving small language models.
In terms of power efficiency, serving small language models on lower-end devices may require optimizing the model architecture and inference process to minimize energy consumption. Techniques such as quantization, sparsity, and model distillation can be employed to reduce the model size and computational complexity, leading to improved power efficiency. However, these optimizations may come at the cost of increased latency due to the additional processing required for model compression and decompression.
Balancing throughput and latency on lower-end devices involves finding the optimal batch size that maximizes the number of requests processed per unit time while keeping the latency within acceptable limits. By experimenting with different batch sizes and monitoring the device's performance metrics, one can strike a balance between throughput and latency that aligns with the device's capabilities.

How can the model replication techniques explored in this work be combined with other optimization strategies, such as quantization or sparsity, to further improve the serving performance of small language models

The model replication techniques explored in this work can be combined with other optimization strategies, such as quantization or sparsity, to further enhance the serving performance of small language models. By replicating the same model across multiple instances on a shared hardware infrastructure, one can leverage the benefits of parallel processing to improve overall throughput and reduce latency.
When combined with quantization, which compresses the model weights to reduce memory footprint, model replication can enable the deployment of multiple quantized instances of the same model on a single device. This approach can lead to significant improvements in resource utilization and inference speed, especially on devices with limited memory and processing power.
Similarly, integrating sparsity techniques, which prune unimportant connections in the model, with model replication can further optimize the serving performance of small language models. By running sparse instances of the model in parallel, one can achieve higher throughput and lower latency while maintaining power efficiency.
Overall, the synergy between model replication and other optimization strategies can unlock new possibilities for serving small language models efficiently on a shared hardware infrastructure.