
Per-tensor quantization method, FlattenQuant, significantly improves inference efficiency for large language models by reducing memory consumption and latency.


coremsg

flattenquant-achieving-efficient-inference-for-large-language-models-with-per-tensor-quantization


FlattenQuant: Achieving Efficient Inference for Large Language Models with Per-tensor Quantization


title_rewrite


Clustered Head Attention (CHAI) reduces memory and compute requirements in Large Language Models (LLMs) by clustering correlated attention heads.


chai-clustered-head-attention-for-efficient-llm-inference


CHAI: Clustered Head Attention for Efficient LLM Inference
