Keyformer: KV Cache Reduction for Efficient Generative Inference

Keyformer introduces a novel approach to reduce KV cache size, improving inference latency and token generation throughput without compromising accuracy.