
Kangaroo, a novel self-speculative decoding framework, uses a fixed shallow sub-network of the large language model as a self-draft model and introduces an additional early exiting mechanism to reduce the inference latency of the self-draft model, achieving significant speedups in large language model inference.


coremsg

kangaroo-efficient-self-speculative-decoding-for-accelerating-large-language-models


Kangaroo: Efficient Self-Speculative Decoding for Accelerating Large Language Models


title_rewrite


Speculative decoding with novel architectures that condition on both context vectors and sampled tokens can effectively predict high-quality n-grams, allowing 2-3x acceleration of highly optimized large language model inference in production settings.


accelerating-large-language-model-inference-with-combined-token-and-embedding-speculators


Accelerating Large Language Model Inference with Combined Token and Embedding Speculators
