
Cambricon-LLM, a novel chiplet-based hybrid architecture, enables efficient on-device inference of large language models up to 70 billion parameters by combining a neural processing unit (NPU) and a dedicated NAND flash chip with on-die processing capabilities.


coremsg

cambricon-llm-a-chiplet-based-hybrid-architecture-for-efficient-on-device-inference-of-large-language-models-up-to-70-billion-parameters


Cambricon-LLM: A Chiplet-Based Hybrid Architecture for Efficient On-Device Inference of Large Language Models up to 70 Billion Parameters


title_rewrite


To enable high-efficiency deployment of large language models (LLMs) on mobile device GPUs, the paper proposes four key optimization techniques: (1) a symbolic expression-based approach for dynamic shape model inference, (2) operator optimizations and execution priority setting, (3) an FP4 quantization method to reduce dequantization overhead, and (4) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference.


efficient-deployment-of-large-language-models-on-mobile-devices


Efficient Deployment of Large Language Models on Mobile Devices
