Zaloguj się
spostrzeżenie - On-device Inference of Large Language Models
Cambricon-LLM: A Chiplet-Based Hybrid Architecture for Efficient On-Device Inference of Large Language Models up to 70 Billion Parameters
Cambricon-LLM, a novel chiplet-based hybrid architecture, enables efficient on-device inference of large language models up to 70 billion parameters by combining a neural processing unit (NPU) and a dedicated NAND flash chip with on-die processing capabilities.
Efficient Deployment of Large Language Models on Mobile Devices
To enable high-efficiency deployment of large language models (LLMs) on mobile device GPUs, the paper proposes four key optimization techniques: (1) a symbolic expression-based approach for dynamic shape model inference, (2) operator optimizations and execution priority setting, (3) an FP4 quantization method to reduce dequantization overhead, and (4) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference.
Produkty
© 2024 by Linnk AI