On-device Inference of Large Language Models

ลงชื่อเข้าใช้

ข้อมูลเชิงลึก - On-device Inference of Large Language Models

Cambricon-LLM: A Chiplet-Based Hybrid Architecture for Efficient On-Device Inference of Large Language Models up to 70 Billion Parameters

Cambricon-LLM, a novel chiplet-based hybrid architecture, enables efficient on-device inference of large language models up to 70 billion parameters by combining a neural processing unit (NPU) and a dedicated NAND flash chip with on-die processing capabilities.

Efficient Deployment of Large Language Models on Mobile Devices

To enable high-efficiency deployment of large language models (LLMs) on mobile device GPUs, the paper proposes four key optimization techniques: (1) a symbolic expression-based approach for dynamic shape model inference, (2) operator optimizations and execution priority setting, (3) an FP4 quantization method to reduce dequantization overhead, and (4) a sub-tensor-based technique to eliminate the need for copying KV cache after LLM inference.

เกี่ยวกับ

ผลิตภัณฑ์

แหล่งข้อมูล