Core Concepts
Layer ensemble averaging can reliably boost the performance of defective memristive neural networks to near-software baseline levels by mitigating the impact of device non-idealities.
Abstract
The content presents a novel layer ensemble averaging technique to map pre-trained neural network solutions from software to defective hardware crossbars of emerging memory devices, such as memristors. The approach is investigated using a custom 20,000-device hardware prototyping platform on a continual learning problem, where a network must learn new tasks without catastrophically forgetting previously learned information.
The key highlights are:
- Layer ensemble averaging involves mapping the same pre-trained neural network solution multiple times onto the defective hardware crossbar, and then averaging the outputs from the non-defective rows to mitigate the impact of device non-idealities.
- Simulation results show that the layer ensemble approach can reliably tolerate up to 35% stuck devices in the hardware crossbar and boost the network's multi-task classification accuracy from 61% to 72% (within 1% of the software baseline).
- Experimental results on the custom 20,000-device hardware platform demonstrate that the layer ensemble network can attain near-software inference performance (within 1% of the software baseline) by trading off the number of devices required for the layer mapping.
- The proposed approach is effective in improving the performance of memristor-based neural networks and can be useful for other applications requiring accurate vector-matrix multiplication operations.
Stats
The average multi-task classification accuracy improves from 61% to 72% (< 1% of software baseline) using the proposed layer ensemble averaging approach.
Quotes
"Layer ensemble averaging β a technique to map pre-trained neural network solutions from software to defective hardware crossbars of emerging memory devices and reliably attain near-software performance on inference."
"For the investigated problem, the average multi-task classification accuracy improves from 61 % to 72 % (< 1 % of software baseline) using the proposed approach."