Comprehensive Reliability Analysis and Fault-Tolerant Approaches for Vision Transformers

Vision Transformers are generally more resilient to soft errors in linear computing like GEMM and FC, but more fragile in non-linear computing like softmax and GELU compared to typical CNNs. A lightweight block-wise ABFT approach and a range-based protection scheme can significantly improve the accuracy of Vision Transformers in the presence of soft errors.