Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues

Core Concepts

Facial expression recognition (FER) models can be made more interpretable by aligning their internal feature representations with spatial cues derived from facial action units, without requiring additional manual annotations.

Abstract

The paper proposes a generic learning strategy to build an interpretable deep classifier for facial expression recognition (FER). The key idea is to explicitly incorporate spatial action unit (AU) cues into the classifier's training to build a deep interpretable model.
The main steps are:

Construct a spatial discriminative heatmap that indicates the most discriminative regions of interest (ROIs) in the input image with respect to the facial expression, based on the image class label and a codebook associating action units to facial expressions.

Constrain the spatial layer features of the classifier to be correlated with the AU heatmap during training, using a composite loss function that balances classification accuracy and attention alignment with the AU map.

This is achieved without any extra manual annotation cost, by leveraging the image class label and facial landmarks. The resulting classifier yields an interpretable layer-wise attention map that aligns with the expert's decision process for assessing facial expressions.
The authors also explore Class-Activation Mapping (CAM)-based classifiers, and show that their training technique improves the CAM interpretability as well.
Extensive evaluation on two public benchmarks, RAF-DB and AffectNet, demonstrates that the proposed strategy can improve layer-wise interpretability without degrading classification performance. In fact, classification accuracy is improved, particularly on the larger AffectNet dataset, suggesting that the spatial action units are a reliable source of discriminative ROIs for basic facial expression recognition.

Stats

Facial expression recognition achieves high accuracy but lacks interpretability, an important aspect for end-users.
Experts rely on a codebook associating spatial action units to facial expressions to assess expressions.
The proposed method builds a spatial discriminative heatmap from the image class label and facial landmarks, and aligns the classifier's spatial features with this heatmap during training.
This is achieved without any extra manual annotation cost.
Experiments on RAF-DB and AffectNet show improved interpretability, as measured by attention and CAM alignment with action units, without degrading classification accuracy.

Quotes

"Unlike standard non-interpretable FER classifiers, we propose a generic learning strategy to build an accurate but, most importantly, interpretable deep classifier."
"Our method does not: add any extra manual annotation, add significant extra computations during training, or change the model architecture or the inference process. In addition, our method is generic. It can be used with any deep CNN or transformer-based model."
"Empirical results showed that both classification and interpretability improve with our method."

Key Insights Distilled From

Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues

by Soufiane Bel... at arxiv.org 04-26-2024

https://arxiv.org/pdf/2402.00281.pdf

Deeper Inquiries

How could the proposed approach be extended to handle more complex facial expressions beyond the basic ones

To handle more complex facial expressions beyond the basic ones, the proposed approach could be extended by incorporating a more extensive set of action units associated with a wider range of expressions. This would involve creating a more comprehensive codebook that includes action units for nuanced or subtle expressions. Additionally, the model could be trained on a more diverse dataset that includes a broader spectrum of facial expressions, allowing it to learn and recognize complex expressions more effectively. By expanding the action units and training data, the model can develop a deeper understanding of facial expressions and improve its ability to interpret and classify complex emotions.

What are the potential limitations of relying solely on spatial action unit cues for interpretability, and how could other types of cues be incorporated

One potential limitation of relying solely on spatial action unit cues for interpretability is that it may not capture all the nuances and intricacies of facial expressions. Action units provide valuable information about specific muscle movements, but other cues such as contextual information, temporal dynamics, or audio cues could enhance interpretability further. Incorporating these additional cues could provide a more holistic understanding of facial expressions and improve the model's interpretability. For example, integrating temporal information from video sequences or audio analysis of speech patterns could offer valuable insights into the emotional state of an individual. By combining multiple types of cues, the interpretability of the model could be enhanced, leading to more accurate and nuanced facial expression recognition.

Could the interpretability insights gained from this work be applied to improve the interpretability of other computer vision tasks beyond facial expression recognition

The interpretability insights gained from this work could be applied to improve the interpretability of other computer vision tasks beyond facial expression recognition. For instance, in object recognition tasks, the model could be trained to focus on specific object features or regions of interest to provide more transparent and interpretable predictions. By aligning the model's attention with relevant visual cues, such as object parts or textures, the decision-making process of the model can be better understood. This approach could also be extended to tasks like image segmentation, where the model's attention could be guided to highlight important regions for segmentation. Overall, the principles of incorporating interpretable cues and aligning model attention with relevant visual features can be generalized to various computer vision tasks to enhance interpretability and transparency.