핵심 개념
The proposed approach comprehensively models multi-scale facial dynamics and hierarchical spatio-temporal relationships among facial action units to achieve state-of-the-art performance in action unit occurrence recognition.
초록
The paper proposes a novel Multi-scale Dynamic and Hierarchical Relationship (MDHR) modeling approach for facial action unit (AU) recognition. The key contributions are:
- Multi-scale Facial Dynamic Modelling (MFD) module:
- Explicitly captures facial dynamics at multiple spatial scales, considering the heterogeneity in range and magnitude of different AUs' activation.
- Adaptively combines the multi-scale facial dynamic features with static facial features.
- Hierarchical Spatio-temporal AU Relationship Modelling (HSR) module:
- Hierarchically models the relationship among AUs in a two-stage manner:
- Local AU relationship modelling: Captures the relationship among AUs within the same/close facial regions.
- Cross-regional AU relationship modelling: Learns the relationship between AUs located in different facial regions.
Experimental results on the BP4D and DISFA datasets show that the proposed MDHR approach achieves new state-of-the-art performance in AU occurrence recognition, outperforming previous static image-based and spatio-temporal methods. The MFD and HSR modules are shown to contribute complementarily to the final performance.
통계
The proposed approach achieved new state-of-the-art F1 scores of 66.6% on the BP4D dataset and 66.2% on the DISFA dataset.
인용구
"The proposed MFD is the first module that adaptively/specifically considers facial dynamic corresponding to each AU at each spatial scale, as each AUs' activation exhibit heterogeneity in both range and magnitude."
"The proposed HSR is the first module that hierarchically learns local and cross-regional spatio-temporal relationship, while previous approaches fail to consider such hierarchical relationship."