Permutation Equivariance of Transformer Models and Its Applications

Transformer models exhibit permutation equivariance in both forward and backward propagation, covering both inter- and intra-token shuffling. This property can be leveraged for privacy-enhancing techniques and model authorization.