Contrastive models trained under cosine similarity losses exhibit nearly constant NTK, while dot product similarity losses lead to drastic changes in NTK.
Contrastive models' relation to NTK and PCA analyzed.