The paper proposes a Generalized Influence Function (GIF) that measures the influence of data on a subset of model parameters, rather than the entire parameter set. This is motivated by the observation that influence functions can become inaccurate as model size and complexity increase, due to the accumulation of errors in irrelevant parameters.
The key aspects of the proposed approach are:
Parameter Selection Criteria: The paper introduces four criteria to identify the most relevant parameters for a given input data - Highest-k outputs, Highest-k gradients, Lowest-k outputs, and Lowest-k gradients. These criteria select a subset of parameters to focus the influence computation on.
Generalized Influence Function: The GIF formulation computes the influence only on the selected relevant parameters, while nullifying the effects of unselected parameters. This is achieved by projecting the gradient changes onto the subspace of the selected parameters.
Modified LiSSA Iteration: The paper proposes a modified version of the LiSSA algorithm, which is used to efficiently compute the GIF. This modified version guarantees convergence without the need for additional regularization.
The experiments show that the GIF outperforms existing influence function methods in data removal and label change tasks, while updating only a small fraction (e.g., 5%) of the model parameters. The updated models closely match the behavior of models retrained from scratch, as demonstrated by output distributions and visualization of discriminative regions.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Hyeonsu Lyu,... at arxiv.org 05-07-2024
https://arxiv.org/pdf/2312.05586.pdfDeeper Inquiries