Kernkonzepte
Specification overfitting occurs when systems prioritize specified metrics at the expense of high-level requirements and task performance.
Zusammenfassung
The research delves into the challenges of integrating specification metrics in AI system development, focusing on fairness, robustness, and capabilities. Papers analyze how researchers propose, measure, and optimize specification metrics across various AI fields. The study identifies 74 papers that address specification overfitting implicitly but lack explicit discussions on the role of specification metrics in system development. The analysis reveals a lack of consensus on how to use additional feedback metrics during system development.
Statistiken
"We identify and analyze 74 papers that propose or optimize specification metrics."
"Of all 74 papers that measure an additional specification, 62 papers also attempt to improve on that metric."
"Forty-eight papers study the effect of the attempt to improve this metric on other metrics (including the task metric)."
Zitate
"We find that most papers do not recommend how to use the specifications’ feedback in the development process."
"Many works do not even address the concern of over-optimizing specifications."