Mitigating Bias in Automated Mortgage Underwriting Models: Evaluating De-Biasing Methods Using Counterfactual Mortgage Application Data

Core Concepts

Automated mortgage underwriting models can replicate historical biases in lending decisions, even when prohibited factors like race or ethnicity are not used as predictors. Several methods to de-bias such models are evaluated, including averaging over prohibited variables, maximizing predictions over prohibited groups, and jointly optimizing accuracy and disparity.

Abstract

This paper examines the problem of bias in automated mortgage underwriting models. It demonstrates that even when prohibited factors like race or ethnicity are excluded from the model, a machine learning model can still replicate historical biases in lending decisions due to correlations between these factors and other predictive variables.
The paper then evaluates several methods for de-biasing such models:

Excluding prohibited variables: This is shown to be insufficient, as the model can find proxies to replicate the bias.

Averaging predictions over prohibited variables: This method retains the predictive power of variables correlated with prohibited factors, but may suffer from model selection bias.

Maximizing predictions over prohibited variables: This novel method approves applicants if any prohibited group would have been approved, addressing both explicit use of prohibited factors and model selection bias. However, it may not be effective when there are many prohibited variables.

Jointly optimizing accuracy and disparity: This regularization approach reduces disparities between groups but can come at the cost of overall predictive accuracy.

The performance of these methods is tested on real mortgage application data with simulated bias added. The results highlight the importance of understanding the specific form that bias takes in order to effectively mitigate it. Methods that work well for one type of bias may perform poorly for another.

Stats

"Simulated bias against Hispanic and Latino applicants results in a denial rate of 19.1% for this group, compared to 9.5% in the original data."
"The overall denial rate in the biased training data is 7.2%."

Quotes

"Even if the variables used in the model do not include race or ethnicity, the model may use other variables (such as location or property type) to better match model predictions to the biased decisions."
"Discrimination by proxy can also include the use of a predictive variable that is correlated with a protected group but of little use in predicting the outcome of interest."
"If some loan officers explicitly used prohibited basis group g in their decisions, then yi < yfair
i for some members of group g."

Key Insights Distilled From

De-Biasing Models of Biased Decisions: A Comparison of Methods Using Mortgage Application Data

by Nicholas Ten... at arxiv.org 05-03-2024

https://arxiv.org/pdf/2405.00910.pdf

Deeper Inquiries

How might the performance of these de-biasing methods change if the underlying bias in the data was due to explicit use of prohibited factors rather than proxy discrimination?

In the context of explicit use of prohibited factors, where bias is directly linked to protected group information being used as predictive variables, the de-biasing methods may face different challenges and outcomes.

Exclusion Method: If the bias is explicitly tied to prohibited factors, simply excluding these factors from the model may not be sufficient to remove the bias. The model may still find proxies for the prohibited factors and replicate the bias through other variables.

FairXGBoost: Regularization methods like FairXGBoost, which optimize for accuracy and disparity between groups, may be more effective in mitigating bias when the bias is explicitly tied to prohibited factors. By penalizing the association between the prohibited factors and outcomes, FairXGBoost can help reduce bias in the model predictions.

Averaging over Prohibited Variables: Averaging over prohibited variables may still be effective in reducing bias if the prohibited factors are explicitly used in the bias. By considering the average prediction over different values of the prohibited factors, this method can help mitigate bias in the model predictions.

Maximum Prediction over Prohibited Variable: The max-over-groups method may also be useful in addressing bias from explicit use of prohibited factors. By taking the most favorable prediction over different values of the prohibited factors, this method can help ensure that applicants are treated fairly regardless of their protected group status.

How could these de-biasing methods be extended or adapted to address bias in other types of automated decision-making systems beyond mortgage underwriting?

These de-biasing methods can be adapted and extended to address bias in various automated decision-making systems beyond mortgage underwriting. Here are some ways they could be applied:

Employment Decisions: In the context of hiring or promotion decisions, these de-biasing methods can be used to ensure that decisions are not influenced by protected characteristics such as race, gender, or ethnicity. By excluding or penalizing the use of these factors in the decision-making process, the models can be made more fair and equitable.

Healthcare: De-biasing methods can be applied to healthcare systems to ensure that medical decisions, such as treatment recommendations or resource allocation, are not influenced by factors like race or socioeconomic status. By removing bias from predictive models, healthcare systems can provide more equitable care to all patients.

Criminal Justice: In the criminal justice system, these methods can help mitigate bias in decisions related to bail, sentencing, or parole. By ensuring that protected characteristics do not influence the outcomes of these decisions, the system can strive for more fairness and justice.
By applying these de-biasing methods to a wide range of automated decision-making systems, we can work towards creating more equitable and unbiased processes across various domains.