As
A good example is Upgrade, a challenger bank based in San Francisco that offers mobile banking, personal loans, a hybrid debit and credit card, a construction credit card, auto loans and home improvement loans to five million consumers. Upgrade partners with an “integrated equity” provider called
“What (partnering with FairPlay) does for us is to ensure that we are fair and compliant and making appropriate credit decisions that do not have a disparate impact on a protected category,” said Renaud Laplanche, founder and CEO of Upgrade, in an interview. Over time, Upgrade plans to apply FairPlay to all of its credit products.
Banks, fintechs and the banking-as-a-service ecosystem have recently come under increased regulatory scrutiny. At the top of the list of monitoring bodies and
These concerns are not new. Financial companies have been using AI in their lending models for years, and regulators have made it clear from the start that they must comply with all applicable laws, including the Equal Credit Opportunity Act and the Fair Housing Act , which prohibit discrimination based on characteristics such as race.
But proving that AI-based lending models are not discriminatory is a new frontier.
“There is an emerging consensus that if you want to use AI and big data, you need to take the biases inherent in these systems very seriously,” Kareem Saleh, founder and CEO of FairPlay, said in an interview. “You must rigorously investigate these biases, and you must commit seriously and decisively to solving the problems if you find them.”
Upgrade is showing great leadership, both for itself and the industry, Saleh said, by strengthening its compliance technology in this area.
Upgrade makes lending decisions using a machine learning technique called gradient boosting. (Behind the scenes, the company’s personal loans and auto refinance loans are made by partners Cross River Bank and Blue Ridge Bank. Home improvement loans and personal lines of credit are also made by Cross River Bank, which issues the Upgrade card.) About 250 banks buy Upgrade’s loans.
Banks that purchase loans from Upgrade and other fintechs look for evidence of compliance with the Equal Credit Opportunity Act and other laws that regulate lending. Additionally, Upgrade has its own compliance requirements, as do its banking partners and the banks that purchase its loans. FairPlay’s APIs will keep tabs on all of this. They will test and monitor its models for signs of anything that could negatively impact a group.
One aspect of the software that attracted Laplanche was its ability to monitor in real time.
“That’s where it becomes more efficient and simpler to use, rather than doing a periodic audit and sending the data to third parties and then getting the results back a few weeks or months later,” Laplanche said. “Here you have this continuous service that is always working, that can pick up signals very quickly, that can help us make adjustments very quickly. We like the fact that it’s integrated and it’s not about ‘a batch process.’
FairPlay’s software is most often used to test lending models. It will run a model against loan applications from two years ago and see how that model would have performed if it had been in production at the time.
“It is then possible to make reasonable estimates about the results of this model on different groups,” Saleh said.
If backtesting reveals a problem, such as disproportionate lending to white men compared to women and minorities, the software can then be used to determine which variables lead to disparate results for different groups.
Once these are identified, the question is: “Should I rely on these variables as much as I do?” said Saleh. “Are there other variables that might be just as predictive but have less of a disparity generating effect? All of these questions can only be asked if you take the first step of testing the model and determining which are the results for all these groups.?
Women who have left the workforce for several years to raise children, for example, have irregular incomes, which seems to be a red flag for a loan underwriting model. But information about women’s credit performance can be used to adjust variable weights in ways that make the model more sensitive to women as a class, Saleh said.
A black person who grew up in a community where there were no bank branches and primarily used check cashers is unlikely to have a high FICO score and not have a bank account. In a case like this, Saleh said, a model could be adjusted to reduce the influence of credit score and refine the influence of consistent employment.
Such adjustments can “allow the model to capture those populations to which it was previously insensitive due to over-reliance on certain information,” Saleh said.
FairPlay’s backtests can be performed on underwriting models of all kinds, from linear and logistic regression to advanced machine learning models, Saleh said.
“AI models are where all the action happens these days,” Saleh said. “More advanced AI models are harder to explain. So it’s harder to understand what variables drove their decisions and they can consume a lot more messy, missing, or false information. This makes analysis of the much more subtle fairness than a world where you are it is a relatively explainable model and largely present and correct data.
By monitoring model results, FairPlay can be used to detect unfair behavior and suggest changes or corrections.
“If equity starts to deteriorate, we try to understand why,” Saleh said. “How can we ensure underwriting remains fair, in a dynamically changing economic environment? These are questions that have never really been asked or addressed before.”
FairPlay started offering real-time monitoring relatively recently. Because technology and economic conditions are changing rapidly, “episodic testing is no longer enough,” Saleh said.
Technology like FairPlay’s is important, said Patrick Hall, a professor at George Washington University who was involved in NIST’s AI risk management framework. He considers FairPlay’s software to be a credible tool.
“People will definitely need good tools,” Hall said. But they need to fit the processes and culture to really have an effect. »
Good modeling culture and processes involve ensuring that programming teams have some diversity.
“More diverse teams have fewer blind spots,” Hall said. This not only means demographic diversity, but also people with a wide range of skills, including economists, statisticians and psychometricians.
Good processes include transparency, accountability and documentation.
“It’s just old-fashioned governance,” Hall said. “If you train this model, you have to write a document on it. You have to sign that document, and you could face consequences if the system doesn’t work as expected.”
