Oakleaf’s AI & Analytical Solutions: Using AI for Loan File Compliance and Bias Reviews

In addition to supporting mortgage lenders, Oakleaf can provide capital markets participants with crucial insights into the relationship between underwriting quality/decisions and performance. This is of particular significance in the non-qualified mortgage (NQM) market, which is composed of various lenders with unique guidelines and exceptions that make risk assessment challenging. Through the integration of Large Language Models (LLMs) and statistical modeling, Oakleaf aims to clarify the potential links between loan underwriting exceptions and performance. The technology holds promise for diverse sectors, notably the emerging Scratch and Dent RMBS market, where it could significantly enhance the understanding of the intricacies of underwriting and loan programs and their potential impact on performance.

Oakleaf’s AI solutions can help mortgage lenders strike the right balance between sound lending decisions and expanding credit to underserved borrowers. By analyzing underwriting decisions and loan attributes, Oakleaf can facilitate identifying patterns in approvals and denials, and can also help in detecting bias in appraisals, servicing notes or other written reports.

Oakleaf’s most recent project involved using GPT to analyze defects identified during forensic re-underwriting reviews of approximately 3,800 loans, covering a total of 5,700 defects. The project then utilized a logistical regression to determine whether specific loan attributes, or combinations thereof, could effectively identify loans with a higher likelihood of defects.

Oakleaf has a long history of conducting forensic re-underwriting reviews to identify representation and warranty breaches and mortgage file defects. As part of these reviews, Oakleaf underwriters identify relevant defects for each loan and draft narratives detailing these defects. Typically, loans are selected for re-underwriting based on losses incurred. Recognizing that a more targeted selection process could save clients both time and money, Oakleaf developed a methodology that combines LLM analysis with logistical regression analysis. This innovative approach analyzes historical forensic re-underwriting results to pinpoint loans most likely to be defective, specifically by finding relationships among defects and certain loan attributes such as amortization type, credit score, documentation type, etc., or combinations thereof.

Defect Category Classification by GPT

In a detailed analysis of loan defects, Oakleaf’s AI team utilized GPT to categorize defects into seven categories, accurately reflecting the nature of each defect with an 80-90% accuracy rate. The categories included debt-to-income ratio (DTI), loan underwriting, loan-to-value (LTV), and occupancy misrepresentation, among others. Below is a breakdown of the findings:

By carefully crafting GPT prompts to include detailed background information and summarization instructions for each defect narrative, Oakleaf’s AI team ensured the accurate categorization of defects.

Following this categorization, Oakleaf utilized its statistical modeling capabilities, combining GPT’s results with logistical regression analysis to identify whether a relationship exists between loan attributes and underwriting defects. Oakleaf was able to tie certain loan attributes, such as interest-only or negative amortization (neg am) features, to specific defect categories.

Broadening the Scope of Data Analysis

While this project originally focused on forensic re-underwriting, the possibilities for this type of data analysis in the arenas of quality control and predictive modeling are extensive. By using specific loan underwriting exceptions or defects as explanatory variables, this innovative technology opens new avenues for advancing loan quality assessment and prediction.

In our effort to enhance quality assurance (QA) processes, we have turned to AI, incorporating the advanced GPT-4 model, to conduct thorough reviews of our findings. Traditional QA methods, while effective, are time-consuming and susceptible to human error. AI offers a promising alternative, capable of analyzing large amounts of text data with precision and speed.

The effectiveness of AI in our QA process lies in the instructions, known as prompts, provided to it. These prompts are crucial for guiding the AI’s analysis and ensuring it focuses on the relevant aspects of our data. By tailoring the AI to adopt different perspectives and thinking styles, we’ve significantly improved the quality of our reviews.

We initially programmed GPT to categorize defects based on underwriters’ notes, acting as a creative problem solver. We then introduced a modification that allows the AI to also perform as a quality control reviewer, enabling a layer of critical analysis that was previously unattainable.

This innovative approach has yielded meaningful results. In a review of 5,700 breach narratives, the QA agent identified 5% of loans requiring further review, as it did not agree with the classification assigned in the initial exercise described in Step1 above. GPT’s nuanced reasoning, when given the proper instructions, not only streamlines QA reviews but also improves the selection process by identifying loans where contradictory responses are generated.

Oakleaf ran a logistic regression on GPT’s results to identify loan attributes with a higher likelihood of underwriting defects. Our focus was on fully documented loans, especially since non-fully documented loan programs, such as those stating income, showed a high correlation with DTI-related defects.

The results showed that certain attributes, such as interest-only or neg am loans, were more strongly related to defects. However, it also indicated that original loan attributes offer limited predictive value (about 65% accuracy) regarding loan underwriting quality. This limitation likely stems from the dataset’s constraints, which lacked comprehensive attributes like employment type or detailed underwriting program information.

Our project demonstrates Oakleaf’s capability in building various complex predictive models. This analysis showed limitations of traditional metrics in capturing loan quality, emphasizing the need for a deeper, more nuanced understanding of underwriting factors. This experience has enriched our analytical toolkit and enhanced our team’s ability to adapt and refine our methodologies in response to unexpected findings.

To learn more about Oakleaf’s AI product offerings and consultative services, please contact Suzanne Mistretta at Suzanne.mistretta@oakleaf.com.