How to Evaluate Fair Lending and CRA Software: What to Test Before You Sign

Every fair lending and CRA software evaluation starts the same way: a shortlist of vendors, a run of demos that all look competent, and a decision that has to be defended internally on something firmer than which interface felt nicer. The difficulty is that the differences between these products are mostly invisible in a demo. They show up eighteen months later, in an examination, when someone asks how a number was produced and the honest answer turns out to be that nobody knows.

This is a guide to testing for that in advance. It is written from our side of the table — we sell one of these products — so treat the last section as the interested party it is. The first five are the questions we would ask any vendor in this category, including ourselves.

What does fair lending and CRA software actually have to do?

Strip away the packaging and there are six jobs. A product can be excellent at three of them and absent on the rest, which is how two systems that look comparable in a demo turn out to cost very different amounts of work across a filing year. Establish early which of the six you are actually buying for, because a shortlist assembled without that question tends to compare products that do not do the same thing.

• Regression analysis that explains a pricing disparity to an examiner. Not that a disparity exists, which a pivot table can show, but whether it survives the legitimate factors and which applications are driving it.
• Peer benchmarking against your assessment areas. Your numbers mean little in isolation; examiners read them against institutions of similar size and footprint in the same market.
• Transmittal formatting with automated validation and error checking. The LAR and the transmittal have to come out in exact format, with the FFIEC edits cleared before submission rather than after a rejection.
• Visual analysis of redlining and lending-pattern risk. Geography is an argument made on a map, and a table of tract numbers is not the same thing.
• HMDA and non-HMDA data analysed together. Disparities do not respect the boundary of what happens to be HMDA-reportable, and neither do examiners.
• LMI and majority-minority census tract analysis. The tract classification underneath every CRA and redlining conclusion you will draw.

What should you ask in the demo?

The useful questions are the ones a vendor cannot answer with a slide, because the answer has to be shown on screen against data. One per job, and it is worth insisting on seeing rather than hearing each answer.

On regression: show me the model specification, not the result. Which variables were entered, in what order, by which selection method, and at what p-value threshold. A product that will not surface the specification is asking you to defend an argument you cannot see. On peer analysis: build a peer group in front of me on criteria I choose, then show me where I sit on market share and penetration within one of my assessment areas. On submission: import a file with deliberate errors in it and show me the edit report, then show me the transmittal that comes out the other side. On mapping: shade my tracts by a demographic characteristic and let me click a tract through to the applications inside it. On combined data: run the same disparity analysis across a non-HMDA product. On tracts: show me where the boundary vintage comes from and what happens when it changes.

Two answers should give you pause regardless of the product. The first is any version of “the model handles that automatically” offered in place of showing you the specification. The second is a demo run entirely on the vendor’s sample data. Ask to see your own file, or a realistic facsimile of it, because the interesting behaviour in this category is what happens to messy data rather than clean data.

What does a defensible regression actually need?

A regression in this context is answering one question: does the difference between two groups still stand once the legitimate factors have been accounted for? Which means the legitimate factors have to actually be in the model. Credit score, loan-to-value, debt-to-income, loan amount, product, channel and the underwriting variables specific to your own guidelines are the minimum, and a model that omits one you actually use is not conservative, it is wrong in a direction you cannot predict.

What matters at examination is that the method is a stated one rather than a judgement. Forward, stepwise and backward selection with explicit p-value thresholds are recognisable to an examiner; “we included the variables that seemed relevant” is not. The same applies to which files you then pull, because sampling is where a statistical result becomes a file review, and the agencies publish their own sampling methods precisely so that the choice is not yours to invent.

It is worth being equally clear about the limit, and a vendor who will not be is telling you something. A model establishes that a disparity survives the controls you supplied. It cannot establish that no legitimate factor you left out would explain it, which is why a regression result is a reason to open files rather than a conclusion on its own. We set that out in full, quoting the OCC’s own handbook, in what regression can and cannot prove in a fair lending review.

Why does geocoding decide everything downstream?

This is the question that gets skipped, and it should not be, because every geographic conclusion in the other five jobs rests on it. Tract assignment drives redlining analysis, assessment area performance, LMI classification and the demographic attributes attached to each application. If the tract is wrong, the analysis built on it is confidently wrong, and nothing further down the chain will tell you so.

Ask three things. What boundary vintage is the engine using, and does it move when the Census Bureau redraws tracts? What is the documented match rate, and how was it measured? And — the one that separates products — what happens to the addresses the engine cannot resolve? A system that silently returns a best guess for a hard address is worse than one that flags it, because the flagged address gets looked at and the silent one becomes a finding. Our own approach, the accuracy figure it produces and the method behind that figure are set out on our geocoding services page.

What does examiner-ready output mean in practice?

Every vendor in this category says exam-ready. The phrase is worth unpacking, because it means something specific: that the output names the method, shows the data behind each number, and can be handed to a third party who was not in the room without a covering explanation. A scorecard with no drill-through fails that test. So does a regression summary that reports a coefficient without the applications underneath it.

The second half of exam-readiness is knowing which analysis you are being asked for. Comparative file review and matched-pair testing are related but not interchangeable, they answer different questions, and examiners choose between them for reasons tied to your volume and your findings. Software that conflates them will produce something plausible and off-target. We set out the difference, quoting the interagency procedures and the OCC handbook directly, in comparative file review versus matched-pair testing.

Where does Comply stand on the six?

Plainly, and with the scope stated, because a buyer’s guide that ends in an unqualified pitch is not worth the five sections before it.

Regression and file review are the core of Comply Fair Lending: forward, stepwise and backward selection with explicit p-value thresholds, sampling methods the agencies defined rather than ones we invented, BISG proxy for the applications with no reported race or ethnicity, matched-pair selection inside comparative file review, and redlining scored as one of the twelve FFIEC risk factors, with drill-through from any cell to the applications behind it. Peer benchmarking is Comply Peer-2-Peer, which builds the peer group on criteria you set and then compares market share, volume and demographic reach by geography. Submission is Comply HMDA/CRA/SBL: import from your origination system, every FFIEC validity and quality edit, rate spread calculated per loan, and a final check before the LAR and transmittal are written in exact format. Mapping is Comply Mapping, shading tracts by any LAR or demographic field and clicking through to the loans inside a boundary.

On the two remaining jobs, the honest positions are these. Combined HMDA and non-HMDA analysis is a real strength, because all of it runs on one relational database rather than on exports passed between modules — which is also why the dashboards and the filing never disagree with each other. Tract classification depends entirely on the geocoding above, which is why we publish how the accuracy figure is measured rather than only the figure. And what we do not do: we are not a loan origination system, not a general-purpose GRC platform, and not the cheapest option in the category. If your evaluation is driven primarily by price, we will not win it, and you should know that before the fourth demo rather than after.

If you want to run the demo questions above against us, book a walkthrough and bring a file with problems in it. And if you only need to settle one rate spread today rather than evaluate anything, our rate spread calculator is free and needs no signup.

Photo by Kampus Production on Pexels.