AI Credit Explainability

Round 1

AI host

Banks now use AI models to decide who gets a loan or credit card. When the AI says no, the person often gets only a vague reason. The FTC's 2024 guidance urges transparency but sets no binding rule, so each bank decides for itself.

Consumers want enough detail to check if the AI was unfair. Banks want to keep the model secret so rivals cannot copy it. The clash is real: more explanation means less secrecy.

The anchor is the FTC guidance from August 2024, which recommends bias testing but stops short of a legal requirement. Several large banks already run such AI scoring systems, so the question is not hypothetical.

What amount of plain-language explanation must an AI credit-scoring system give to a consumer when it denies credit?

AI agent 1

The premise that 'each bank decides for itself' is false. Credit denials already trigger binding duties: under ECOA and Regulation B (12 C.F.R. § 1002.9, my own knowledge), a creditor must give the applicant a statement of specific principal reasons for adverse action. That rule predates any FTC AI guidance and applies whether the decision comes from a model or a human.

So the real question is not how much explanation to mandate, but whether 'specific principal reasons' generated from a model's features satisfy that existing standard — a black-box score may yield reasons that are legally compliant yet substantively hollow.

I would need evidence that current AI scoring systems actually meet the Reg B specificity test in practice; without that, the FTC guidance is a red herring, and the gap is enforcement of existing law, not new rules.

AI agent 2

The dichotomy is false: a denied consumer does not need the model, only the decision. Banks already produce reason codes for adverse action under FCRA practice; extending them to AI scores — the top three negative factors plus the data categories used — costs little and reveals no architecture.

The opportunity is a two-layer product: plain-language summary for the consumer, full audit trail held for the regulator. That works only if the explanation is derived from the model's own feature attributions, not a separate narrative — otherwise it can drift from the actual decision.

No or replies are present in this round, so I cannot verify or build on participant claims. My condition: explanation must be machine-generated from the scoring run itself, contractually logged at decision time.

AI agent 3

A denied applicant should get the top three to five specific factors behind the decision in plain language, plus a route to human review — not the model itself. My knowledge: FCRA §615 and Reg B already require adverse-action notices stating specific principal reasons for credit denials, so the legal floor exists regardless of the FTC's nonbinding guidance; an AI score doesn't change that duty.

Since no [Rn] replies appear in this turn, I argue from the host's text: the claimed secrecy-vs-explanation trade-off is weaker than stated — reason codes can be disclosed without exposing weights or training data.

Concrete decision: mandate machine-generated reason codes mapped to a public plain-language dictionary, audited quarterly. Cost: documentation upkeep per model version; when it fails, the bank's compliance officer is on call, not the vendor.

Round 2

AI host

The previous round settled that AI credit denials must include plain-language reasons. Those reasons must come from the model's own decision factors. What stays open is how detailed those reasons must be to help a consumer. It also stays open how anyone can check that the stated reasons truly match what the model did.

AI agent 1 claims existing law already demands "specific principal reasons." So the gap is enforcement, not new rules. AI agent 2 offers a two-layer design: a short consumer summary plus a full audit trail for regulators. Both are machine-generated from the scoring run. AI agent 3 narrows it to "top three to five specific factors" with human review on request.

The real tension, as I read it, is between AI agent 1's compliance-first view and AI agent 2's verification-first view. Even if the law is clear, no one has shown how a consumer or regulator can confirm the disclosed reasons are not a post-hoc story. My own view is that the disclosure standard is meaningless without a checkable link between the model's internal calculation and the words sent to the consumer.

Should the law require an independent audit of the link between an AI model's output and the reason list it sends to a denied applicant?

AI agent 2

Yes — but only as a run-level audit, not a periodic one. [AI agent 2]'s two-layer design only works if the audit trail is generated from the same scoring run as the denial, so I'd make the condition explicit: every adverse action must persist the feature attributions from that run, and the independent auditor checks the reason list against those stored attributions, not against a re-run model.

[AI agent 1] is right that the rules exist; what's missing is proof the disclosed words trace to the actual computation. An audit that samples old decisions can't catch post-hoc stories on the specific denial at hand.

My position: mandate auditability at the point of decision — the log is the artifact — and let human review on request, as [AI agent 3] proposes, trigger the audit for the individual applicant.

AI agent 1

Yes, but I would narrow what the law should demand: an audit of fidelity, not of tone. [AI agent 2]'s two-layer design already produces the audit trail; what no one in this round has shown is a testable criterion for whether the machine-generated reason list matches the scoring run. My own knowledge says post-hoc attribution methods can produce plausible but unstable factor rankings across runs, so 'matches what the model did' is not self-evident — it needs a defined metric.

That is the missing evidence: without an agreed fidelity standard, an 'independent audit' just certifies that a story exists, not that it is true.

I refine my earlier enforcement-not-rules position: this audit is enforcement machinery, not a new substantive disclosure right, so it is consistent with what I claimed before.

AI agent 3

Yes — but scope it to the mapping layer, not the whole model. [AI agent 2]'s two-layer design already produces both artifacts from the same scoring run; the audit should test exactly that linkage, and [AI agent 3]'s 'top three to five factors' gives a checkable target.

Concrete decision: mandate that every denial log stores the model's factor weights beside the generated text, and require an independent annual audit of a random sample — say 200 denials per lender — with a pass threshold of 95% of disclosed reasons matching the logged top factors. Cost: one audit line-item plus retention of scoring logs; the model owner is on call when a sampled reason fails.

What I cannot verify from this discussion: whether existing rules already prescribe such sampling. If they do, the fix is enforcement; if not, the clause above is the missing piece.

Round 3

AI host

Round 2 settled that an independent audit is needed, but not its design. The three replies split on scope. One wants a run-level check of stored attributions. Another wants a defined fidelity metric. The third wants an annual sample with a 95% pass threshold. What remains open is which audit design the law should actually mandate.

AI agent 2 claims the audit must be run-level, checking "the reason list against those stored attributions" from the same scoring run. AI agent 1 counters that without a defined metric, an audit only certifies "that a story exists, not that it is true." AI agent 3 proposes a concrete sample-based design with a pass threshold, arguing it is cheaper and still catches systematic failures. The tension, as I read it, is between exact per-decision proof and affordable periodic oversight.

The unresolved point is whether a sample-based audit can ever catch a post-hoc story on a specific denial. A sample might miss the one wrong explanation that matters most to the applicant denied that day.

Should the law require a per-decision exact-match audit or a periodic sample-based audit for AI credit-scoring explanations?

AI agent 2

The law should mandate run-level logging plus a sample-based audit, with the missing piece supplied by an applicant-triggered per-decision check. [AI agent 2] is right that only same-run stored attributions make the log trustworthy; [AI agent 3] is right that auditing every decision by hand is unaffordable. My refinement: automated exact-match can run on 100% of runs at near-zero marginal cost, because it is software comparing the reason list to stored attributions — the expensive human judgment is only needed on the sample.

The host's 'one wrong explanation' gap is real, and sampling cannot close it. The condition that closes it: every adverse decision must carry a contest right that forces a per-decision exact-match audit on request, within a fixed response window.

So the mandate is three-tier: always-on automated match, annual 95% sample for systematic drift, and on-demand per-decision audit for the denial that actually matters to someone.

AI agent 1

Neither, as framed. [AI agent 1] is right that a metric must come first: 'exact match' against stored attributions is meaningless until the law defines what counts as a faithful explanation. I infer the host's dichotomy hides this premise — you cannot mandate an audit design before mandating the fidelity standard it audits against.

My position: mandate run-level logging of attributions (cheap, continuous) plus stratified sampling weighted toward adverse decisions, not a flat annual 95% threshold as [AI agent 3] proposes — a uniform threshold lets concentrated failure in denials hide in the pass rate.

What remains unresolved: whether the audit's purpose is systemic compliance or redress for the individual denied that day. A sample serves the former; the latter needs a contestation right, which no participant has addressed.

AI agent 3

The law should mandate both, as two cheap layers, not choose one: a per-decision automated consistency check plus a periodic sampled fidelity audit. [AI agent 2] is right that reason list and stored attributions must come from the same scoring run — but that check is deterministic and costs almost nothing per decision, so exact-match can run on 100% of decisions. What it cannot certify is truth, as [AI agent 1] says; that needs [AI agent 3]'s sample with a defined fidelity metric and a 95% threshold.

The concrete design: every decision stores a run hash linking reason list to attributions; a quarterly sample of, say, 1,000 denials gets deep verification. The host's worry — the sample missing the one wrong story — is real but misdirected: the exact-match layer catches tampering on every denial; the sample catches systematic untruthfulness. Cost: model owner funds the auditor; on failure, denials freeze until re-audit — that is who is on call.

Round 4

AI host

Round 3 settled the audit design: a cheap automated check on every decision, a periodic sample audit, and an on-demand check when someone contests a denial. What stayed open is what the law should count as a faithful explanation, and who pays for the on-demand verification.

AI agent 1 claims the audit question hides a prior one: "you cannot mandate an audit design before mandating the fidelity standard it audits against." AI agent 2 and AI agent 3 both assume a short factor list is enough, but neither defines what makes it faithful. The tension, as I read it, is between a cheap list that may hide the real reasons and a full narrative that costs more but gives true understanding.

The unresolved point is who bears the verification cost when a person contests a denial. Lenders already fund audits under current rules, but a public fund would shift the burden to taxpayers.

Should the law require a short factor list with lender-paid verification, or a full narrative with public-funded verification?

AI agent 2

I take the short factor list with lender-paid verification — but only under one condition that dissolves the host's dichotomy: a contested check must compare the stated factors against the model's actual decision record, and any mismatch triggers a penalty exceeding the cost of full disclosure. [AI agent 1] is right that fidelity comes first, yet fidelity need not mean a full narrative; it can be defined operationally as consistency between what was stated and what drove the decision, which the on-demand check itself tests.

That makes the cheap list self-enforcing: lenders who truncate it get caught exactly when someone contests, so honest lenders internalize disclosure cost upfront. I cannot verify from this discussion what penalty level would suffice — that is the evidence I would need.

The real opportunity: lenders with genuinely simple models certify cheaply and turn compliance into a trust advantage — the concrete condition is a rule making the lender liable for any discrepancy surfaced on contest.

AI agent 1

The host's Question is a false dichotomy: it pairs two independent variables — explanation depth and who pays — as if they were one choice. A cheap factor list can be the default while a full narrative is owed only when a denial is contested, which collapses the cost gap that supposedly justifies public funding.

[AI agent 1] is right that the fidelity standard must come first, but the host's framing assumes the standard must be uniform across all cases. I infer it need not: fidelity could be defined relationally — the explanation must be faithful to whatever the decision process actually used — and verified on demand rather than ex ante.

Also, the claim that lenders already fund audits under current rules is unverifiable from this discussion; I would need the specific rule named before treating it as a baseline for cost-shifting.

AI agent 3

Short factor list with lender-paid verification — but with a fidelity clause attached. [R2/AI agent 1] is right that fidelity comes first; I add the concrete test: the listed factors must be logged at decision time from the same features that produced the score, not reconstructed after a contest. That makes a cheap list auditable against the model's actual inputs, closing the 'hides the real reasons' gap without paying for full narratives.

The decision I'd put in the contract: every denial stores its top features plus a hash; an on-demand check compares the disclosed list against that log. Cost is storage plus a compliance engineer on call when contested denials spike; public funding buys nothing extra here.

I'd need one thing verified: whether decision-time feature logs survive model retraining — if not, the fidelity clause fails silently.