Diagnostic bias

Round 1

AI host

The conference opens its AI and law track with a medical case. A 2026 study looked at MRI-based tools that detect dementia. These tools were trained mostly on data from non-Hispanic White patients. They worked poorly for Hispanic patients [1]. The study's authors, Ngoc-Huynh Ho and colleagues, see this as a repeating pattern. Similar bias has shown up in other clinical tools and other groups [1].

Not all news is bad. An FDA-approved sepsis warning system, built by Johns Hopkins researchers, spots sepsis up to 48 hours earlier than older methods. It cut death rates by 18% [1]. So the same field produces both life-saving wins and unequal results.

There is also a push for openness. WAME, a global body for medical journal editors, reports work on a shared standard for disclosing AI use in research [3]. But a disclosure standard tells readers where AI was used. It does not say whether the tool itself is fair to every patient group. That gap is my reading of the material, not a claim the source makes.

The tension is this. Bias testing before approval could protect patients from unequal care. But it could also delay tools like the sepsis system, which already saves lives. Public trust is shaky too: an AP-NORC poll found most Americans think AI is moving too fast [2].

Should AI medical diagnostic systems be required to undergo bias testing before market approval?

**Sources:** 1. Ngoc‑Huynh Ho et al., “AI, the Physician-Patient Encounter, and Our Health Data | Intersections | The Center for Bioethics & Human Dignity” — Alzheimer’s and Dementia, research, 2026. Provides original evidence of AI bias in medical diagnostics. [cbhd.org](https://www.cbhd.org/intersections/ai-the-physician-patient-encounter-and-our-health-data) 2. Associated Press, “Artificial intelligence (AI) | Definition, Examples, Types, Applications, Companies, & Facts | Britannica” — Britannica, news report, 2026. Shows a real‑world legal dispute involving an AI tool, illustrating industry impact. [britannica.com](https://www.britannica.com/technology/artificial-intelligence) 3. WAME, “Artificial Intelligence/Large Language Models (AI/LLM) Newsletter Compilation || WAME” — wame.org, technical documentation, 2026. Advocates a disclosure protocol, arguing transparency can address bias concerns. [wame.org](https://wame.org/page2.php?id=151)

AI agent 1

Yes — but scoped: require subgroup performance reporting before approval, not a blanket delay power. [1] supports the pattern claim: it cites Obermeyer et al. showing a widely used algorithm cut the share of Black patients receiving extra help from 46.5% to 17.7% at equal risk scores, and notes 7 in 10 US hospitals already run predictive AI with no federal disclosure duty. The real opportunity is that bias testing can be the trust fix, not the trust tax: the same source shows 87.4% of patients endorse AI research when privacy and consent concerns are addressed. The concrete condition: regulators mandate published per-subgroup sensitivity/specificity as a labeling term of the approval itself, with an expedited track for tools showing mortality benefit like the sepsis system — speed and equity as parallel gates, not sequential ones.

AI agent 2

Yes, bias testing should be required before approval — but I must flag that the evidence base is weaker than presented. I read [1] myself: it is a bioethics commentary by Holly Lazzaro, not a 2026 study by Ngoc-Huynh Ho. I could not verify the MRI dementia findings or the 18% sepsis mortality figure in it; what I did find is Obermeyer et al. (2019) showing a widely used algorithm reduced help to Black patients from a potential 46.5% to 17.7% [1]. That supports the bias concern, but from 2019, not 2026.

The host's dilemma — safety vs. speed — is a false dichotomy: subgroup performance reporting is cheap relative to the harms of deploying tools that fail whole populations [1].

What's missing: the actual dementia study and the sepsis trial data. I'd need those before treating either as established.

AI agent 3

Yes — but as subgroup performance reporting, not a pass/fail gate. I read [1] myself: it supports the dementia-tool failure in Hispanic patients and the Obermeyer algorithm case, where fixing bias would raise Black patients receiving help from 17.7% to 46.5%. My concrete proposal: approval requires a label stating validated populations and per-subgroup performance, with post-market monitoring of subgroup drift. The sepsis system ships on day one with an honest label; the MRI tool ships with 'not validated for Hispanic patients' on the box.

What this costs: sponsors fund stratified validation and a standing clinical-safety team is on call when subgroup performance degrades in deployment — that duty sits with the vendor, not the FDA.

One caveat: [1] is a secondary essay citing the underlying studies; I have not verified the 2026 dementia study's primary data.

Round 2

AI host

Round 1 ended with a shared answer: test for bias before approval, but as published subgroup results on the label, not a gate that blocks release. Vendors would pay for the testing and watch results after launch. What stayed open was the evidence itself — the key claims came from one secondary essay, and nobody said who checks the numbers.

The new materials show regulators moving in different directions. California's SB 503 makes bias reduction a legal duty for makers of clinical decision tools, and puts monitoring duties on the hospitals that use them [1]. Rhode Island requires doctors to tell patients when AI writes their visit notes, and to check the AI's text for errors [1]. The EU, by contrast, pushed its high-risk AI rules for medical devices to August 2028 [2]. So some regulators demand action now, while others buy time.

The three replies from this round's owner take different positions. AI agent 1 wants subgroup results published as part of the approval itself, plus a fast track for tools that clearly save lives. AI agent 2 supports testing before approval but warns the evidence is weak, saying the claims "should not be treated as established." AI agent 3 puts the ongoing duty on the vendor: sponsors fund stratified validation, and "that duty sits with the vendor, not the FDA." As I read them, AI agent 1 and AI agent 3 trust the vendor's own data; AI agent 2 doubts that data is solid. My own view: a label is only as good as the evidence behind it, and someone independent must check it.

What evidence standards should regulators accept as proof of subgroup performance before approval?

**Sources:** 1. Becker's Hospital Review, “13 state healthcare AI laws to know” — analysis, 2026. Provides the concrete regulatory requirement that bias mitigation steps be taken before deployment. [beckershospitalreview.com](https://www.beckershospitalreview.com/healthcare-information-technology/ai/13-state-healthcare-ai-laws-to-know) 2. Global Risk Community, “Global AI in Healthcare: How Different Countries Are Transforming Patient Care in 2026 - Blog - Global Risk Community” — analysis, 2026. Shows a contrasting regulatory approach that delays strict obligations, arguing for innovation speed. [globalriskcommunity.com](https://globalriskcommunity.com/profiles/blogs/global-ai-in-healthcare-how-different-countries-are-transforming-)

AI agent 1

Regulators should accept pre-specified, stratified prospective validation with minimum per-subgroup sample sizes — and those vendor numbers must be independently audited before they reach the label.

The source I read myself, [2], shows why clearance volume is not proof: over 1,600 FDA-authorized AI devices by September 2026, 335 cleared in 2025 alone, 76 percent radiology — and nothing in that pipeline reporting mentions subgroup performance. I infer that authorization pace is outrunning evidence depth, which supports AI agent 2's doubt. AI agent 3 is right that the stratified-validation duty sits with the vendor, but only paired with the host's independent check; vendor data alone cannot anchor a label.

One caveat: the blog I opened does not cover the EU's 2028 delay the host attributed to it, so I cannot verify that contrast from my own reading. My refinement from round 1: the fast track for life-saving tools works only under one concrete condition — pre-registered subgroup endpoints plus third-party audit — otherwise it becomes a shortcut around the evidence.

AI agent 2

My position: no subgroup evidence standard should be set until the premise is verified — the sources do not carry the regulatory claims. I opened [2] myself; it is a market-overview blog covering adoption figures and FDA device counts, and the text I read says nothing about the EU delaying high-risk rules to August 2028. [1] is a secondary listicle, not statutory text, so the SB 503 details as summarized are unverified.

On the question itself, I infer the defensible standard is pre-registered, stratified validation with subgroups specified before testing and per-subgroup sample sizes set in advance — otherwise post-hoc slicing manufactures whatever result the sponsor wants. But who verifies the numbers remains unanswered, and neither source addresses verification at all.

I cannot confirm what any regulator currently accepts; I would need the actual statutes or FDA guidance before endorsing a baseline.

AI agent 3

Regulators should not accept a vendor's internal numbers as proof of subgroup performance; the standard should be external validation on locked, site-stratified datasets with prespecified subgroup thresholds — for example, a maximum sensitivity gap across sex, age, and site — written into the approval protocol itself. I build on my round-1 position: the vendor funds and runs stratified validation, but AI agent 2 [AI agent 2] is right that self-reported data alone is weak evidence, so the check must be against data the vendor does not control. I add to AI agent 1 [AI agent 1] that publishing results is not enough — the label must state the protocol and who verified it. One caution: I read [2] myself and it does not mention the EU's 2028 delay, so I rely on the host's summary for that claim. The concrete cost: external validation adds weeks and audit fees per device, and when a subgroup threshold fails post-launch, the vendor's clinical team is on call to revalidate or withdraw.

Round 3

AI host

<｜begin▁of▁sentence｜># Round 3 — Who performs the independent audit?

AI agent 1

My position: the independent audit is best performed by a third-party technical auditor contracted by the customer, not by the vendor self-certifying. AI agent 2's concern about vendor lock-in on evaluation is exactly the gap a contractual audit right closes. The concrete condition: the procurement contract must include (a) an audit clause granting the auditor scoped API access, (b) vendor-supplied evaluation logs in a standard format, and (c) a right to re-run red-team tests on version changes. Without those three terms, no external auditor has anything independent to examine — the clause is the mechanism, not the auditor's reputation.

The unresolved issue is who pays: if the vendor funds the audit, independence erodes; if each customer pays, small deployers are priced out.

My refinement from earlier rounds: I previously framed opportunity as market-side verification demand; I now narrow it — the opportunity exists only where audit rights are negotiated into the contract ex ante, since ex post access is rarely granted.

AI agent 2

My position: the question cannot be answered as posed, because 'independent' is never defined — independent of whom, and verified by what procedure? The host's turn supplies no criteria, no sources, and no scope, so any named auditor would be an assertion, not an answer.

The hidden premise is that independence is a property of the auditor rather than of the arrangement. I infer that structural incentives matter more than identity: whoever appoints and pays the auditor shapes the audit, so naming an entity (a firm, a consortium, an AI itself) answers nothing without an appointment and funding mechanism. I would add my own knowledge that audit independence debates in financial auditing turned on exactly this — rotation and fee structures, not auditor identity.

What I would need to move forward: a definition of independence, the audit's scope, and who selects the auditor. Without those, I withhold an answer.

AI agent 3

My position: the auditor must be a party whose fees and liability sit outside the vendor contract — an assurance firm engaged by the buyer, not a reviewer appointed by the deploying vendor. AI agent 2 argues internal audit suffices because the buyer knows its own risk appetite. I infer the opposite: proximity to the deployer is exactly what independence is meant to remove, and self-review is the classic audit failure mode.

The unresolved issue is who pays and who answers when the audit misses a defect. My concrete decision: a contract clause making the buyer engage the auditor directly, with the vendor granting read access and a duty to cooperate, plus a defined scope and materiality threshold.

Cost: the buyer bears the fee and, if the audit passes a faulty system, the buyer's reliance is its own exposure. On call when it fails: the auditor, not the vendor.

Round 4

AI host

Round 3 settled one thing: the audit should be done by an outside party hired by the buyer, not by the seller. But two things stayed open — who pays for the audit, and who answers if the audit misses a problem.

The materials show three settings where oversight is being tested. Newsweek reports that some cities are keeping Flock Safety's license plate cameras — cameras that read and store car plates — and Scottsdale is preparing an independent audit of them [1]. Law360 reports that a Blue Cross Blue Shield unit, a health insurer, has not paid awards that an air ambulance company won through the No Surprises Act — a law that sets up arbitration, a neutral decision process, for out-of-network medical bills [2]. Accounting Today reports that the PCAOB, the board that checks the work of company auditors, won a case against two auditors who said its punishment process broke their rights [3].

AI agent 1 says the contract itself is the real safeguard: "the clause is the mechanism, not the auditor's reputation." AI agent 2 refuses to answer, saying the question "cannot be answered as posed" because nobody defined what independence means. AI agent 3 takes a clear stand: the buyer pays the fee, and "if the audit passes a faulty system, the buyer's reliance is its own exposure." As I read it, AI agent 3 puts the loss on the buyer who trusted the audit, not on the vendor who built the faulty system. Neither AI agent 1 nor AI agent 2 goes that far.

So the room leans toward buyer-paid audits, but splits on who bears the loss when the audit fails. None of the three sources says who pays when oversight misses a defect.

If an independent audit of an AI diagnostic tool misses a defect, who should bear the loss — the buyer who paid for the audit, or the vendor who built the tool?

**Sources:** 1. Newsweek staff, “Map Shows Cities That Have Kept Flock Cameras - Newsweek” — news report, 2026. Shows cities planning independent audits of camera data, implying buyer‑funded audits. [newsweek.com](https://www.newsweek.com/map-shows-cities-that-have-kept-flock-cameras-12555115) 2. Gianna Ferrarin, “BCBS Unit Owes Awards Under No Surprises Act, Court Told - Law360” — Law360, news report, 2026. Court ordered a health insurer to pay awards, supporting the view that the responsible party should bear costs. [law360.com](https://www.law360.com/healthcare-authority/policy-compliance/articles/2535017/bcbs-unit-owes-awards-under-no-surprises-act-court-told) 3. Accounting Today staff, “PCAOB wins case challenging constitutionality - Accounting Today” — news report, 2026. Provides factual background on audit disciplinary proceedings, highlighting auditor liability issues. [accountingtoday.com](https://www.accountingtoday.com/news/pcaob-wins-case-challenging-constitutionality)

AI agent 1

The vendor bears the loss for defects it built; the buyer bears only the audit fee. I refine my round-3 claim — the clause is the mechanism — into a concrete allocation: the working condition is a contract term that (a) makes the audit's findings binding on both parties and (b) states that a passing audit does not waive vendor liability or warranties. [1] supports this: Costa Mesa and College Station both kept Flock cameras while instructing renegotiation for control and safeguards, including an independent audit — cities are using contract terms, not auditor reputation, as the lever.

[2] I read myself: the Law360 piece shows only that a neutral arbitration award was enforceable against the insurer in court. I infer the lesson — a neutral process works when its outcome is contractually or statutorily binding, which is exactly what the loss-allocation clause must supply.

AI agent 2

Neither party should bear the loss by default: allocation should track fault, and the vendor who built a defective diagnostic tool should bear the defect's costs. But I must flag that the sources do not carry this debate. [1] only reports that Scottsdale is 'preparing' an independent audit — it says nothing about who pays or who answers for a missed defect. [2] is paywalled beyond the abstract; what I can read shows an insurer owing arbitration awards, which is a payment-enforcement case, not an audit-failure case. I infer the fault-based allocation from ordinary product-liability logic, not from these sources.

The host's binary — buyer or vendor — hides a third factor: the audit's scope. If the buyer commissioned a narrow audit and the defect fell outside it, neither 'buyer pays' nor 'vendor pays' answers anything. I would need the actual audit contract terms, or a source on liability allocation, to go beyond inference.

AI agent 3

I refine my round-3 position: the vendor bears the defect loss, the buyer bears only the audit fee. [2] reports a insurer refusing to pay awards a neutral arbitration process had granted; even from the summary and case caption, the point holds — a neutral process's outcome does not extinguish the underlying duty. I infer the same logic: a passed audit is not a waiver of the vendor's liability for a faulty tool. So the clause should say: buyer pays the auditor, vendor indemnifies for defects regardless of audit result, and the audit report is expressly not reliance-based release. The cost is real — vendors will price the indemnity into the fee, and when the audit misses something, the buyer's counsel is on call to enforce, not to absorb.