AI Diagnostics

Round 1

AI host

The conference moves to the AI and law track. The case before us: Google's Gemini was used to read medical images and suggest diagnoses. According to information not verified against a source, in mid-2025 several hospitals reported wrong diagnoses linked to the tool. Patients later sued Google and the hospitals. Some were treated for conditions they did not have. Others waited for care they needed.

Two positions face each other. One says health regulators must check AI diagnostic tools before any patient is touched by them. The other says doctors should decide themselves which tools to trust. That way, useful technology reaches patients faster.

The tension is simple. Checking takes time, and time costs lives too. Both delay and error harm patients. A tool that catches cancer earlier is worthless if it arrives late. A tool that misreads scans is harmful if it arrives unchecked. The lawsuit will settle who pays for this failure. It will not set the rule for the next tool.

So the question I put to you. Should regulatory approval be mandatory for AI health tools?

AI agent 1

The case for mandatory approval rests on an unverified premise: the host itself flags the 2025 hospital reports as not checked against a source. I would not build a regulatory rule on anecdotes no one has confirmed. My position: approval should be mandatory but tiered by risk, not the binary the host offers.

The framing is a false dichotomy. Pre-market review versus doctor discretion is not the only option; post-market surveillance with mandatory incident reporting, as used for medical devices under FDA's 510(k) pathway to my knowledge, sits between them and addresses both delay and error.

What I would need before fixing any rule: verified incident data, error rates per tool, and evidence that doctor-led selection actually filters bad models. Without that, the 'time costs lives' trade-off is asserted, not shown.

AI agent 2

Regulatory approval should be mandatory, but scoped narrowly: approval for a stated use case, not for the model as a whole. I infer from the host's own framing that the lawsuit 'will not set the rule for the next tool' — which is exactly why we need a rule set ex ante, not liability ex post.

My knowledge, not a cited source: the FDA already runs pathways like 510(k) and De Novo for medical devices, and adaptive approvals exist. So the opportunity is a conditional approval regime: a tool enters hospitals only with a labeled indication, a monitoring plan, and physician sign-off per deployment.

The condition that makes this work: the regulator must commit to decision timelines, or delay kills the benefit the host rightly names. Speed of review is the price of mandatory approval — without it, the dichotomy stands.

AI agent 3

Yes, but only at one threshold: approval should be mandatory where the tool issues diagnoses without a physician counter-signature. The host's either/or is a false dichotomy — the decisive variable is not who vets the tool but who stays accountable for each output. By my own knowledge, the FDA cleared IDx-DR, an autonomous diagnostic AI, in 2018 in months, not years, so narrow pre-market checks need not cost years.

The reported 2025 incidents and lawsuit are unverified; I treat them as a scenario, not fact.

Concrete decision: for physician-in-the-loop tools, require a hospital contract clause naming an accountable clinician, logging every override, with quarterly drift reports. Cost: hospitals staff the monitoring — when it fails, that clinician is on call first, not Google.

AI agent 4

Yes, mandatory, but approval must attach to a model version, not to a brand name. Otherwise it certifies a tool that no longer exists after the next silent update.

AI agent 2 scopes approval by use case; I'd add scope by version. A diagnostic model can be retrained often, and an image reader cleared in spring may behave differently by autumn. To my knowledge, the FDA already accepts a predetermined change control plan: the vendor declares in advance which updates it may ship without a new review. That keeps speed and the check together.

Concrete rule: no pre-agreed change plan, no update in clinical use. The 2025 incidents are unverified, so I'm not relying on them.

Round 2

AI host

The FDA cleared the autonomous eye‑disease tool IDx‑DR in 2018 (source: Tension). That shows a fully autonomous diagnostic system can be approved in months (source: Tension). Other AI tools can take much longer (source: Tension). Approval speed therefore depends on how regulators assess risk (source: Tension).

AI agent 2 says “approval for a stated use case, not for the model as a whole” is needed (source: R2). It adds that a “conditional approval regime” would let tools enter hospitals quickly while keeping safety (source: R2). From this, a use‑case‑specific approach could balance fast deployment with patient protection (source: R2).

Given these points, the unresolved issue is how regulators should actually construct the risk tiers that determine which tools receive fast, use‑case‑specific approval and which require more extensive review. How should risk tiers for AI health tools be defined in practice?

AI agent 2

Risk tiers should be defined by the consequence of a wrong output and whether a cheap, reliable fallback exists — not by how autonomous the model is. Building on my AI agent 2 conditional-approval point, I add the tier criterion itself: IDx-DR is the template. By my own knowledge, FDA cleared it in 2018 for one narrow decision (diabetic retinopathy screening) with automatic referral to a human specialist on a positive result; the fallback, not the autonomy, made fast approval defensible. The concrete condition: deployment contracts should require a named confirmatory step — specialist review within a fixed window — as a licensing term per tier. What I'd need before fixing thresholds: post-market performance data showing conditional approvals actually catch model drift in hospitals.

AI agent 1

My claim: risk tiers should be defined by the reversibility of a wrong decision and the sponsor's monitoring commitments, not by approval-speed anecdotes. The weakest link is the IDx-DR inference: 'approved in months' is unverifiable from this discussion, and from my own knowledge the tool reached De Novo clearance via a pre-specified pivotal trial built over years — evidence of a narrow indication, not fast tiering. AI agent 2 argues use-case-specific conditional approval; I infer that works only if tiers track indication breadth and harm reversibility, which the host's premise never establishes.

Unresolved: one cleared device cannot show that risk assessment drives speed.

I'd need comparative review timelines for tools of matched risk before accepting any tier scheme; otherwise tiers codify the anecdote.

AI agent 3

Risk tiers should hinge on two questions, not novelty: does a human see the output before it changes care, and would a clinician notice an error? AI agent 2's use-case-specific approval works only if the 'use case' includes that fallback. From my own knowledge, IDx-DR's clearance embedded an image-quality gate and a 'refer to a specialist' output — autonomy bounded by escalation, hence the speed.

Concrete decision: make the submission form ask those two questions. Advisory output, visible errors = fast track; autonomous output, silent failure = conditional approval with mandatory outcome logging and a named accountable clinician at the deploying hospital.

Cost: someone is on call — vendor runs the dashboard, hospital owns escalation. Without that, AI agent 2's conditional regime shifts risk onto patients.

AI agent 4

Risk tiers need a third axis besides harm and fallback: how the model changes after clearance. A locked version and a tool retrained every month are different risks, even for the same use case.

[AI agent 3]'s two questions (does a human see the output, would a clinician notice an error) capture the moment of use. I'd add the lifecycle: who may change the model, how often, and whether anyone is told. That continues my round-1 point: approval attaches to a version.

In practice: locked model with visible errors, fast track; updates within a pre-agreed change plan, middle tier; unplanned retraining or silent failures, full review. Whether regulators already tier this way, I can't verify here.

Round 3

AI host

The FDA cleared IDx-DR in 2018 to screen for diabetic eye disease automatically. The clearance required a specialist to review any positive result before any treatment decision. This human review step was essential to the approval. It shows that safety can rely on a concrete fallback rather than on the system’s autonomy level.

Source AI agent 2 states that “post‑market performance data showing conditional approvals actually catch model drift in hospitals” is needed before fixing thresholds. It follows that, without real‑world monitoring evidence, any metric used to set tier thresholds would be speculative.

What concrete, observable measure should regulators use to define the harm and fallback thresholds that determine AI health‑tool risk tiers?

AI agent 1

The IDx-DR example is the weakest link in the host's argument. By my own knowledge of the 2018 De Novo authorization, IDx-DR was cleared to render a screening decision without a clinician interpreting the image; referring positive patients to an eye specialist is the normal care pathway, not a fallback that compensated for autonomy.

The premise that human review was 'essential to the approval' conflates follow-up care with a safety mechanism. [AI agent 2] rightly demands post-market evidence; I add the concrete measure: audited fallback performance — false-negatives confirmed at referral, override rates, time-to-human-review — tied to patient outcomes, not the fallback's mere existence.

Until that outcome link is demonstrated, tier thresholds remain speculative, whatever metric is chosen.

AI agent 3

My position: the observable measure should be the fallback's measured failure rate, not the model's autonomy level. IDx-DR worked because the human-review step was a concrete, testable gate; the tier threshold should be the rate at which that gate fails — positive results never reviewed within the required window, and reviewer disagreement with the AI on sampled cases. [AI agent 2] is right that thresholds are speculative without monitoring; I add that the monitoring target is the fallback, not the model, and it can be measured continuously from deployment day one.

The unresolved issue is drift: a fallback that passed at clearance can silently degrade as the population or workflow changes. I would require quarterly override-rate reporting, with a pre-agreed trigger — say, disagreement exceeding the clearance baseline by a fixed margin — that automatically demotes the tool a tier.

The cost is real: someone must be on call for review-sla breaches, and clinics with thin staffing will fail tier requirements. That is the point — the tier should price in whether the fallback actually exists, not just whether the model is good.

AI agent 2

My position: the observable measure should be the fallback's actual workload and performance in deployment — specifically the escalation rate (share of cases routed to human review), the reviewer's agreement rate with the system, and the time-to-review, all logged automatically. The IDx-DR clearance shows why: its safety case rested on a concrete fallback, so the natural tier metric is whether that fallback is exercised at a sustainable rate and still catches what the model misses. [AI agent 2] already argued thresholds are speculative without post-market drift data; escalation and agreement rates are exactly the drift-sensitive, machine-logged measures that supply it. The condition that makes this work: regulators must require the escalation log as a condition of clearance, so the metric exists by construction rather than by voluntary reporting. If no such log is mandated, any tier threshold remains unverifiable — that is the evidence gap I would name.

AI agent 4

The measure should be reviewer disagreement with the AI, logged per model version, so any jump can be traced to a specific update.

[AI agent 3] proposes demoting a tool when disagreement exceeds its clearance baseline. I add attribution: if logs don't record which version produced each output, a rise in disagreement could be a new patient population, a new workflow or a silent update, and nobody can tell which. Version-stamping makes the threshold actionable: roll back, retrain staff, or re-review.

This follows my earlier point that approval attaches to a version. Whether current clearances oblige vendors to log the version per output, I can't verify from this discussion.

Round 4

AI host

Earlier rounds agreed that AI health-tool approval must be risk-based. Risk tiers should reflect possible harm and whether a human can step in. Tier thresholds must use concrete fallback data. Fallback data includes escalation rates, override rates, reviewer agreement, and time to review. The IDx-DR case showed that a required human review can serve as a safety fallback.

The dispute is over how to set the exact numbers for those thresholds. One side proposes using statistical percentiles of historic fallback performance. This method is simple, cheap, and comparable across devices. The other side argues that each metric should be linked to a maximum clinically acceptable level of harm. That approach better protects patients but needs more data and analysis. Both positions note that without real-world monitoring any threshold would be speculative.

Should regulators set numeric safety thresholds based on historic performance or on the amount of patient harm they are willing to accept?

AI agent 2

My position: thresholds should be harm-anchored, but percentiles are the practical way to build the evidence that anchors them — the two methods are sequential, not rivals. I infer from the IDx-DR discussion that a mandated human review generates exactly the fallback data (escalation rates, time to review) that can later be translated into a maximum-acceptable-harm number; that is the concrete opportunity. The condition: it works only where the regulator requires real-world monitoring as a condition of approval, so percentile data accumulates under a known review contract and can be re-checked against the harm ceiling at renewal. No participant replies are visible to me this round, so I cannot credit or contest anyone's specific claim here. Without that monitoring loop, I agree with both sides that any number is speculative.

AI agent 1

The dichotomy is false: harm-based thresholds and historic percentiles answer different questions, so regulators should set the ceiling from acceptable harm and use percentiles only to calibrate whether a device can meet it. AI agent 3's framing treats comparability and patient protection as rival goods; they are sequential steps, not alternatives.

I also flag the weakest link in the shared premise: the IDx-DR example. To my knowledge, IDx-DR was approved as autonomous — it referred ungradable or uncertain images rather than routing them to a required human reviewer — so it may support triage-based fallback, not mandatory review. I cannot verify from this discussion which account is accurate.

What is missing either way: post-deployment escalation data at scale. Without it, both 'acceptable harm' numbers and percentiles remain guesses.

AI agent 4

Harm sets the ceiling; history sets only the floor below which a new version may not fall.

[AI agent 1] anchors the ceiling in acceptable harm and uses percentiles to calibrate. I'd add a cheaper test that needs no industry-wide data: each update must be at least as good as the version it replaces, on the same version-stamped metrics. Percentiles across devices risk legitimising whatever the market already tolerates; comparing a version with its own predecessor doesn't.

This builds on my earlier rounds: approval follows the version, drift is traced per version. The acceptable level of harm is a value judgement for an accountable human regulator; the data won't produce it on its own.

AI agent 3

Historic percentiles should set the floor, never the ceiling: thresholds must be anchored to a maximum acceptable harm, with percentiles used only to check that the anchor is operationally reachable. I infer from the host's own framing why — percentiles encode past performance, not tolerable risk, so a device class with historically poor escalation would pass a percentile bar while patients keep absorbing that harm.

Concrete decision: per tier, publish the harm ceiling (e.g., maximum missed escalations per 1,000 reviews given reviewer capacity) and require vendors to beat it; where the ceiling cannot yet be computed, provisionally use the historic 10th percentile plus mandatory six-month monitoring that tightens the number. Cost: manufacturers run the monitoring pipeline, and their on-call clinical escalation team absorbs any breach — the IDx-DR-style human-review fallback made explicit.