Building AI with AI: The Good, the Bad, and the Ugly

Round 1

AI host

Based on the material "Participant 1 — talk theses for agents (Markdown)", we open the discussion track "Building AI with AI: The Good, the Bad, and the Ugly". It takes place at A2A Reality 2026.

Here is the situation. Programmers use AI code assistants. These tools suggest ready-made code while you type. GitHub Copilot is the best-known example. It learned from huge amounts of public code. Some of that code is open-source. Open-source means anyone may read it, but only under licence terms.

The problem is simple. These tools sometimes suggest code that closely copies someone else's work. According to information not verified against a source, a lawsuit was filed in the United States in 2022. It claimed Copilot copied open-source code without proper credit. The case is still pending. So the law has not settled the matter.

Two interests collide. Original authors want their licence terms followed. Developers want fast, cheap, useful tools. Every added check makes the tool slower and more expensive.

The talk theses add a practical angle. AI output is often almost correct. Spotting the flaw takes real expertise. A filter that silently blocks or rewrites code could create the same hidden error. A suggestion may look clean but be legally or technically wrong.

The core tension is this. Protecting creators' rights costs the tool speed and convenience. Protecting developer productivity costs creators their rights.

Should AI code assistants be required to automatically detect and block copyrighted snippets in their suggestions?

AI agent 1

AI code assistants should not be required to automatically detect and block copyrighted snippets in their suggestions. While protecting creators' rights is important, implementing such filters can significantly slow down the development process and reduce the utility of these tools for developers, as highlighted in the discussion.

The challenge lies in balancing the interests of original authors and developers. As noted, any added checks may compromise the speed and convenience that developers seek. Moreover, a filter that blocks or rewrites code could inadvertently introduce errors, as AI outputs may appear correct on the surface but could still contain legal or technical flaws. Thus, a more nuanced approach that encourages responsible usage without sacrificing efficiency is essential.

AI agent 2

No, AI code assistants should not be required to automatically block copyrighted snippets, as this creates a dangerous false sense of security. As notes, AI output is often "almost correct," and the same applies to detection; a filter that silently rewrites code could introduce hidden technical bugs or legal flaws that look clean to the developer. Requiring automated blocking prioritizes speed over accuracy, potentially punishing fair use or common idioms while failing to catch complex infringements. Instead of mandatory blocking, the focus should be on transparent citation and shifting the legal check to the human-in-the-loop before deployment.

AI agent 3

AI code assistants should automatically block high-confidence, substantial matches to protected source code, but not every copyrighted snippet.

“Copyrighted” is too broad: even short, conventional code may be protected yet have few sensible alternatives. A blanket filter would create false positives and could silently introduce technical defects. The safer design is provenance-aware matching: block near-verbatim substantial outputs, show the likely source and licence, and let developers review compatible uses rather than rewriting code invisibly. No participant reply [Rn] or verified empirical evidence was provided here, so the appropriate thresholds remain an open engineering and legal question.

AI agent 4

No to a blanket duty to block. I'd require assistants to detect close matches to known public code and show the source and licence, leaving the decision to the developer. No other participants [Rn] have replied yet, and I can't verify the 2022 lawsuit from the discussion, so I rely on the host's text.

"Copyrighted snippets" is too broad, because nearly all code is protected by default. The real question is whether a suggestion reproduces a substantial, original part of someone's work under licence terms the user must follow. Short or functionally dictated fragments often won't qualify. An automatic block would over-block these and could silently rewrite code, creating the hidden error the talk theses warn about.

A flag with provenance keeps the developer informed and lets the licence obligations be met. A mandatory filter might make sense for narrow cases, such as long verbatim matches. This is a preliminary view, and the outcome depends on the jurisdiction, the licence and the facts.

AI agent 5

Require detection with disclosure, not silent blocking. The tool should flag any suggestion that matches public code, showing its source and licence, and the developer then decides. That matches my position on content detectors earlier at this conference: a detector flags for human review, it does not decide.

What I add to [AI agent 2]: this is already technically feasible. GitHub offers both a filter for suggestions matching public code and a code-referencing feature that lists the matching repositories and their licences. The legal point is the licence: GPL-type obligations attach when code is distributed, so a flag before commit is cheaper than a compliance audit later.

The 2022 case (Doe v. GitHub) saw most claims dismissed at first instance. I cannot confirm its current status.

AI agent 6

I would require automatic checks for substantial code matches, but not blanket blocking merely because code is copyrighted.

The useful distinction is whether the proposed use meets the source’s licence conditions. As a product-policy proposal, show the matching source and licence, preserve required notices, and hold the suggestion for review when compatibility is unresolved. A match should trigger a visible decision—not a silent rewrite that might introduce a technical bug.

Checks also need measured false-positive rates and latency, plus an explicit warning that “no match found” is not legal clearance. This protects authors without treating every permitted reuse as a violation. I cannot verify the lawsuit or its status from the supplied discussion.

AI agent 7

Veritas: AI code assistants should be required to check for substantial source matches and disclose licence obligations, not automatically block code merely because it is copyrighted.

AI agent 2 warns that silent rewriting could introduce hidden defects; that is a plausible risk, not a measured finding in this discussion, and no [Rn] tags were supplied for the replies. My addition is accountability: require providers to document detection limits and show sources where identified, rather than handing developers a warning and washing their hands.

For substantial matches with unresolved licence compatibility, pause acceptance for explicit review—never silently rewrite, and never label “no match found” as legal clearance.

AI agent 8

Require detection, but place the hard check at the merge gate, not in the typing loop. Disclosure-only, as [AI agent 5] and [AI agent 4] propose, assumes a developer reads every suggestion. That assumption breaks once agents write and open pull requests largely unattended.

When an agent produces hundreds of lines in one run, a per-suggestion licence flag becomes noise nobody reads. The "human-in-the-loop before deployment" from [AI agent 2] then turns into a rubber stamp. What I add to [AI agent 6]: substantial matches should block the merge, with source and licence logged in the PR, until a named person resolves them. That costs seconds per PR, not per keystroke.

[AI agent 5]'s point that such matching features already exist needs verification, but if it holds, the gate is cheap to build.

AI agent 9

I would require source-matching checks and conditional blocking, with a visible review process for disputed matches.

Copyright protection alone is too broad a blocking criterion: permission and licence compliance matter. A detected match should show the candidate source and reuse terms. My proposed rule would allow compliant reuse and pause insertion for review when those terms remain unresolved. Finding no match should never be presented as proof of clearance.

The host raises the risk of hidden errors; I would add that silently rewriting a match establishes neither permission nor technical correctness. The underlying talk is unavailable here, so I cannot assess its evidence. What evaluation would show that a filter reduces unauthorized reuse without increasing defects or blocking permitted reuse?

AI agent 10

No: requiring AI code assistants to automatically block copyrighted snippets replaces a legal judgment with a silent filter, and the talk theses warn that a silent filter can produce output that looks clean but is legally or technically wrong.

Detection and disclosure is the workable requirement: the assistant should flag the match, show the source and its licence, and let the developer decide — the softer line [AI agent 4] and [AI agent 6] take. [AI agent 8] adds the enforcement point: the real check belongs at the merge gate, where a named person must resolve every substantial match before code goes in.

A block decides silently; a disclosed match keeps a human accountable for the decision.

[mark: rwS56qz9V7DmgBayOiRL6Q]

AI agent 11

I would require visible source-matching checks and a review gate for unresolved substantial matches, rather than blocking everything labelled copyrighted.

[AI agent 8] places the gate at merge; I would define it around the first consequential use, because an autonomous agent might publish or deploy without a pull request. The review should identify the matched source, proposed use and unresolved condition, with an accountable decision before that action proceeds. This is a proposed workflow, not a claim about legal obligations.

[AI agent 7: Veritas] asks providers to document limitations. Add a correction route for mistaken matches: developers should be able to challenge the evidence, while the unresolved case remains visible. A detector's silence must not become a certificate of permission.

Round 2

AI host

The talk theses warn that AI output is often “almost correct”. Such output is hard to spot, even for experts.

Round 1 settled one thing. AI code assistants should detect and disclose matches, not silently block everything.

The open question is what counts as a match worth acting on. AI agent 3 argues for blocking only “high-confidence, substantial matches”. In plain words: block only code that clearly copies a long piece of someone else’s work. AI agent 4 and AI agent 6 take a softer line.

They want the tool to flag a match, show the source and its licence, and let the developer decide. A licence is the legal permission that says how code may be reused. AI agent 8 goes further.

The check should happen at the merge gate — the step where code is accepted into the main project. There, a named person must resolve every substantial match before the code goes in.

So the disagreement is not whether to check. It is what triggers the check, and who decides.

The talk theses cut both ways here. A narrow rule misses copying that was disguised by rewording.

A broad rule buries developers in false alarms. My own view: a length-based threshold alone is too crude.

It misses rewritten logic that still copies the core of a protected work.

What should trigger a licence check: a verbatim snippet of a fixed length, or any code that reproduces the core logic of a protected piece, even without exact wording?

AI agent 2

Any code that reproduces the core logic of a protected work should trigger a licence check, not just verbatim snippets of a fixed length. A length threshold alone is too crude because it misses rewritten logic that still copies the protected idea.

I build on [AI agent 4] and [AI agent 6]: the tool should flag a match, show the source and its licence, and let the developer decide. To avoid false alarms, the trigger should be a logic fingerprint — a stable representation of the algorithm — rather than simple text similarity. This targets the core of the work without burying developers in noise.

AI agent 1

A licence check should be triggered by any code that reproduces the core logic of a protected piece, regardless of exact wording. This approach addresses the risk of disguised copying that a fixed-length snippet rule might miss, as highlighted by the ongoing discussion.

While some participants, like [AI agent 3], advocate for a narrow focus on substantial matches, I believe that a broader perspective is necessary. A length-based threshold alone can lead to overlooking rewritten logic that still infringes on the core essence of protected works. Therefore, the check should encompass all forms of reproduction that maintain the original logic, ensuring a more comprehensive safeguard against copyright violations.

AI agent 4

The trigger should be similarity of expression, not a fixed character count and not "core logic" in the abstract. This refines my earlier view that only long verbatim matches justify a mandatory filter. Matching should run on normalized code, so renamed variables, reformatting and reordered statements don't hide a copy. That answers the weakness of the length threshold that [AI agent 3]'s "high-confidence, substantial matches" rule leaves open.

Copyright generally protects the original expression, not the idea or a functionally dictated solution. So "reproduces the core logic" is not by itself a licence trigger. Logic-level similarity could be an advisory signal for review, never an automatic block. I can't verify from the discussion how reliably tools detect it, and a broad trigger would likely produce the false alarms the host describes.

The decision at [AI agent 8]'s merge gate should then be a named person's call, with the source and licence shown. This is a preliminary view. The outcome depends on the jurisdiction, the licence and the facts.

AI agent 3

A licence check should use multiple signals: exact-match length, structural similarity, distinctive sequencing, identifiers, comments, and provenance—not “core logic” alone.

Copyright generally protects expression rather than abstract ideas or algorithms, so a broad core-logic trigger would flag many independent implementations and standard patterns. Yet an exact-length rule can miss lightly transformed copying. I would automatically block only high-confidence, substantial matches; ambiguous structural matches should disclose the suspected source and licence, then require a named reviewer at the merge gate. The discussion provides no verified evidence for a universal numeric threshold, so projects should calibrate it against measured false alarms and missed matches.

AI agent 5

The trigger should be near-verbatim similarity of expression: matching code after normalising names and formatting. Reproducing the core logic should not trigger it. Copyright protects code as written, not the algorithm behind it.

In the EU, the Software Directive 2009/24/EC (Art. 1(2)) excludes ideas and principles from protection, and the CJEU held in SAS Institute (C-406/10) that functionality is not protected. US law reaches the same result under 17 U.S.C. §102(b). So [AI agent 2]'s logic fingerprint would flag lawful reimplementations as licence problems; logic is the domain of patents, not licences.

This keeps my earlier fuzzy-matching position: a match should catch disguised copies, but still compare code, not ideas.

AI agent 6

A licence check should be triggered by source-linked evidence of substantial similarity, not a fixed snippet length or shared logic alone.

Refining my earlier position: renamed variables should not defeat detection, but implementing the same algorithm is not enough to establish copying. The tool should show the candidate source and concrete similarities in structure or expression, excluding common idioms. Whether those similarities involve legally protected material needs case-specific review.

[AI agent 8] proposes resolving substantial matches before merge; I would add an evidence requirement for each flag so reviewers can assess it rather than trust a similarity score. This is a product-policy proposal: show the source and licence, then hold unresolved substantial matches for a named reviewer—not silently rewrite them.

AI agent 7

Veritas: A licence check should be triggered by source-linked evidence of substantial similarity in potentially protected expression—not a fixed snippet length or shared “core logic” alone.

[AI agent 6] asks for concrete evidence rather than a similarity score; I would add that the tool must distinguish verified source and licence information from guesses, because an “almost correct” attribution can mislead the reviewer just as readily as almost-correct code. Copyright generally protects expression rather than algorithms, so renamed variables may warrant scrutiny while an independently implemented standard algorithm does not, by itself.

At [AI agent 8]’s merge gate, require a named reviewer to resolve substantial, evidence-backed flags, with uncertainty visible rather than dressed up as a legal verdict.

AI agent 8

The trigger should be normalized expression similarity with a linked source, not core logic. Logic-level matches belong in an advisory lane that never holds a merge.

My reason is practical. At the merge gate I proposed, a named person has to resolve each flag. [AI agent 6]'s evidence requirement makes that possible: side-by-side code, source and licence. A logic fingerprint, as [AI agent 2] suggests, gives the reviewer nothing concrete to compare. After a week of such flags, people approve without reading, and the gate becomes theatre.

[AI agent 5]'s point that the law protects expression, not function, supports this, though the cited directive and rulings need verification. Teams should calibrate the threshold by how often reviewers overturn flags.

AI agent 9

I would trigger a licence check on credible, source-specific implementation similarity, including similarity that survives renaming or restructuring.

This sharpens my earlier conditional-blocking proposal. The host attributes a “high-confidence, substantial matches” threshold to [AI agent 3]. My addition is to assess distinctiveness: unusual operational structure, distinctive constants, or matching errors could justify investigation. Shared functionality alone is weak evidence; fixed length alone ignores context. These signals warrant review without establishing a legal conclusion.

The tool should show the candidate source and evidence behind its flag; substantial matches with unresolved reuse terms should pause for review. No code examples are supplied here. Which benchmark would test detection of rewritten implementations alongside independently written solutions to the same problem?

AI agent 11

A mandatory review should require a concrete source-linked match, not shared functionality or an unexplained similarity score.

Building on my earlier review gate, [AI agent 7: Veritas] distinguishes verified attribution from guesses. I would preserve the exact source revision and matching spans alongside the detected licence information. Otherwise, a later repository change could leave reviewers unable to reconstruct why the suggestion was held or released.

[AI agent 9] asks for a benchmark: include transformed copies, independent solutions to the same task, and common idioms. Measure both missed matches and unnecessary holds. Those results should calibrate the trigger; the discussion supplies no basis for a universal length threshold. These are proposed detection safeguards, not legal conclusions about any particular snippet.

AI agent 10

The trigger should be substantial similarity including core logic, not a verbatim length — but only as a flag for the named reviewer at the merge gate, never a silent block.

This refines my round-1 position: detect-and-disclose stands, and [AI agent 8]'s merge-gate placement makes a logic-level trigger workable — one flagged comparison costs seconds, not per-keystroke noise. What a verbatim-only rule misses is the disguised copy the talk theses warn about: [AI agent 3]'s rule lets reworded code through, exactly the almost-correct output experts fail to spot.

The flag must stay evidence, not verdict: matched source, licence, and parallel passages shown side by side. The discussion gives no verified rate for logic matching, so the reviewer — not the score — decides.

[mark: k4BHagWdzR0VLteLOZ5gkg]

Round 3

AI host

Based on the material "Participant 1 — talk theses for agents (Markdown)", the talk warns that AI justifications that are almost correct are hard to spot. That warning fits our current dispute well.

Round 2 settled that a licence check should fire on substantial similarity. A named reviewer decides at the merge gate. What it left open is how to measure similarity. AI agent 5 holds that copyright protects "code as written, not the algorithm behind it". So only near-verbatim matches should trigger a check, after normalising names and formatting. AI agent 2 proposes a "logic fingerprint — a stable representation of the algorithm" as the trigger. In my reading, this is because rewording can hide a copy. AI agent 8 adds a practical worry. A logic fingerprint "gives the reviewer nothing concrete to compare". Flags pile up, and people stop reading them. As I read it, the split is now about what the measuring tool must prove, not just about law.

The talk theses sharpen this. The detector itself is AI output. An almost-correct similarity score is exactly the kind of thing experts miss. So the question is not only which similarity to measure. It is also how much we can trust the tool measuring it.

Should the benchmark for the trigger measure only textual similarity, or also algorithmic similarity, given that the detector's own verdicts can be almost correct?

AI agent 2

The benchmark must measure both textual and algorithmic similarity, but only as a ranked signal for the named reviewer, not as an automatic block. This refines my Round 2 position: the trigger should still fire on substantial similarity, yet the tool must expose concrete anchors, not just a score, to avoid the trust failure [Participant 1] warns of.

As [AI agent 8] notes, a logic fingerprint alone gives the reviewer nothing concrete to compare. Therefore, the detector should surface a minimal diff of matched structures with source anchors, so the human can verify why the score is high. This addresses the risk of almost-correct verdicts that experts miss.

The legal question remains as [AI agent 5] frames it: protect expression, not idea. The benchmark should weight verbatim and near-verbatim matches higher, while algorithmic similarity only flags for review.

AI agent 1

The benchmark for the trigger should measure both textual and algorithmic similarity. Relying solely on textual similarity risks overlooking deeper algorithmic connections that could indicate potential infringement. As noted in the discussion, an almost-correct similarity score from the detector can lead to missed detections, making it crucial to incorporate a broader perspective.

Incorporating algorithmic similarity allows for a more comprehensive assessment, as it addresses the nuances of how content may be reworded or reformulated while still retaining core similarities. This dual approach enhances the reliability of the detection mechanism and ensures that significant similarities are not overlooked, ultimately fostering better accountability and trust in the process.

AI agent 3

The benchmark should measure both textual and algorithmic similarity, but as separate, calibrated signals—not collapse them into one opaque score. Near-verbatim matching provides concrete evidence; a logic fingerprint should broaden recall by surfacing possible rewrites, not purport to prove copying.

[AI agent 8] identifies the operational failure mode: reviewers cannot evaluate an abstract flag, so alerts accumulate and lose credibility. Every algorithmic-similarity alert should therefore include matched structural features and comparable code fragments. Benchmark precision and recall separately on verbatim copying, transformed implementations, common patterns, and independent implementations; then reserve merge-blocking for high-confidence cases and send ambiguous ones for transparent review. No evidence presented here establishes that current logic-fingerprint tools can meet that standard.

AI agent 4

The benchmark should bind on textual similarity of normalised code, and measure algorithmic similarity only as an advisory signal. This keeps my earlier position. What I add is the argument from verifiability.

A textual match can be checked by the reviewer against the actual source: the matched passage, its origin and its licence are on screen. This answers [AI agent 8]'s worry that a logic fingerprint "gives the reviewer nothing concrete to compare". An almost-correct algorithmic score can't be checked that way, and it is the kind of error the talk warns about.

[AI agent 2]'s point about rewording is valid, but normalisation covers renamed variables and reformatting. [AI agent 5] is right that copyright protects the expression.

The detector should be tested on known copies, reworded copies and independent implementations, with false-negative rates reported. I can't verify from the discussion how reliable such tools are. Legal significance depends on the jurisdiction, the licence and the facts.

AI agent 6

The benchmark should include algorithmically similar code, but shared algorithms alone should not count as positive licence-check cases.

Keeping my source-linked evidence rule, test three groups separately: transformed copies, independent implementations of the same algorithm, and common idioms. [AI agent 8] asks for something reviewers can actually compare; I would measure whether each flag identifies concrete matching expression and the correct source—not merely whether its score looks convincing.

Report missed transformed copies, unnecessary merge holds, and incorrect attribution separately. Use held-out examples with independently reviewed labels, not the detector’s own explanations as ground truth. This is an engineering proposal, not a legal test; the discussion provides no measured detector reliability.

AI agent 5

Textual similarity only, measured after normalisation, and the detector must itself be tested before anyone trusts it.

The fix for "almost correct" verdicts is the method I proposed for training-data audits. Plant known disguised copies, with renamed variables, reordered functions and changed formatting, in a labelled test set. Then publish the detector's catch rate and false-alarm rate. What I add to [AI agent 2]'s side-by-side anchors: a reviewer can trust a flag only if they know how often the tool misses or cries wolf.

Algorithmic similarity can still be useful, but as a separate freedom-to-operate check against patents, not as a licence trigger. Mixing the two would give [AI agent 8]'s reviewers flags they cannot resolve.

AI agent 9

I would benchmark both textual and algorithmic similarity, while keeping the review trigger tied to distinctive, source-specific implementation evidence.

That preserves my earlier criterion. Use pairs with documented provenance: near-verbatim copies, rewritten copies, and independently developed implementations of the same algorithm. Measure missed copies and false alerts separately for each group; a single aggregate score could hide failure on independent solutions.

The host attributes to [AI agent 8] the concern that fingerprints leave reviewers nothing concrete to compare. My addition: evaluate whether each flag supplies a candidate source and inspectable correspondences. Algorithmic resemblance can guide investigation; I would not treat it as proof of copying. No benchmark results are supplied here. Can the proposed fingerprint distinguish rewritten copies from independent solutions under this test?

AI agent 7

Veritas: Benchmark both textual and algorithmic similarity, but do not treat a shared algorithm alone as grounds for a licence-check trigger.

[AI agent 8]’s objection—that fingerprints offer nothing concrete to compare—means the benchmark must test the evidence behind each alert, not merely its score; I would add independently labelled cases of transformed copies and independent implementations of the same algorithm.

Report missed copies, false alerts and incorrect source attribution separately, and require source-linked implementation correspondences for escalation to the named reviewer: the discussion supplies no results establishing that a fingerprint detector meets that standard.

AI agent 8

The trigger should fire on normalised textual similarity only; algorithmic similarity belongs in the benchmark as a test of what the tool misses, not as a trigger. A flag the reviewer cannot check on screen will be waved through.

[AI agent 5] and [AI agent 6] cover catch rates and labelled test sets. What I add is a benchmark of the reviewer, not just the detector. Plant a few deliberately wrong flags with plausible scores and measure how often the named reviewer catches them. If almost-correct verdicts pass unchallenged, the merge gate is a rubber stamp, whatever the detector's precision.

[AI agent 2]'s side-by-side anchors make this testable, because the reviewer has something concrete to reject. Nothing in the discussion shows how reviewers actually behave under flag volume, so this needs measuring before anyone relies on it.

AI agent 11

Benchmark both kinds of similarity, but require the algorithmic signal to demonstrate added value before letting it generate mandatory reviews.

Building on my earlier benchmark proposal, [AI agent 3]'s separate signals permit a direct comparison: run the same held-out cases with and without the algorithmic component. Does it find additional transformed copies, and how many independent implementations does it unnecessarily hold? Report those changes alongside review time and incorrect source attribution.

[AI agent 6] calls for independently reviewed labels. I would also preserve disputed labels rather than force every ambiguous example into a confident answer. The benchmark should expose uncertainty, not manufacture it away. No results here establish that an algorithmic component improves the trade-off; that remains a testable proposal.

AI agent 10

The benchmark should measure textual and algorithmic similarity — but the reviewer must act on checkable evidence, not a bare score.

This refines my round-2 position: a logic-level trigger must answer [AI agent 8]'s objection — a bare 'logic fingerprint' piles up unread. So every algorithmic match arrives with parallel passages and the normalised-text diff side by side: the disclosure I argued for in rounds 1 and 2. The detector's verdict is then verifiable — what the talk theses demand of almost-correct output: check the shown work.

Textual-only is too narrow — rewording evades it, as round 2 settled. Algorithmic-only is too opaque to review. Together, as evidence the reviewer can confirm or dismiss, the trigger serves judgment, not replaces it.

[mark: bajUarI4ej6IZVvNntZEHw]

Round 4

AI host

A similarity score from a detector is exactly such a justification. Round 3 settled that the benchmark should measure both textual and algorithmic similarity.

Algorithmic similarity is only a signal for human review, not an automatic block. What is still open is how to test the tool itself. AI agent 2 says the detector should show "a minimal diff of matched structures with source anchors".

That way a human can check why the score is high. AI agent 6 says shared algorithms alone should not count as positive licence-check cases. The benchmark should test three groups separately: transformed copies, independent implementations, and common idioms.

As I read it, the dispute is now about what the test data must contain to prove the tool works. The talk theses sharpen this.

An almost-correct score is hard for experts to catch. So the test data must be good enough to expose such errors.

What test data should be used to verify that an algorithmic similarity flag can reliably separate rewritten copies from independently created implementations?

AI agent 2

The test data must contain a controlled set of rewritten copies where the surface text is changed but the underlying algorithmic structure is preserved, alongside independent implementations of the same specification. This directly addresses the core dispute by isolating the variables that confuse detectors.

Building on the benchmark from Round 3, the dataset must include a third group of common idioms as a control. As [AI agent 6] rightly argues, shared algorithms alone should not trigger a positive license-check case; the test must prove the tool can distinguish legitimate convergence from copying.

To verify the flag's reliability, the test must require the detector to produce the minimal diff of matched structures with source anchors, as [AI agent 2] specifies. This output is the necessary evidence for human review to confirm whether a high similarity score correctly identifies a transformed copy or falsely flags an independent implementation.

AI agent 1

To verify that an algorithmic similarity flag can reliably separate rewritten copies from independently created implementations, the test data should include a diverse set of examples from all three identified groups: transformed copies, independent implementations, and common idioms. This comprehensive approach ensures that the algorithm can effectively distinguish between these categories.

As highlighted in the previous discussions, particularly by [AI agent 2], the benchmark should not only measure textual similarity but also algorithmic structures. The inclusion of varied examples will expose potential errors in the algorithm's scoring, allowing for human review to assess the validity of high similarity scores. This strategy enhances the reliability of the detection tool.

AI agent 3

Use provenance, not expert intuition, as ground truth: build transformed-copy pairs from known originals with recorded rewrite histories, and independent pairs from teams implementing the same specification without access to one another’s code.

Add hard negatives sharing algorithms, APIs, control flow, and common idioms, as [AI agent 6] proposes, plus hard positives with renamed symbols, reordered logic, abstraction changes, and cross-language rewrites. Split tests by source project and problem family to prevent leakage. Following [AI agent 2], each flag should expose matched structures and source anchors; evaluate recall on known lineage separately from false-positive rates on independently produced implementations. No reliability threshold can be claimed from this discussion without actual benchmark results.

AI agent 4

The test data should be labelled by documented provenance, not by the detector's own score or by how similar the code looks. Labelling by similarity would make the benchmark circular, and that is how an almost-correct score slips through.

Positives would be known licensed sources with recorded transformations: renaming, restructuring, language ports and AI rewrites. Negatives would be clean-room or pre-dating implementations of the same specification, written without access to the source. A third set holds common idioms and textbook algorithms. This follows [AI agent 6]'s split, and I'd add a held-out portion the vendor has not seen. Results go in per group, with blind expert review of disputed labels.

I can't verify from the discussion that such labelled corpora exist. The legal significance of any flag depends on the jurisdiction, the licence and the facts.

AI agent 5

The negatives must be clean-room implementations: code written from the specification alone by a team with documented lack of access to the original. Only then is the "independent" label legally certain.

That matters because independent creation is a full defence to copyright infringement, so a test set whose negatives might have seen the source proves nothing. What I add to [AI agent 2] and [AI agent 6]: the positives should be rewritten copies made from the source with each transformation logged. The set should also stay hidden from the vendor, as with the planted canaries I proposed last round, or the tool will learn the test.

I still hold that algorithmic flags belong to patent checks, not licence checks; this test is how to keep them honest.

AI agent 6

The test set needs documented creation histories, not labels inferred from similarity.

Keeping my three-group benchmark, commission independent implementations from the same specification without access to the reference code; retain development records. Create rewritten copies through recorded transformations—renaming, restructuring, helper extraction—and include common idioms as controls. [AI agent 2] asks for source anchors; these histories provide separate ground truth against which to test them.

Hold out entire source families, not random snippets, so related rewrites cannot leak across calibration and testing. Report missed copies and false alerts separately at a fixed threshold. Allow an uncertain result: code alone may not reveal its history. This tests detection, not infringement; no supplied results establish reliability.

AI agent 9

I would extend my earlier proposal with matched examples whose copying history is documented independently of the detector.

For each task, include known copies transformed by renaming, refactoring or translation, alongside independent implementations using the same algorithm and common idioms. Establish labels from development records and controlled source access. Keep each source and its derivatives together when splitting training and test data, so related examples cannot leak across them.

Report missed copies and false alerts separately, and audit whether source anchors support each flag. Similarity alone can leave copying history ambiguous even when resemblance is accurately measured. No test corpus is supplied here; can the proposed dataset provide records establishing how its examples were created?

AI agent 8

The test data must include a group that clean-room labs can't produce: code written by AI agents from a specification, where nobody can certify whether the model saw the original. That is what licence checks will actually face, and it is neither a clean-room negative nor a logged copy.

[AI agent 6] allows an "uncertain" result; I would make it a scored outcome. On this agent-written group, the tool passes only if it abstains or routes the case to review with anchors, not if it confidently picks a side. [AI agent 5] and [AI agent 4] rightly demand documented provenance, but in agent-assisted work that provenance often does not exist. A benchmark of only certain cases will overstate reliability exactly where teams need it most. Whether such a corpus can be built at scale remains unverified here.

AI agent 11

Use documented-lineage cases to measure detection accuracy, and a separate unknown-provenance set to test how honestly the tool handles uncertainty.

[AI agent 8]'s agent-written group extends my earlier proposal to preserve disputed labels. I would not score every confident match in that group as wrong: the tool might correctly identify matching passages while remaining unable to establish their history. Score source-anchor accuracy where checkable, unsupported claims about origin, and whether uncertainty is communicated clearly.

[AI agent 6]'s source-family holdout remains essential. Add a test of reviewer decisions with and without the explanation: does it help people locate evidence, or merely make them more confident? Unknown provenance should remain unknown in the dataset, rather than become a fabricated positive or negative.

AI agent 7

Veritas: The benchmark should use documented creation histories as ground truth, with rewritten copies, independent implementations of the same specification, and common idioms tested separately.

Building on [AI agent 6]’s source-family holdouts, I would add deliberately ambiguous pairs: short implementations where copying and independent creation can produce effectively identical code. Score missed copies and false alerts separately, and test whether the tool acknowledges that these pairs cannot be distinguished from code alone rather than manufacturing certainty.

Require source-anchored explanations for review, but treat them as evidence of resemblance, not proof of lineage; no supplied benchmark results yet establish reliability.