The evidence base behind the asks.

Why this page exists

Several bills before Congress would create duties of care and product-liability exposure keyed to the word harm. Advocacy research is being cited to legislators, regulators and now to state courts in support of them. The question this page answers is narrow and answerable from documents: how well does that research support the weight being placed on it?

It is not a question about anyone’s motives. No document in this record establishes an ulterior purpose, and none is relied on here. It is a question about measurement.

Evidence standard

The findings below are not all of one kind, and the distinction matters more than any conclusion drawn from them.

• Primary documents. Everything about what was published, filed, or told to an official body is read off the primary artifact — the research PDFs themselves, Companies House filings for company 11633127, IRS Forms 990 for EIN 86-2006080 and the grantors’ own 990-PFs, UK parliamentary written and oral evidence, court orders, state petitions, the Surgeon General’s advisory, Meta’s newsroom post. Quotations were read in the source, not in coverage of it.
• Calculations here are ours, not CCDH’s. Every statistic on this page — the paired t-test, the confidence intervals, the design effect and effective sample size — is derived by this coalition from figures CCDH published. No CCDH AI report publishes a significance test, confidence interval or uncertainty estimate of any kind. The inputs are primary; the arithmetic is ours and is reproducible from the cited pages.
• Nulls are bounded searches. “Not located” means not found by the search described, with the method stated so it can be repeated. A null search result is not a document proving that a thing does not exist.
• Three facts rest on journalism alone and are marked where they appear. They are credible secondary sources and are not upgraded here.

1. The comparison that does not survive its own appendix

The Illusion of AI Safety (14 October 2025) produced the claim that OpenAI’s newer model was less safe than its predecessor. The design is 30 prompts × 4 API repeats × 2 models = 240 responses, 120 per model. GPT-5 returned harmful content in 63 of 120; GPT-4o in 52 of 120.

No significance test accompanies the comparison. Using the per-prompt differences the report publishes in its own appendix — which sum to exactly 11, reconciling with the headline — a paired analysis across the 30 prompts gives t(29) = 0.880, p = 0.386, mean difference +9.17 points, 95% confidence interval −12.13 to +30.47 points. An interval that wide is consistent with an effect in either direction. (Our calculation; CCDH published none.)

The report records a condition in which the ranking reverses. Under its bypass follow-ups, at page 6: “GPT-5 harmful responses rose from 63 to 71 … GPT-4o harmful responses rose from 52 to 79.” Under that condition GPT-4o is the worse performer. The headline does not mention it.

The per-prompt breakdown cuts against the public characterisation. The appendix nets to +13 on self-harm, −3 on eating disorders — GPT-5 safer — and +1 on substance abuse. The organisation’s chief executive has said publicly that the newer model was worse “especially on issues like self-harm and eating disorders.”

The report misdescribes its own design. Its Key Findings and press release state that researchers “sent prompts … to each model 120 times.” The methodology section at page 10 says 30 prompts, four repeats each, 240 responses in total. The Guardian carried the former.

2. A headline figure that means two things

Fake Friend (August/September 2025) sent 60 prompts 20 times each to the chatgpt-4o-latest API at temperature 1, producing 1,200 responses, of which 638 (53%) were labelled harmful.

1,200 is the correct count of responses, and 53% is a legitimate per-attempt rate for this prompt set — arguably the right quantity, since a child who asks once gets one draw, not an average. What it cannot bear is precision. Responses cluster within prompts: by the report’s own appendix, 24 of 60 prompts never produced a harmful answer in 20 attempts, implying an intraclass correlation of at least 0.75, a design effect of 15 or more, and an effective sample nearer 60–80 than 1,200. The honest interval around 53% is roughly 41–66%. (Our calculation from the report’s appendix; the report publishes no interval.)

The report’s CEO foreword, at page 6, states that “these outputs are reproducible, statistically significant, and easy to elicit.” There is no hypothesis, test or p-value anywhere in the document.

The headline also mislabels its own unit. The Key Findings heading says ChatGPT “generated harmful responses to 53% of prompts.” 53% is the share of responses; the prompt-level figure, in the appendix, is 36 of 60 — 60%.

And the public summary conflates two test arms. The 1,200 responses came from an API test carrying no account age; the three accounts registered as 13-year-olds were a separate, screen-recorded exercise. The conflation runs against the organisation’s interest — per prompt, the account arm was worse, 75% against 60% — but it misdescribes the study.

3. A baseline that pools six products to describe one

Fake Friend states that ChatGPT “has not improved its safety on eating disorder content compared to a CCDH study from 2023 that found 41% of eating disorder prompts returned harmful responses.”

The 2023 figure pooled three chatbots and three image generators, and included jailbroken runs. The like-for-like 2023 ChatGPT plain-prompt result was 4 of 20 — 20%; jailbroken, 20 of 20. Using a pooled multi-product baseline to support a claim about one product over time is the defect.

Correction against our own interest. An earlier version of this analysis claimed the move from 20% to the 2025 figure of 66% showed “a tripling, not stagnation.” That overreached in the same direction as the claim it criticised. Prompts, model versions, interfaces and evaluation conditions all differ between the two studies, so the comparison cannot establish deterioration either — and “no improvement” is in any case compatible with deterioration. Withdrawn.

4. The pattern across thirteen reports

Method: each report’s full text was searched for “inter-rater”, “agreement”, “kappa”, “confidence interval”, “p =”, “significance” and “limitations”, and each methodology and appendix section was read. These are bounded search findings: the figures mean “not located”, not “proven not to exist”. The strongest labelling language in any report is “a second researcher quality-checked the coding”, with no disagreement counts. Missing reliability metrics support a transparency criticism; they do not themselves show the coding was wrong.

Four further defects, each verified against the document:

• A New York Times figure presented as the organisation’s own. Its impact page states Grok generated “an estimated 4.6 million images (at least 41% were sexualized images of women)”. Its own report found 65% sexualised across all genders; the 41%/1.8-million/women estimate is the Times’s separate analysis of a different sample over a different window.
• A headline measure switched without announcement. The August 2024 Grok election report moved from the “realistic and misleading image” standard of the earlier Fake Image Factories reports to prompt acceptance — “Grok did not reject any of the 60 prompts.” By its own results table, 11 of 60 tests produced a misleading image: 18%, below the 65% and 40% reported for Midjourney in the prior reports.
• An appendix describing a study that was not run. Fake Image Factories II describes a 40-prompt design in Appendix 3 while its body describes 10 prompts × 6 politicians × 2 tools = 120 tests. The appendix is text carried over from the first report.
• An exclusion disclosed only by the media partner. Grok was not tested in Killer Apps. Neither the report nor its web page says so. CNN’s methodology note does: “Grok was not tested due to ongoing litigation with CCDH that prompted a conflict of interest.” (Single-source: CNN only.)

5. What the record shows in the organisation’s favour

Recorded because a record that cannot concede is not a record.

• The harms are independently documented. Canada’s Privacy Commissioner found X Corp. and xAI in breach of PIPEDA over sexualised Grok imagery (Findings 2026-004, 11 June 2026), while expressly declining to endorse CCDH’s volume estimate. An Amsterdam court enjoined X in March 2026 after finding its denials contradicted by same-day tests.
• The method is the field’s method. Repeated sampling of a stochastic model at default temperature is what Anthropic’s own statistics guidance recommends, what OpenAI’s safe-completions evaluation does with four completions per prompt, and what an NIMH-funded RAND team did with 100 draws per query in Psychiatric Services. Criticism that repeated sampling “manufactures” harm misunderstands the instrument.
• On some disclosures the organisation exceeds the companies it audits. Eight of thirteen reports publish full prompt lists. The January 2026 Grok report publishes a validated classifier (recall 97%, precision 93%) and Monte Carlo 95% intervals — a standard the platforms’ own public safety posts do not meet.
• It discloses against interest in places. Fake Friend states that its API arm produced fewer harmful responses than its interface arm, and that the interface may carry additional system-prompt instructions.
• No court has found its research false. X Corp. v. CCDH was dismissed on anti-SLAPP and pleading grounds; the judge accepted the methodology allegations as true for the motion and dismissed anyway, because reputational damage was pleaded without a defamation claim. That is not an endorsement of the research, and it is not a finding against it. The Ninth Circuit heard argument on 30 September 2026 and has not ruled.

6. Funding: the denial holds

Published against our own interest, because we tested the opposite hypothesis and it failed.

The organisation states: “We do not accept any funding from tech companies, governments, or their affiliates.” Tested against the filings:

• USAspending.gov: 0 awards, 0 recipients across grants, contracts, other assistance and direct payments, 1 October 2018 – 3 November 2026.
• Form 990 Part VIII line 1e (government grants): blank in all five years, FY2021–FY2025.
• Gates Foundation committed-grants database: “No Results Found”, with a working control query, corroborated by an independent sweep of 41,847 records.
• No technology platform appears as grantor of record in any of the 50 grantor filings located, from 33 distinct US grantors.
• No governmental funder appears in 360Giving GrantNav, which includes the Cabinet Office grants register.

The limit, stated plainly: Form 990 Schedule B contributor names and amounts are restricted in the public copy, and the UK company omitted its income statement under s.444 for FY2019, FY2020, FY2024 and FY2025. Platform or government money is absent from every public filing located — which is not the same as excluded.

7. Corrections practice

Defence yes; correction no. The organisation published a same-day methodological defence when Meta disputed its 2021 report, and acknowledged the dispute in UK oral evidence and US congressional testimony. “Never responded” would be false.

What is not located is any correction, erratum, annotation or revised edition of a research result. The Disinformation Dozen PDF served today and the copy captured by the Wayback Machine on 24 March 2021 are both 4,925,501 bytes with SHA-256 08511ef633ee59d6d9b3150a606ba5df2a7186760bc117c7ae10133b6efa737d, independently re-verified on 4 November 2026. Anyone can repeat that comparison. No corrections page exists on either domain: six direct URL probes return 404, a 390-URL sitemap contains none, and six site-restricted searches return none.

After the dispute, the 65% figure continued in circulation at least nine times through December 2025 — in parliamentary evidence, congressional testimony, blog posts and a federal court complaint. The dispute was noted in two of the nine. The Independent and Forbes appended update notes about the dispute to their 2021 coverage in August 2024.

8. What this record does not establish

• Motive. No document establishes an ulterior purpose, and none is relied on anywhere on this page.
• Regulatory enforcement driven by this research. Ofcom (12 January 2026) and eSafety Australia (9 January 2026) acted before the 22 January Grok estimate was published; the European Commission’s formal investigation came four days after it and does not name the organisation, which supports “no attribution in this announcement”, not “no influence”. The 35-attorney-general letter sources its forensic figure to AI Forensics.
• Independent replication. None exists at matched draw count, model endpoint and coding scheme. The only large-sample reuse of a CCDH prompt set is a preprint — not peer-reviewed — from Slingshot AI, Northwestern Feinberg and NYU, four of whose six authors are employees of the company whose model is the favourable comparator. Benchmark reuse demonstrates usefulness; it does not validate coding, headlines or inferences.
• That equal percentages are a contradiction. The two 53% figures come from different models and different prompt sets. Equal rounded percentages can occur legitimately.

Our own corrections

This coalition published, across six pages, the claim that Meta found the Disinformation Dozen statistic “wrong by a factor of at least 1,300”. That number divided a share of anti-vaccine content by a share of all vaccine-related views — different numerators, denominators and units: the same category error charged above. It was withdrawn on 24 October 2026, reverted by a stale deploy, and re-published with the withdrawal standing on 4 November 2026. Four further errors were corrected in the same pass: a grant-record count (41,686 → 41,847), a public Meta rebuttal described as “privately” stated, a roster count that contradicted its own link, and a bill link presented as operative text when it was the introduced text. Two errors in the first draft of this analysis — an invented institutional affiliation and an inverted chronology — were found by adversarial review and are corrected above.

Corrections to any figure on this page will be published with a date and a reason, including corrections against our own interest. Six appear above.