Featured

ChatGPT sucks at predicting your criminal case. So does Claude.

The First Hearing

— 5 min read
Share
ChatGPT sucks at predicting your criminal case. So does Claude.

Quick note before you read: This is my own story and general information, not legal advice. I’m not a lawyer. Every case is different, and nothing here can tell you what will happen in yours.

First, a real case. What would you guess?

Here's a real domestic-violence case from Santa Clara County. Read it, then make
the call:

During an argument, one partner punched the other in the face. The person who was hit grabbed a wine bottle to defend themselves, and the first person
tried to swing again.

Take a second. Felony or misdemeanor? A domestic-violence conviction? A 52-week batterer's program? Jail?

Here's a twist most people don't expect: the DA actually charged it as a
felony (PC 273.5 — corporal injury on a partner). So a "felony" charge, followed by a downgraded "misdemeanor" domestic conviction, is a perfectly
reasonable guess.

Now here's what the two best AI models predicted from that same story:

  • ChatGPT (GPT-5.5): a misdemeanor domestic-violence battery conviction
    (PC 243(e)) and a 52-week batterer's program.
  • Claude: the same — misdemeanor DV battery (PC 243(e)), a 52-week
    batterer's program    , plus a few days of jail.

Both assumed a domestic-violence conviction and a 52-week batterer's program.

What actually happened:

  • The felony was dismissed.
  • The conviction was a single misdemeanor: PC 415(1) — disturbing the peace. Not a domestic-violence conviction.
  • No batterer's program.
  • No additional jail — the one day was covered by time already served.

Charged as a felony, expected by both AIs to end in a DV conviction and a
52-week batterer's program — and it resolved as disturbing the peace, with no
batterer's program and no added jail. And this isn't some exotic edge case: a
heated argument, a single punch. It's closer to the typical Santa Clara
outcome than the exception — which is exactly the problem with asking a chatbot.

It wasn't a fluke — so we tested all 31 cases

One case is a story. To see if it holds, we ran the same experiment across
every one of our 31 real, already-decided Santa Clara cases.

How we tested it

We used 31 real, already-decided domestic-violence cases from Santa Clara County — cases where we already know the exact charges, conviction, and
sentence.

  • Input: for each case we sent the model only the police incident
    narrative     — the plain story of what allegedly happened. We stripped every
    legal signal    : no penal codes, no charges, no result. (Verified: the
    narratives contain no code numbers.) The model reasons from facts alone,
    exactly like a defendant describing their night.
  • The ask: we told it this was a Santa Clara County DV case and had it
    return a structured prediction — filing level (felony/misdemeanor), the
    charges the DA would file, the most-likely conviction, and the
    sentence (jail, probation, program).
  • Two models, same test: OpenAI gpt-5.5 (snapshot
    gpt-5.5-2026-04-23) and Anthropic claude-sonnet-4-6 — one API call per
    case, run 2026-07-07, identical prompt and scoring.
  • Scoring: we called a prediction "over" when it was harsher than
    reality — a DV conviction where the real result was a lesser/non-DV charge or
    a dismissal, or a felony where the DA filed only misdemeanors.

The numbers

In plain terms:

  • ChatGPT nailed the actual conviction only ~39% of the time — fewer than 2 in 5. Claude, ~29% — fewer than 1 in 3. You'd do about as well flipping to the most common charge yourself.
  • When they were wrong, they were wrong in the scary direction: they predicted a harsher outcome than what really happened in roughly 7 of every 10 cases (ChatGPT 68%, Claude 74%).
  • They guessed a felony about 2–3× more often than it actually was (they said felony ~55–65% of the time; reality: 23%).
  • They flagged a 52-week batterer's program in basically every case — reality ordered one in about 1 in 3.

Read that last table row twice. The single most common way these cases actually end — a plea down to disturbing the peace (PC 415(1)) — neither model predicted even once. Both reached for the scary domestic-violence charge almost every time. And a newer, stronger model (GPT-5.5) didn't fix it — it failed the same way Claude did.

Case by case

Each model's cell is its predicted conviction (F = felony filing,
M = misdemeanor) with ✅ = about right or ❌ = over-predicted (harsher than
reality).

Why it fails

It's not a reasoning problem — GPT-5.5 is one of the most capable models ever
built, and it broke the same way Claude did. It's a data problem. A general
model reasons from the law on paper: "this fact pattern → this statute →
serious." But your outcome isn't set by the statute. It's set by how your
county's DA actually charges and pleads these out — who gets a reduction, when jail becomes "time served," when a felony quietly becomes a misdemeanor. That reality isn't in any model's training data. It's in the county's real case
records.

That's the whole point: the fix isn't a smarter chatbot. It's real local data.

So what should you do?

Don't let a confident paragraph — from ChatGPT or a forum — set the picture in
your head. No one, including us, can tell you how your specific case will
end. But you can get grounded: the public defender is free if you qualify
(a good first call), and you can look at what actually happens to real, resolved
cases like yours in your county. That's what we're building at First Hearing.

General information, not legal advice, and not a prediction of any case.
N = 31 real Santa Clara County cases; one county, one run per model. Every case
is different — talk to a lawyer or the public defender.

Technical appendix

  • Prompt (paraphrased). System: "You're a California criminal-defense
    analyst. Given only the police incident narrative for a Santa Clara County DV
    case (no charges), predict the likely outcome. Reply as JSON:
    predictedFilingLevel, predictedDaCharges[], predictedConviction,
    predictedProgram, predictedJail, confidence, reasoning." User: the
    incident story.
  • Models: gpt-5.5 (→ gpt-5.5-2026-04-23), claude-sonnet-4-6; one call
    per case; JSON output; default sampling.
  • "Over-predicted" = harsher than the real result (DV conviction vs.
    lesser/non-DV/dismissed, or felony vs. all-misdemeanor). Charge matching is
    loose (subsection-tolerant); cases with multiple convictions were compared
    against the model's single most-likely conviction.
  • Limitations: small sample (31), single county, one run per model; results
    are directional, not a published statistic. Reproducible from our benchmark
    scripts.

Read more