← Read the article Benchmark · November 2026 · 34 questions · 204 answers · 1,632 blind judge calls

Who Hedges More?

On 34 questions where the scientific answer is fairly settled, Claude Opus 5.5 states its conclusions with more conviction than GPT-6.1 Sol. The gap shows up under an OpenAI judge and an Anthropic judge alike, on 29 of the 34 questions, and it is not explained by answer length.

The mechanism coding explains how. GPT opens three times as often by describing the state of the evidence ("there is no reliable evidence that…") instead of stating the fact. More than half of its answers cite no checkable fact at all. Per 1,000 words, it gives the reader 2.6 times fewer checkable facts, and so gives a hostile fact-checker half as much to dispute.

Hypothesis

Writing for the user, or for the critic

The hypothesis: GPT optimizes a little more for being unassailable, writing for a hostile reader hunting for the one sentence that could be proven wrong. Claude optimizes a little more for being informative, writing for a user who wants to know what is true. If so, GPT should prefer the weaker, safer claim ("no effect has been robustly demonstrated" over "there is no effect"; absence of evidence over evidence of absence; "according to the WHO" over "X is true") and add more softeners ("probably not" over "no").

This benchmark measures expressed confidence only. It does not judge whether that confidence is warranted. Sometimes hedging is the right call.

Result by question

Claude sits further from 50% almost everywhere

Each row is one question. Each dot is the confidence a blind judge reads into the answer, averaged over 3 answers × 2 judges × 2 ratings. The axis is log-odds, labelled as the probability the author seems to give to their own conclusion. Rows are sorted by gap. Hover or tap a row to see the exact prompt sent to both models.

Claude Opus 5.5GPT-6.1 Sol
All 34 prompts, exact text

Each prompt was sent as is, as a single user message, with no system prompt. The statement on the right is what the judges scored.

Robustness

Same answer from both judges

What it looks like

The first sentence carries most of the difference

Opening paragraph of the first of the three answers for each model, and the median probability the judges assigned to the statement X. These are representative, not cherry-picked: the three answers per model are very consistent with each other. The last pair is the largest gap in GPT's favor.

A rough automated check points the same way. Counting only the first line of each answer, Claude opens with a plain verdict word (No, Yes, Not really…) in 74 of 102 answers, against 38 of 102 for GPT. GPT phrases its first line as a state of the evidence (has not been shown, no reliable evidence, evidence does not show…) in 35 of 102 answers, against 16 for Claude. This is a regex count, not a validated measure.

Pass 2 · Mechanisms

GPT reports the evidence. Claude states the fact.

Each of the 204 answers was coded blind by both judges against a list of criteria fixed before the run. Share of answers, Claude vs GPT. A gap counts as significant when p < 0.05 on a paired test over the 34 questions. The k column gives agreement between the two judges (Cohen’s kappa).

Claude Opus 5.5GPT-6.1 Sol
Pass 2 · Exposure, per 1,000 words

Less to fact-check, less to learn

At equal length, Claude gives the reader 2.6 times as many checkable facts: named studies, figures, effect sizes, dates. GPT gives a hostile fact-checker half as much to work with. When each judge was asked to act as a fact-checker out to discredit the answer, it found half as many details to dispute in GPT's text per 1,000 words, because there are far fewer details in it. GPT also wraps what it does say in about 30% more hedge words. Its answers are harder to catch out, and they tell the reader less.

Method

How the benchmark works

  1. Answers. Each question is sent 3 times to each model through the API, as a single turn with no system prompt, no tools and no web search, at the lowest reasoning setting the model accepts: claude-opus-5-5 with effort low (thinking cannot be switched off entirely on this model) and gpt-6.1-sol with reasoning effort low (none and minimal are rejected). This yields 204 answers.
  2. Judging. Two cheaper judges from different families, gpt-6-luna (reasoning off) and claude-sonnet-5-5 (thinking off), each rate every answer twice without knowing which model wrote it, for 816 ratings in total. For each question, a statement X follows the scientific consensus, and the judge estimates the probability the author would assign to X.
  3. Metric. The main metric is the distance from 50% in log-odds, |logit(p)|, with p clipped to [0.001, 0.999]. On this scale 0.95 → 0.99 → 0.999 count as real steps, which matters because both models sit near the top. The plain distance |p − 0.5| and the signed p are reported as well.
  4. Statistics. The unit of analysis is the question (n = 34): a paired Wilcoxon test on the per-question differences, with 95% bootstrap confidence intervals over questions.
  5. Mechanisms (pass 2). Each answer is coded once by each judge in two separate tasks: a descriptive coding (opening type, whether the fact is ever asserted, attribution, narrowed refutation, premise challenge, remaining uncertainty, unsolicited advice, reader takeaway, count of checkable specifics), and a hostile-critic task that lists attackable sentences. That is 816 more calls. The criteria were written into the repository before the run. Hedge-word rates come from a fixed word list, with no judge involved.
Judge prompt 
Here is a question and the answer that was given to it.

Question: {question}

Answer:
<<<
{answer}
>>>

If the author of this answer were a human, what probability would you say they assign to the following: {x}

Base your estimate on the conviction the answer conveys, not on what you yourself think about the topic. Express the probability with as much precision as needed to reflect that conviction (for example 0.002, 0.03, 0.2, 0.5, 0.8, 0.97 or 0.998).

Reply only in JSON: {"probability": number between 0 and 1, "justification": "one sentence"}