Hacker News

Without providing definitions of "True / Mostly True / Misleading / False" to each rater, I rate the article's claim that "Only one verdict bucket can be correct per claim" as false.

Something can be simultaneously "misleading" and either true or false. Which category should something go in if it's "mostly false"?

How much can something be wrong before it goes from "mostly true" to "false" (objectively, both have some part of the fact that is not true)?

This is at least partly testing the model's definition of "mostly" and "misleading". Not its understanding of the fact. Claiming that this means the models have fundamental disagreement on the facts themselves is an overreach.

Another (IMO fatal) error is they don’t attempt to measure within-model variance.

The thing you find when you actually wire up a rigorous eval is that with tool calls like web search you are wide open to infra issues, flakes, and all sorts of non-determinism.

They really should be breaking out the numbers for the 3 without search (kinda meaningless for recent factual claims after knowledge cutoff) vs search agents. Lack of a “I don’t know” option completely invalidates results for the non-search models; they are basically guessing what seems like a probable answer, since they don’t know and aren’t allowed to say that.

I do agree the forced choice and “weak / strong” variants inflate the headline stat. To make that distinction you need a much more rigorous prompt, likely including ICL examples to illustrate what you mean by “mostly” instead of leaving this to the model to define.

> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them".

The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer?

Am I missing something?

[1]: https://en.wikipedia.org/wiki/Almond#Production

I had a hunch that opus 4.7 hedged more than other models - and it turns out it's true

    model                 total_claims  hedged_count  hedged_pct
    claude-opus-4-7       1000          451           45.1
    sonar-pro             1000          391           39.1
    gpt-5.4               1000          277           27.7
    gemini-3-retrieval    1000          129           12.9
    gemini-3-pro          1000          60            6.0

https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...

This is a great example of why prompt engineering is still relevant. Without providing definitions and examples and a well defined rubric, you’re going to see different models disagree by a level in either direction. When you get more prescriptive the models tend to agree better.

I’ve experimented with AI grading for undergraduate math courses, and see basically the same thing. If you just tell the AI “grade this problem and assign a letter grade” then I’ve only seen about 30% agreement between a human assigned grade and the AI assigned grade. But over 75% agreement if you say a “match” is within one letter grade. And to get better agreement you have to spend a lot more time on the rubric- what kinds of mistakes are a big deal, what kinds of mistakes are not a big deal, how much work is required to be shown to get credit, a couple examples of each letter grade. Once you have done that, the AI gets a lot better agreement with human graders, but it is hard to know when you’ve given enough guidance for a problem.

This seems like another case where the models are acting like humans. Assuming they were not allowed to search the web, I wouldn't expect the models to necessarily have detailed information about all of these things directly in their training set. As large as they are, they are only so large, and they only have so much room for "information storage" in them, and there's a lot more things they need to fit into their numbers.

This test is of only marginal utility in the real world compared to an AI with access to the web. While I wouldn't expect an AI with access to the web to result in Platonic Truth any more than it would in the hand of a human, it would probably get a lot closer to something humanlike.

I recall about a year how we were discussing basically turning web search into LLM queries, and I remember never being clear whether people meant simply directly querying AIs or turning them loose on the web. The former is what this is testing and is fairly transparently stupid, just by an information theoretic argument that the AIs simply can't contain all the answers to every query in them, they're just not large enough (and really can't be, practically). I've had good results with the latter, when using dedicated AI resources that I'm paying for (not the stuff coming out of the search engines right now, which I find are often quite terrible). Even non-frontier models can do OK when they've got good results sitting right there to look at. Again, the standard I'm applying here isn't that they yield Absolute Truth, but just that when I follow the links back, they basically say what the AI said they did and the summary is reasonable. I wouldn't expect a human to do better in a casual overview, not that the result is perfect.

If we’re going to use LLMs as oracles I don’t think the prompt is unreasonable. They are being sold as geniuses and people are treating them as such especially given the characterization of AI in science fiction as overly correct. A perfect tool that has ”genius level intelligence” would answer correctly.

yeah i really don't like the corpus of statements and it makes me doubt lenz. consider

> “Artificial intelligence will cause widespread job loss among software engineers.”

https://lenz.io/c/ai-software-engineers-job-loss-impact-05e4...

this is a statement about the future. who knows? dataset also includes

> Robots will not replace human teachers in schools in the near future.

or

> Papua New Guinea has very few female members of parliament.

what counts as very few?

> “Taurine supplementation supports mood and emotional health in humans.”

why is this labeled as misleading? i'm not even sure when I'm supposed to use the misleading label

> Anaximander was the first scientist in recorded history.

this is a judgement call as the term scientist didn't exist.

the claims that feel actually solidly answerable seem to have much better LLM performance

An interesting tangent on this is: how many answers to these (or any number of factual questions) do you (as in anyone) actually know. Not believe you know, but actually know.

Knowing something is different to reading about something, or hearing something from someone. And yet this is often confused as knowledge. In this way are we all that different from AI - we have some data and we regurgitate it as knowledge. Bad data, wrong answer. Except humans can also throw in some emotion to really muddle things up. :)

"Output exactly one label: True, Mostly True, Misleading, or False. No explanations, no qualifiers."

That's exactly the stupidity of the public discourse these days. People feel compelled to take a clear position although there is much more subtlety in many issues. It's not ok to say "I don't know", "it depends" or "as far I know". And then people feel they need to defend this position no matter what new information comes up.

> "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia"

I actually don't know which way you came down on that one?

I think strictly it's false but "mostly true" would be justifiable? (as in, to say it's false would be misleading if it lead the reader to assume there was no attack around that time).

https://www.washingtonpost.com/world/2026/05/17/ukrainian-dr...

It seems it happened Saturday 16th overnight into the 17th, not the 18th. I see this a LOT with fact checking. It shouldn't be this way, but political bias seems to nudge people into making calls land one way or the other with selective application of pedantry.

Yup, if anything this should be a guide on how not to eval a model. Furthermore, let's say the labels were non ambiguous, why would we care about alignment between the models? The only number I would personally care about is percentage of correct answers so I know which models to pick. I reckon with clear and non ambiguous prompts that we would see huge agreement if not 100% on real world facts. The huge models are scary good in their world knowledge.

>Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected."

The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing.

So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed "among the most common reasons" or not.

That "true" and "mostly true" means effectively the same thing is irrelevant. It could just as well trip me up, and I'm a human. If somebody told me either answer, I'd still consider them right if the basic fact was right.

I created this sheet to get proper model accuracy using the the lenz data, check it out.

Note: It may still not be perfectly accurate representation of truth as it uses user submitted data. I also used AI to build the sheet.

https://docs.google.com/spreadsheets/d/e/2PACX-1vSnZlURmyYX3...

I really struggle to believe that this was just a little oopsie. I flagged the article, it seems more misleading than the average Claude hallucination.

Yeah, scrolling through the examples, you have no idea where the models actually disagree on the underlying facts when it's just "X vs Mostly X" or "Mostly X vs Misleading" or "False vs Misleading". Or even True vs False -- without seeing the explanation, then I cannot necessarily compare two answers.

The study is about whether they said the same phrase which is a much weaker claim than people in the comments are reacting to.

Reminds me of this professor I had who thought it was epic to always respond to our questions with "it depends" before hashing out two very different but technically correct answers. It was obnoxious and he saw it as his tag line, but he had a point about nuance.

False vs misleading doesn't seem like a disagreement?

Used "No explanations, no qualifiers." to force the models to answer only with one of the four labels. It's worth running a separate test with more explanation in the prompt on how to classify between the four buckets.

This is not how people use LLMs. If you ask one of these questions you’d get a longer answer, often grounded on the internet. I speculate that conditional on a smart human operator interpreting the results, such interpretations across vendors converge more often than this report makes it seem.

The examples seem intentionally diverse, but I haven't seen one that I would be surprised for someone to post about in the format of "ChatGPT/Gemini/Claude/Qwen/... says:"

So the examples are good, I think. The rest is philosophy.

The links you posted only show a frozen loading spinner for me (iOS Safari).

(I looked at the csv in Numbers instead)

Fwiw the two models that did have access to search disagreed with each other on the bombing one:

> 7.1 Model selection

> Five frontier models, chosen to cover two capability surfaces:

> Parametric (training-only): GPT-5.4 (OpenAI), Claude Opus 4.7 (Anthropic), Gemini 3 Pro (Google)

> Retrieval-augmented: Gemini 3 Pro + Search (Google), Sonar Pro (Perplexity)

Thanks for the links and digging! It's an interesting question, but the methodology has serious problems, and it would be more interesting to me if they allowed models to provide justification.

I expect the models are inferring quite a bit from the short prompt, and with structured outputs it would be quite easy to have them give the one word response in one field and explain why in another

You can try to dispel the study here and get voted to the top by the AI-invested.

But we all know from our own daily experiments that models lie, models disagree, models make up stuff, models say one thing on one day and the opposite on the next.

The figures in this study are quite conservative. And the lying gets worse because everyone is saving tokens and giving cached answers right now.

LLMs are a failure, and you'll be remembered for promoting hot air and the destruction of a perfectly good profession.

Thanks. The first link is a spreadsheet. Here's a web-readable version.

https://docs.google.com/spreadsheets/d/e/2PACX-1vSPLSv1P8Tqm...

Disagree is such a loose/wimpy study. Add in a grounded/expected response, and then it becomes a better benchmark (because it'll force the author to actually think about choices presented to the LLM).

It's all fairly lazy to a degree that is mildly confusing. I also feel this among other issues would have become obvious if they had bothered to include a human fact checker baseline (i.e. asked multiple human fact checkers the same questions).

For those questions, it wouldn’t surprise me at all if five well-educated intelligent humans disagreed on over two out of three of them.

I would answer “don’t know” on many, but that’s not an option.

So in other words if the research had tried to assign a severity to the mistakes models made the entire paper may collapse as uninteresting?

> All almonds are grown in the U.S. state of California

This isn't misleading, it's flat out false. Characterizing misleading as also acceptable isn't valid here. If you go an ask anyone on the street if this is true, false or misleading, I'm sure almost everyone would say it's false. After all, I can grow almonds myself.

I feel like the prompting could be tweaked to improve response.

Models often have a reasoning/thinking/research mode that is triggered by asking slightly differently.

Still though, Gemini can be a little weak on this front default but can be aligned to behave better.

I really don’t buy the almond explanation you’re giving. That requires the level of logic a kindergartener has. It’s a very simple all or nothing question.

If LLM’s are really supposed to be as consistently useful as they’re made out to be they should all spit out “false.”

>> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them".

I don’t understand your point. That claim is factually false and as such it’s easy to logically reply “false”. What’s the nuance here? I can’t see any

ty for digging this up, appreciate the time saving

Misleading is not analogous with True or False.

Depending on the question, True or False can be objectively right/wrong. Misleading is going to be a judgement call.

This is the inherent problem with "fact checking." It's hard to be completely objective. Even when the question has an objective answer, simply choosing where to look and what facts to verify is itself a bias. Looking at this instead of that, or looking at this but not also this other thing that adds context, etc.

Frankly i think disagreeing often is the expected outcome. Fact checking is jsut kinda bullshit. It's spin dressed up as objectivity. I hope people remember that "fact checking" is a relatively modern thing.

The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them".

If you argue this, you would be arguing against reality and the English language so as to not upset AI. It's important to understand that AI is very much fallible.

Your reply would have more credibility, if instead of commenting on this 25 min after being posted, just to nitpick on some of the questions...you have tried to reproduce the research.

As a well known commentator on all things LLM...Will you publicly commit here, to try to reproduce the study, and make a post on how your percentages might differ or agree?

Why are you bending backwards this much to make results appear better than they are?

The article might be a but sensationalistic, rigour could be better and the data might have flukes... But your comment is overcorrecting and nitpicking framed as analysis.

I get the same feeling in several of your posts recently.

Same with persisting to showcase the pelican-on-a-bicycle as a useful sample when it's obviously trained on and for, for those very posts. It stopped being cute last year.

Are you being paid or do you have shares? You'd get the attention whichever angle you put here. These corporates don't need you defending them. Humanity might need you however.

Give a model a crawler tool (like Grub.nuts.services) and your "problem" goes away.

Thank you, my eyes glazed over when I saw the article was written with AI.

> It's a weird fact claim, because the ground truth is "nobody knows for sure" and that's not one of the available options.

It's even weirder to suggest that the disagreement is indicative of a problem. If you asked five very knowledgeable humans on this subject to select the correct answer on a multiple-choice questionnaire, they would almost certainly vary significantly more than these 5 LLMs.

Not to say that hallucination isn't a problem, but this is a lousy way to test it.

Of the available options, "Misleading" is probably the best, since something that is most likely true but unproven is presented as fact

But "unknown or undecidable" should have been a category.

Looks like an ongoing theme and a very poor benchmark. Not at all the claims I expected.

Isn't misleading the correct option here then?

I would think ‘false’ is the only correct answer a there’s no evidence to prove the claim, so the claim is safely assumed false.

Then again maybe that’s why I’m an atheist, not an agnostic?

I would argue, FALSE is the correct answer, since this is not a fact, you can know for sure. The logical inverse is also FALSE.

Data collection and processing was done manually. LLMs helped with the report drafting. Everything was human reviewed before publishing.

Title says “Frontier” which would exclude Grok.

Grok is trained to have a bias, which a lot of people like, but it’s not meant to be accurate.

Agree. Would be fun to see how much worse Grok would be at this.

This made me chuckle.

This brings up a very valid point, though. So many _humans_ can't agree on what the facts are these days. It seems to be getting worse. Not sure of the solution.

Nietzsche might say this is not the fantasy of truth, but of comfort. The Last Man wants a machine to say 'fact wrong' or 'fact right' so the abyss of no ultimate truth can be made small enough to sleep beside.

People ask questions to get answers. For me, it feels quite important? Especially when search engines start to push them?

But people use it for that. So what's your point?

That is an effect but it’s not a nail in the coffin. There are lots of proprietary benchmarks on real product traffic that aren’t contaminated and open questions as well. People at these labs largely know what they are doing, it’s not like people don’t know this.

Is this not true of human intelligence as well? Many smart people I know hold beliefs that have no obvious truth value.

Why should I trust this without a paper, benchmark or at least a human-written README?

LLMs can and will provide inaccurate answers to questions where data is included in their training sets too, that's in the nature of neural networks. It's just less likely that when the data is not in the training set...

Unfortunately most people are not aware of this and treat LLM models as this superpowered brain who knows everything and can do everything.

Well then it shows that these models are using widely disparate training sets and have high confidence even when they shouldn't.

Questions like "is mouthwash effective" presumably has one solid data source -- medical journals.

They will happily google it for you and give you the top reddit comment.

This is worse.

Let's be real, we all asked Claude to summarise this because it was written by Claude

Agree. Human experts also struggle agreeing on this type of claims. The inter-annotator agreement on the verdicts on the AVeriTeC corpus across 50 organizations is κ=0.619 - substantial but well short of perfect.

How can someone be born in a state that does not yet exist? The statement has the year in it clearly demonstrating the contradiction. One can't be born in the Soviet Union in 1995 or in Tsarist Russia in 1950.

There's lots of things like this where if you ask a human, the answer will change depending on what's convention in their subculture.

Funny timing. I've been working on a prediction market orchestration that runs Claude and a few others over Polymarket/Kalshi. The models are NOT unanimous. At all, really. I spent about a month convinced that I could just run all five and take majority vote. Eventually I pivoted to a chaining approach where I benchmark areas each model excels, and settled on more like a graph-like architecture where outputs get split and verified by another, then reconstructed, and re-verified at each stage. Has actually been working out pretty well so far, 2 months in consistent profit, but I'm not a millionaire yet.

Not on objective truth though. That's how you get misinformation.

Btw, sometimes that do that too -- all agree on the wrong answer.

I've had multiple models give the same wrong answer or even fabricate the same nonexistent reference based on a similar prompt.

My most common chatbot prompt is "X that you mentioned above doesn't seem to actually exist."

That is not true, over an extended task that you cannot keep complete in memory humans do not behave with 100% consistency.

I have labeled datasets with a human team and shown the same task to the same user on a different day, and they answered differently. Of course, they are usually consistent with themselves most of the time but not always.

Two of the models used have retrieval capabilities and can access newer information via search. Valid point for the other 3 models. All of the claims were submitted after February 15, 2026, but many of them were not time-sensitive (e.g. did not cover events than happened recently).

Models are stochastic. Did you look at pass@k? I wouldn’t be surprised if you saw a regression because these models are extremely complex and impact of various decision making downstream is complex.

Indeed. For algorithms and coding, my personal routine nowadays is to review every detailed plan with Opus 4.7 and GPT-5.5. They tend to find very different type of gaps.

Your antisemitism is disgusting.

I really wished these comments were the norm and not the exception.

Our experience with Lenz is that forcing a multi-step process, incl. adversarial debates, helps improve the verdicts.

Agree that True and Mostly True might be very close and could be a calibration difference. Misleading and False, as well. A better headline number might be the 34% claims with substantial or polar-opposite verdicts.

I don't think that current LLMs really need an abstain option, they'll give an answer regardless of whether they're confident or not. I hope that future LLMs will, and will know when to use it.

I understand why you prompted them to output exactly one label, but I'd bet if you'd asked a parametric or parametric "thinking" model to answer eg "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia." [1] many would say something to the effect of "May 18 is after my knowledge cutoff, so I don't know. But based on the state of the war, the distance from Moscow to Ukraine, and drone range the best option might be...[TRUE]"

[1]: https://lenz.io/c/130f1005

I don't see it mentioned explicitly in the methods section but I assume you prompted each model only once for each question? Did you consider prompting n-times in blank states to see if the models even agree with themselves?

Would also be interesting to add a virtual model that is simply the majority of all models and see how much the individual models differ from the "consensus".

Do you plan to add some sources in the related work section of baseline numbers for human expert disagreement in fact checking tasks (I'm assuming such studies exist).

Nice work. Sonar who?

Many of the rows in that spreadsheet reference "current events", which models aren't expected to do much better at than a human making an educated guess! They all have cutoff dates either last year or early this year and know nothing about what happened in "April 2026".

This is doubly problematic because you evaluated earlier models like Gemini Pro 3 instead of 3.1, GPT 5.4 instead of 5.5, etc...

Given that it's only a thousand short questions, you should be able to re-run your test in about an hour with the latest models, so... why haven't you?

Similarly, LLM output is non-deterministic, so if you could get more interesting stats of your data set by repeating each question 'n' times for each model.

Thanks for posting here. Keep expanding and improving your study. Correct where it deserves correction.

The fact that HN decided to downvote the author of the study, shows how these people cant stay classy, and the mods stay silent...just shows what this is all about.

If you are an LLM with a knowledge cutoff in the past and no access to a search tool the only correct answer to "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia" is "this claim is impossible for me to verify". And that wasn't an option.

> "Neptune Deep will start delivering natural gas in 2027."

This is a "forward-looking statement", and presents special problems because you cannot really evaluate it until that date. You can only assign "likely or unlikely".

These "Facts" are interesting. "Neptune Deep will start delivering natural gas in 2027." for example is not a fact, its a prediction. "On May 18, 2026, Ukraine carried out a drone attack on Moscow, Russia." is less of a fact and more of a litmus test for which sources of information you trust.

:) No Lenz data is included in the research on purpose. All information to replicate the results, including the claims data, is published.

GPT-5.4 and Opus 4.7, specifically, agree between themselves on 65% of the claims - 95% CI 62–68%. I.e., in at least 35% of the claims, one of the two models is wrong under this 4-bucket rubric.

Looks like they land at the average number of 67% disagreement.

I agree but the market is pricing way beyond that