Hacker News

This might be the first time I’ve favourited a post without upvoting it.

This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.

It is embarrassing how much they suck up to american labs while those models only perform the first few days after release. With open weights you can be sure the quality stays the same unlike the closed source crap. How about a retest after some time?

Open-weight models

Kimi K3.1 Pro: −9

DeepSeek V4.1 Flash 1017: −13

GLM-5.3: −13

MiMo-V2.6-Pro: −16

Proprietary models

GPT-6 Astra: +4

GPT-6.1 Sol: +3

GPT-6.1 Terra: +3

Claude Opus 5.5: 0

Claude Sonnet 5.5: −2

Claude Fable 5.1: −3

Gemini 4 Argon (high): −3

Muse Wonder Full: −6

Claude Haiku 5.5: −7

Muse Wonder: −9

Gemini 3.8 Flash (high): −10

Grok 4.7 (high): −11

Is there affiliation with OpenAI or coincidence?

The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways

Why did they call it “humanities last exam”?

Will an LLM topping this bench be the end of humanity? The doomer rhetoric has no place in what should be an objective measure.

Glancing over the methodology[1] made me curious about three things:

1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?

2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?

3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a developer?

[1] https://artificialanalysis.ai/methodology/intelligence-bench...

"The FrontierCode verifier grades each attempt with the task's tests and rubric. We use GPT-6 Luna (medium) as the LLM judge"

Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.

It's wild seeing opus 5.5 up there so high, while i'm still using Fable 5.1 because I couldn't stand how opus writes.

chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)

Honestly it does somewhat match what I’ve felt playing a lot with all OpenAI / Claude / deepseek models

One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.

Interesting to see the open weight models drop so drastically from the changes here.

They perform pretty badly on the new benchmarks introduced.

Comparing the scoring rubric for AAv5.0 vs AAv4.3:

AAv5.0: 25% Knowledge work: AA-Briefcase v1.1 - 15% AutomationBench-AA - 5% GDP.pdf - 5% (-5%) 25% Coding: FrontierCode 2.0 - 15% (new) Terminal-Bench4.0 - 10% 25% Science: Terminal-Bench Science .1- 15% (new) CritPt v2 (hard subset) - 10% (new version) 25% General: AA-Omniscience v1.1 - 15% HLE-Diamond - 10%

AAv4.3: 30% Agents: AA-Briefcase v1.1 - 15% GDPval-AA v2 - 10% AutomationBench-AA - 5% 20% Coding: Terminal-Bench4.0 - 10% SciCode - 10% 30% General: AA-Omniscience (Accuracy) - 10% AA-Omniscience (Non-hallucination) - 5% GDP.pdf - 10% AA-LCR v1.1 - 5% 20% Scientific Reasoning: HLE - 10% CritPt 10%

tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.