Hacker News

The summary "There are still clear limits. Gemini 3.5 Flash remains a better practical choice [than GPT 5.6 Sol] for high-volume detection and counting in our benchmark, especially at its price." seems rather understated !

GPT 5.6 Sol was outperformed on all benchmarks by Gemini 3.5 Flash, apart from a single exception (OCR) where Fable was the winner.

Gemini 3.5 Flash not only outperformed GPT 5.6 Sol, but did so at 1/3 of the cost.

Anecdotal, opinion:

Gpt is really good in vision stuff, or at least their MoE seems to be really cohesive. From my experience Claude models can be really good at language but the moment they need to look at a picture and decide why the design is not good what parts need improvement it degrades a lot. My easiest benchmark is giving them a screenshot of a feature in my app and tell it "identify non-normative UI blocks and improve readability and consistency". Sol does a great job at re-structuring the page into composable units that build upon each other and the general looks and feels of the app. Claude tends to over-focus one one part while completely forgetting about the rest or the cohesion as a whole.

A better headline would be "Gemini 3.5 Flash is the best vision model". It tops almost every single benchmark shown in the article.

Penny sample shown looks like failed EXIF orientation registered by the model/harness. The coins are correctly marked, it's rotated 90 degrees.

It is funny to me seeing Sol used for what a "traditional" AI model can do already (counting pills).

We have vision models for our pharmacy and I could never imagine taking the latency hit to use a Sol in our robotics, it would be likely 25-50x slower.

Ironically, the pill counting example selected to showcase "the best vision model" can be easily solved with OpenCV template matching, a technology created 25 years ago.

Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.

In the third vision bench result, Sol is 100% correct but the expected has 1 error. Seems like an oversight.

In the next bench, Sol looks like it’s correct again but the bboxes are rotated 90 degrees for some reason.

Vision is still embarrassingly bad.

ChatGPT Pro with GPT 5.6-sol: https://chatgpt.com/share/6a834217-ca8c-83e8-a8e8-45d5b8797b...

The puzzle: https://activityvillage-files.s3.eu-west-2.amazonaws.com/s3f...

Dumb question: When your testing "ChatGPT 5.6 Sol" are you testing an actual LLM or some visual pre-processor stack that sits in front of it (along with a maybe a bunch of other such pre-processors) that is bundled into what's call "ChatGPT 5.6 Sol"? I.e. last I checked LLMs had a something like a 30-100K token alphabet to work with and it's hard to imagine how throwing pixels arrays at one directly would work.

From our experiments it’s the best video captioning model in the world by a mile. This was not the case a year ago.

When reasoning got introduced a year ago to GPT 5, on average the model performed worse than GPT4-o for short video clip captioning (Ie hallucinating actions that didn’t happen). The old GPT 5 was extremely finicky in terms of fps sample rate.

The other SOTA LLMs (like Gemini Pro) have clearly been optimized for long video understanding, since they can’t see almost anything sub-second (even if you up the frame sampling rate).

Sol is the first model we’ve seen to accurately caption complex sub-second movements (eg woman suddenly turns heard head to right). It’s robust to different fps sample rates so I can only guess that they trained on videos sampled at different fps.

It's not clear to me from the article, are they asking sol to output bounding box coordinates with some kind of structured outputs?

Anecdotal but I've seen it use python to crop, zoom, and "enhance" (fiddle with sharpness and brightness) images to read sections of handwritten census data from the 1800s. Feels like that there might just be a mismatch of capabilities when it comes to straight outputting coordinates but I bet the model is better at actually finding the answer given any tools available. Which I get is a bit of an apples and oranges situation.

I've also tried to use it to identify an old pair of glasses and it didn't stand a chance, so I do think it's not quite there yet when it comes to some vision tasks.

I currently have fable organize a bunch of 5.6 sol agents when working on my personal projects. This makes me wonder if I should add something along the lines of "For tasks that involve visual analysis, have gemini 3.7 look at images generated."

Overall I've been hooked on using agents from different companies for what they are best at (Thanks to Theo). Fable is expensive, but unmatched for planning and top level organization of other agents. Sol is fast, will persistantly go after goals (sometimes to its detriment), and does well with computer use.

I actually favor Qwen3.8 and run it locally + use the Token-Plan on AlibabaCloud, when I need faster results. Kind of favor it over GPT5.6 Sol.

Also it seems to be more capable, need to test more, but I think it's at least getting on par and it's fully open-source and open-weights.

Here's some benchmarks:

https://benchlm.ai/compare/gpt-5-6-sol-vs-qwen3-8-max

https://qwen.ai/blog?id=qwen3.8#full-benchmark-table (incredible UI/UX demos)

https://venturebeat.com/technology/qwen3-8-max-arrives-with-...

EDIT: Am I early to the discussion, or is none else using Qwen3.8-max?

I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?

In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.

So far I haven't seen a single model succeeding at transcribing sheet music, but I just tested it again with 5.6 Sol and it nailed the small test case. Fluently reading music requires multiple years of training for most people, but I feel like accurately following the horizontal lines trips up vision models in particular.

My anecdotal evidence says its still as blind as any other model, it has no taste, no attention to any sort of detail.

One of my friends (and BIL) own an architecture firm. They use AI to generate and quickly update renderings but they run into the equivalent of the 6 fingered hand problem. I sent him this article I wonder if the updated models can catch and fix mistakes made by previous models.

I wonder what about UI review, which model is the best?

I've decided it's "good enough" after I saw it properly quote a string of text that was very roughly highlighted within a nested visual context. It also identified the context correctly (modal inside webapp inside screenshot of user desktop).

I run the free service https://countrx.app/ so i have some idea what goes into counting.

The performance as a general model is indeed really impressive and i think they might actually win compared to fine tuned models.

Their feedback loop of training on user data is incredibly strong. I've learned that lots of accuracy results depends on threshold configs, which llms should be able to dynamically set.

Or the future will develop in llms using fine-tuned models as tools? Inference cost and speed does still seem to be below user expectations.

But for being able to one shot with this accuracy... IMPRESSIVE

I really like the combo 5.6 Luna & Sol for price and performance and would be perfectly happy if they stayed here for a moment without mucking about with sidegrades that I think AI evolution has often felt like lately.

It’s gotten so good that I now have infrastructure to render all mermaid/plantuml in my project to png and have AI’s always load both text and image versions. And they are instructed to review the rendering as part of the diagramming cycle (for layout, salience, usefulness, etc). They can now produce useful diagrams that help reach shared architecture understanding.

All of your use cases are very advanced.

I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.

I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.

I will add that all frontier models were fine with such tasks from the early 2024's.

"Best iPhone ever" vibes.

I would love more vision benchmarks! Once I asked the model to inspect a completely black picture and it hallucinated a nice wooden kitchen wall. Took me some time to figure out where the kitchen came from...

I usually go to https://arena.ai/leaderboard/vision/pareto for a nice overview of current models.

5.6 Sol looks nice, but the Gemini 3.5 Flash comparison is interesting. It’s cheaper and still came out ahead on detection and counting, which doesn't really give me much of a reason to use Sol since Flash is much cheaper and hence much easier to scale. Not to mention we now have 3.6 Flash too

One of the use cases I've wondered about for AI is giving it a picture of the "spice wall" in a grocery store and asking it to find all jars of e.g. cardamom. This takes me an annoyingly long time to do when I'm shopping, so it would actually be useful.

It's vision capabilities poisoned my cucumber bed, misidentifying the malaise and having me spray them down with water, which only spread the fungus that gemini later informed me was actual cause, which I went and checked myself.

I hope that whatever was lost at GDM in the last few months, didn't include their extra focus on vision capabilities.

Luna is pretty strong as well. been using it for projects the last two weeks and its strong

Seed Turbo 2.1 is incredibly detailed in describing every physical feature. I use that one for vision tool calls through Venice API.

I didn't expect Gemini 3.5 Flash to top basically every metric in this article.

Do you think we’re getting closer to models that actually understand what they’re seeing, or are they just getting really good at recognizing patterns?

Are any of these vision benchmarks binocular in order to introduce depth perception?

I keep waiting for these AI companies to assemble the parts into a great autonomous driving module.

Does any popular NVR make a good use of LLMs (especially local models) getting decent at vision?

I agree. It did very well on an extremely challenging task.

I asked it to recognize and draw the very faint reflection of what I was wearing, visible in only a tiny black part of a very brightly lit poster behind glass.

In addition, the poster itself also happened to contain similar clothing.

You can see the reference images and its output in my writeup here: https://medium.com/@rviragh/gpt-5-6-sol-very-good-image-reco...

While a human can focus on the reflection easily, this is an enormous challenge for a vision model. It's very impressive.

For the last 2 weeks I've been trying to get Codex to "outpaint" a wonderful image it generated as placeholder art for a level background.

After I increased the game's resolution, I asked it to increase the image's size while keeping the same scale and existing content, and gosh, it constantly keeps getting something wrong no matter what I tell it, even on Sol Max with the $100 Pro subscription.

An organically-grown meat-based pixel-artist could have recreated the image and more within 2-3 days, in exchange for food and shelter.

I hate these "The best X thing Y has ever released".

Unlike when Apple says "it's the best iphone we've ever made", LLMs are more or less interchangeable. So "OpenAI's best model" means nothing if "Anthropic wipes the floor with them" or "[open weights model] is 10x cheaper for 1% less quality".

As a reader, it feels like these titles are click bait.

the last image - it's barely visible to human eyes

> GPT 5.6 Sol is the best "vision" model OpenAI ever released

I mean I should hope so, as it is also the latest one

It's really quite good! I was amazed recently by its utter inability to read some faded handwritten cyrillic on the back of a wood carving - 3 or 4 words only, reasonably clear letter forms I found recently, and then stepped back a bit and thought about how insane that was as a benchmark - I just expect it to work so reliably on other OCR and translation tasks that it was surprising to encounter such a failure

Where are the Qwen benchmarks in this? I would be more interesting to see how Qwen performs.

Fuck ack. I'm working on a new benchmark that combines strong visual requirements with tool and coding requirements. I haven't even tested Sol yet, but between Sonnet, Terra & Luna I already see much better results from OpenAI's models. I'm not releasing anything yet as I still have issues in my harness that need to be fixed.

Am I the only one who cannot read the date on the blister pack even fully zoom in my phone?

If that is the full quality image given to the model, I think it's not surprising that the model confused with 03/2022.

I'm a little bit disappointed that vision seems to fall before language at scale.

It seems pretty counter intuitive that we can't do vision significantly better with specialized techniques.

Which is to say, still not ready for any production workloads yet. As in, it cannot reliably count the amount of objects in an image.

Still very impressive, but nowhere near the text chat revolution. OpenAI still trying to strike their second lightning

Still not quite as good as gemini.