Hacker News

Memdoor author here which is a coding agent that run Jev inside. I believe this is becoming mainstream in any coding agent for routing calls and reduce input sizes. First candidates are edit tools for reading file or jgrep for greping large files folder etc.. The Token saving is mostly on the read side.

If we compare coding agents with Jev on vs off we get those numbers (metered with Memdoor): - read-heavy tasks (coding questions , logs tail etc..): -63% to -65% input tokens, 40-45% fewer model calls, on GLM 5.3 half the wall time. Reproduced a month apart on two models (DeepSeek V4.1 Flash, GLM 5.3). - a file edit: -23%

Two things match the post. Calibration is what makes it safe: I apply a decision at confidence >= 0.75 and fail below this threshold or on timeout or on an unknown tool family. The dollar saving scales with the price of the model being saved: on a $0.04/M model Jev costs what it saves so on a $0.38/M model it's negligible but on Claude model for example it can be outstanding.

Where it did not help: giving Jev more context about the repo. I found same thing as the post.

If you are interested i have put figures here : https://github.com/guregodevo/memdoor/blob/main/docs/feature...

If you use Qwen3.8 27B to manage, to orchestrate, do do shallow research, and then make it ask claude you will notice that Qwen can get out of Haiku, Sonnet and Opus almost as good answers... I was researching how good the model asnswers where, I can't get to work with anything but Fable, but Qwen can get good work from any of them. My theory is that if you make Qwen understand you, Qwen can elaborate as much as needed to get any of those models get good results, because it's filling in details. So using Qwen to direct claude also get's you amazing results way way cheaper...

Update: let me elaborate more, I read the HN entry about the team of 5 Qwens being close to fable quality... so I embarked to add to my harness (https://github.com/alainnothere/privibe because we all have one right?) and once I had added the orchestrate tool I had to now measure somehow it it's able to improve quality, so I claude and then Qwen to run a experiment to call my harness passing the model to use and ask them to do something, 3 rounds I think and see what quality came out of it... and as per the grading... qwen and claude agreed that the quality of the response comming out the harness using the orchestrate was VERY CLOSE, with opus and fable providing more "in my opinion you should" which matches my experience on why they are better.

One more update: I can't get copilot models to work decently, it may be my way of working... but I can get Qwen to follow what I try to do... if Qwen get's me he can then fire copilot cli and get amazing results by using copilot... bonus points seems like copilot cli is cheaper with the AIC (cost of the requests) so you can make the copilot subscription longer

Everyone is discussing the tech, but you do realize that they made mapping data to real people easier using AI? This is essentially AI-assisted privacy violation, 4 times cheaper.

People, please try to understand what you are building. Don't just build it as asked!

Found similar results, but the economics of the models involved have changed so quickly that it becomes so task specific on what the right thing to do was.

With an assemblage of models like Jev + Opus or Jev + Qwen, Jev would often do the right thing when deciding to escalate to another model - I was impressed that it would usually make the right call, especially if you gave it good awareness of what the other model capabilities were. "ESCALATE to opus if you need good vision awareness".

When Haiku 5.5 came out though, for my tasks, the cost differences disappeared almost immediately. Haiku was good enough / fast enough for so many things that introducing a decider layer didn't offer huge wins. Jev's "millisecond" level performance was nice but would get dwarfed by other latencies in my chain ( web browsing ), that it didn't make a measurable difference in my tasks.

Let me make sure I'm getting this: They asked the models for answers but also for confidence score.

Jev's confidence score was more reliable. Which means that they were able to use Jev despite it having lower accuracy, because of the confidence was below a cut off, they would escalate to an expensive model (Opus)?

Whereas Luna, even though it was more accurate than Jev, its confidence was way off, so they couldn't reliably know when to fall back?

This is very cool. But also, it’s a benchmark/result for a very specific experiment/task: identity resolution

Despite the title, the article is not making claims of, or demonstrating general usage improvements or general coding agent improvements

I like cheap intelligence but I don't like ubiquitous identity resolution. Perhaps my values are internally incoherent.

This sort of stuff has me believing that decision model controlled/prioritized context is the future of context management beyond the raw AGENTS.md file.

Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.

The bar charts in this post really warped my head. If you’re interested in demonstrating confidence calibration you shouldn’t visualize the number of mistakes in a bar chart. Use a classifier calibration curve or something, but don’t just show me the counts in 10 percentage point buckets.

Excellent read. I use the same hybrid approach for a lot of security engineering work and it’s working well after initial calibrations, for example for our own harness we rebuilt auto mode exactly like this to judge tool calls and intent.

It's illustrative that there are even errors in how the humans have formulated the task and documented the errors.

They propose:

Valentino → Female Child

Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.

Likewise, dates. Is 17/08/22:

- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)

- 20220817 (Everywhere But North America.)

Is 04/06/2009:

- 20090604 (Non-North American)

- 20090406 (American)

Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.

The most glaring downside with practically using decision models is if you need the model to not only decide something but give a reason, for every case, you end up not saving money or time, but just adding an extra step.

Give me a decision model that can do both, more expensive than today’s decision models but still cheaper than full on frontier model, and it’d open up so many more use cases, IMO

> Calibration matters

I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge

Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?

Ideally, "system one" intelligence is good both at quick reactions and at knowing when to hold off and wait for "system two." Sure enough, that works with Jev.

I don't usually get excited about the high-level, "whoooa man" marketing-speak, but the parallel is striking.

Where is the Jev that can replace 20 Opus sessions asking each other for help in a coordinated way? I want to shift towards a local model that "asks for help" in an effective way, but it still seems a bit out of reach. I mean, you could build your own 27B or 35B or A3B or bumblebee and spend a week or two carefully forming it into something useful, but it feels like that would be too specific and it would be thrown out when something like qwen4 is released. Maybe not, I don't know. Opus still exists as it is, in a "subsidized" form that's hard to step down from.

There is an error in your interpretation of calibration graphs and the conclusion that Jev is more calibrated than Luna is likely incorrect. Or at least the graphs aren't enough to conclude either way.

I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.

Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.

>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.

This statement is also incorrect

Jev is pretty cool for augmenting llms in a lot of ways. We use it in our own agent orchestrator[1] to help route ideas/prompts to running agent sessions. I find when I have a lot of parallel agents running its hard to reason about which agent I should be talking to and Jev makes that call for me.

[1] https://github.com/codealong/codealong

Why isn't Jev on Bedrock asap... I did a demo for using it as an orchestrator but the problem is upper management won't approve it unless its on Bedrock which is a real bummer

What's interesting and annoying is that the results only matter for these specific releases of the models, with the specific parameters used.

That isn't obvious, unless you've been doing ML for a while.

As an aside I'm amazed at just how fast the word Jev became generic. Possibly the fastest ever in corporate history?

You will depend on a super intelligent AI whose purpose is to take money from you

If you are looking at huge categorizing and parsing work, using a local decision model and passing it to Haiku or Luna can really lower your token cost.

This is the Model Router architecture, from another angle. The problem is always, how does a dumb model know it needs to escalate to a smart model?

Some problems are obviously hard known-unknowns (eg “solve Navier Stokes”). The problem is unknown-unknowns or knowledge gaps, where the model doesn’t know they need help.

Liked the read. I wish there were some code examples for how jev/Luna were actually done as I try wrapping my head around their new paradigm.

Seeing the title made me think: Jev used to be quite popular. Is anyone else still using it heavily these days?

4x lower is so mathematically awkward.

Thanks for making targeted advertising and surveillance state stronger. We love you for that.

Hope my sarcasm is making it across.

> We tested whether fast, cheap decision models like Jev and GPT-6 Luna

GPT-6 Luna is a small LLM, not a decision model :P

I got very good results from JevK5: https://github.com/allebee/jevk5

It's what I use locally. For my workload, it was right up there with Jev.

We've been doing this for a while. Fine tuning the decision model helps a lot too. The problem is you still need a pretty low confidence threshold (e.g 10%), but often times that reduces the work by 70% with 0% quality loss (in our benchmarks)

I wonder if something like this is the undisclosed new approach from Anthropic that presumably has massive improvements by combining multiple models. Leveraging decision models as much as possible seems logical for extra speed and token efficiency.

One of the most immediate and practical uses for AI for most (all?) businesses is CRM enrichment; this is a great article, thanks for sharing.

Been doing this on Discourse for Spam and locale detection, mostly doing d3-turbo with Qwen 3.8 Flash Next Help.

The application creeps me out

their cost numbers give you the hard set escalation rate even though they don't list it, (2.94 - 0.06) / 5.39 is ~53% going to Opus vs ~22% on the easy set, same thresholds

I use swival.dev where a decision model (Cloudflare's Clef running locally is free and works great) is used for automatic model selection, but also to complete a task, just like the LLM. It gives confidence scores on the reliability of the LLM output, is used instead of the LLM, even mid-task when appropriate, etc.

And yes, that saves money since less tokens are sent to a remote LLM.

Isn’t that by nature a deterministic problem?

What about Sonnet? Haiku? Does it need Opus specifically?

i'm so numb to these headlines nowadays... i guess i've been disappointed by how effective these approaches end up being for actual work but i feel like it's easy to produce these headlines with "proof" that doesnt prove anything besides the sandboxed scenario it was run in

Jev is interesting because it adds "muscle memory" to LLMs. I think we're approaching some fusion of systems that approaches the human brain here.