Hacker News

Every major AI shop has a ton of in-house classifiers already, big, small, generalist, specialized. Some are used in inference pipelines (e.g. safeguards), some are used in data preparation, training, analysis and investigation, research, various one-off and intermediate tasks etc. Offering them on a public API doesn't always make business sense. I don't see much substance to this buzz, looks like people that are new to all this are discovering that classifiers exist, they are more efficient at classification, and many tasks commonly done with generative models are classification in disguise. Which is not bad at all, a fresh look at their use is great to have.

This article is extraordinarily hard to read. It’s tummelvisioned on OpenAI and things like tool calling which are only relevant to the extent that llms have been tuned to make relative choices, but this applies to all LLMs. Also, some really outdated references. LLM written, perhaps?

Also, moat discussion is the lowest form of discussion. I don’t care if jev has a moat. Did it get the interface right? What other past ideas have we overlooked that if given some love, could kick the door down like jev did?

Really silly stuff.. people wanting to talk about moats when there’s no castle. Moat talk merely projects the illusion of being engaged but, much more often than not, it’s hollow engagement.

I find it unlikely. OpenAI is all in training models with reasoning with RL, and Jev-like models are the total opposite. They are made to not reason at all to be fast. If you want to add reasoning on top, you might as well use a conventional LLM because you lose the price and speed benefits when you output auto-regressive tokens. I don't think OpenAI will even bother with this.

> My main assumption is that Jev is using something quite close to a conventional large language model. As evidence of this, Latent Space reports that many of the early clones are indeed LLM-based.

Not proof that this is the case with Jev though. It might use non causal text encoder for the state, which could make sense given that it's very good for its price.

There is one benefit that Jev has, that it is not OpenAI and thus it's probably less likely to steal your own work.

Why would OAI need to follow Jev? I really think this is paid by Jev. Jev itself won't have lunch money in a shortwhile because there are literally 10s of free alternatives available which can be run locally on basic consumer hardware. Terrible utility aside, there is no sensible business proposition in Jev.

For context on what "Jev" is: https://news.ycombinator.com/item?id=49717558

If OpenAI releases something similar to what Jev does, then that would be like admitting defeat. Their whole spin is AGI and world ending danger. Why would somebody with an AGI at home make something like Jev which is intended to be a part of some SW the AGI is going to replace anyway.

I'm not exactly following through with the claim, can someone explain how the built-in classification would not necessitate more tokens used, or be much different from turning on reasoning? Not that I don't see the difference, I just doing see how OpenAI would do it well.

Rather than focusing on OpenAI in particular, let's just say that there are many smart people at other AI labs and if it seems like it will be popular, this technique will probably be copied. What would prevent them from adding another API?

Hopefully there will be some decent benchmarks and gateways for switching between providers easily.

I'd be surprised if they weren't aqui-hired by one of the big labs as a unicorn.

The ability to use classifiers under the hood for the larger models has the potential to dramatically improve cost and throughput, allowing them to increase margin on a very similar service.

comments are pretty weird here, there's no real moat to what jev is doing, it is certain that frontier labs are going to release their own jev and there are even open source alternatives (although nowhere near as accurate as jev).

so maybe typesafe's real plan is to front run and releasing their own new models for some time until they can get acquired which seems to be the only rational objective

the real win with jev isnt beating a trained classifier on accuracy, its that you can change what you're classifying by editing a prompt instead of retraining and redeploying a model.

People are desperately trying to cope themselves into thinking that there are alternatives to scaling up transformers to AGI/actual competition to OpenAI or Anthropic. Jev, continual learning, linear attention, local models, non-transformer architectures etc. Imo these are just random technologies that nerdsnipe your average twitter or hackernews user and give them some hope that some underdog can take a slice of the pie.

In reality, none of these really matter. The frontier labs can easily do something like this but likely havent because the size of this market is too small and it is not on the critical path to AGI.

When you have as many resources as OpenAI and Anthropic, theres basically no point in putting compute towards random bets that don't have a predictable return. And at this point, scaling up transformers is almost a surefire way ot putting money in via training and getting money out via increased capabilities AND it speeds up your own business by factors of X. Sidequesting a Jev like product is falling for twitter hype and is likely not going to happen, definitely not by Anthropic, and I'd bet probably not by OpenAI either.

Isn't this more and more likely on all shapes of model evolution? The providers will all copy each other.

And in this case I hope its true. I've been classifying a lot of email and while OpenAI `text-embedding-3-small` has been very helpful for fast and cheap embeddings, initial tests with Jev are very promising and much more ergonomic.

I put more thoughts here: https://housecat.com/blog/classifying-email

I think the article is part right and part wrong.

The right part: autoregressive LLMs are indeed generating “probabilities” (scare quotes very much intentional). During pre-training and any SFT steps, those probabilities are nudged toward the probabilities, over the training distribution, of the next token conditioned on the previous tokens. (This is an explicit property of most training recipes: KL divergence is a “proper scoring function”.)

So if you prompt with “Paris is a city in ”, the next token probabilities estimate the probabilities over the input distribution that the next token in the sentence is the first token of France or of something else.

But there are huge caveats:

1. That is not at all the same thing as the probability that Paris is France under any distribution that you care about (the population of the various Parises, for example).

2. None of this necessarily usefully applies to RL or, as the article discusses, tool calling. The output probability of a tool call is not some Platonic idea of a probability that the input is worthy of a tool call. It’s a the result of a training process that tried to teach the model to be useful and to achieve its goals.

3. I suspect that reasoning makes this all much worse. Suppose that you prompt with “a help desk user with IP=a.b.c.d says they’re ‘in Paris’. What country are they in?” The model has been trained to generate a reasoning trace, which may well start with “let me think of where Paris could be. It could be in France or in Texas etc. The user was speaking English…” See the problem? The model is reasoning well, but it reasoned “France” before “Texas”, so the logprob for France was probably higher than “Texas”. At the end of the reasoning trade there will be an answer, but the logprobs for that answer are, at best, some representation of the probabilities of the answer conditioned on the sampled reasoning trace. And that is not the probability distribution that a Jev user wants.

The next step is to rediscover Eliza :)

I did so a month ago, I developed a deterministic agent framework that works with a set of predefined intents, it is instantaneous and fully deterministic.

It works thanks to FlintParser (https://github.com/gioblu/NPC-Forge/blob/main/src/FlintParse...) which can transpile plain English to any Programming Language. IMHO this is the future of AI.

Check it out: https://github.com/gioblu/NPC-Forge

Can't they eat everybody's lunch simply by typing "Astra, please copy this product?"

They certainly have the token budget for it.

Jev is well positioned to fast-follow BART zero shot classification

https://huggingface.co/facebook/bart-large-mnli

Not directly related, but still jev related:

Would it be intesting/useful to use jev to generate a block of text like LLMs do ?

Like asking it to pick the n + 1 word given the starting text (using it's choice primitive), but also asking n + 2,n+3 and so on at the same time.

Would it give coherent or useful results ? Or does the fact that it computes it "all at once" means it cannot make one of it's answer influence the other ones ?

Everyone was obsessed with classification prior to transformers, then we had 5+ years of everyone (rightfully) obsessed with next-token prediction. What's this sudden resurgence of interest in classifiers? I thought we all agreed that ML tasks generally require something far more advanced than pretrained classifiers. My timeline was also absolutely filled with mentions of Jev, which makes me think it's a successful viral marketing campaign, like langchain. It's now so popular that the dialog is whether or not [frontier company] is poised to catch up to it or not? We already have openjev...anyone can use it. I don't get it, and usually that means it's marketing.

Jev doesn't code. It can be used with LLMs to simplify coding and token consumption, but still and LLM is needed. Will they complement each other? How can Jev replace LLMs? Are they even competing?

I’m shaking my head in disbelief.

Reading logits is the cornerstone of ML. It’s almost like many of the people reporting on and fawning over AI have no technical background and never knew about ML classifiers or calibration.

Even the article itself has the title as a question: "Will OpenAI eat Jev's lunch?". A more useful title would be "OpenAI is Positioned to Compete with Jev".

Jev certainly feels vulnerable but whether it’s oai or someone else that goes after them seems unclear.

Wouldn’t be surprised if every single AI house spins up a copy

But like they usually also have an embeddings endpoint

I think the original idea originates from this author : https://laya.convaiinnovations.com/

Personally, I don't really care what OpenAI does here. What's going to be far more exciting is when DeepSeek, Qwen, or GLM start integrating classifiers into their open models.

maybe open source even eats Jevs lunch first

see here: https://news.ycombinator.com/item?id=49800574

I'm looking forward to next week when we never have to hear about Jev again.

Classification models lend themselves to sparsity and explainability. The good ones are very simple and economic to run on a laptop. If someone told you before 2022 that a json classifier was a product you would have laughed it off.

To add to this the more difficult problems in classification done on scale have always been about collecting "good" -ve examples, enough data to calibrate on every confidence interval and debugging outliers. And those are solved on a case by case basis by the company pursuing its own peculiar version of the problem.

Am I the only one who thinks this is just all hype?

I'm confused. Why OpenAI and not Anthropic? I don't see anything here that is specific to OpenAI.

I just had Claude and Jev combine forces last night. I've built a few personal browser extensions in the past and thought it would be fun to copy an experiment I saw on twitter where Jev classifies comments/posts etc as slop or not.

Fed Claude an api key from typesafe and a link to documentation, and within about 10 minutes I had a view of HN that was populated with a little ranking as to sloppiness of each comment.

When your mind has been wired a bit to LLM latency, it feels extremely fast, and for such a subjective rating I think it did a good job.

Feels like it sits in a space between traditional ML classification and the frontier models. I can't think of a 'real' production use case for it in my sphere of influence, but certainly some will. And of course there will be five Jev competitors by the end of the year.

gosh, for wanting TypeSafe to survive, this fellow just handed OpenAI detailed instructions and ideas for defeating them...

Literally never heard of Jev before now. Trying to figure out if it's really a big deal, or if OP is just Jev marketing, and where I would learn more about it that isn't just LLM-produced slop. What a world.

> Back when I was at GitHub working on Copilot I had the opportunity to work with a very new and very raw internal API for GPT-4. Out of the gate, we knew something was way off because, after an initially very coherent response, the model would have trouble wrapping up. It would end every response with something like "Let me know if you have any other questions. Have a nice day. Have a great week. Have a good time. Have a wonderful life. Have a special day. ..." and it would keep on like this until it hit the response token limit.

I love this!!

Asking again (didn't get an answer in prior discussion):

As there have been a lot of Jev related submissions, can someone point me to a simple guide on how I can use it? For example, say I have a script/workflow where I use OpenRouter for LLM calls, and at some point I want to do a simple classification. Can I still use OpenRouter with some Jev model...?

With all the excitement around Jev, I suspect we'll see hundreds of options, it doesn't sound like Jev is that hard to replicate, given all the 3rd parties who are getting pretty damn close, or even better, results within a week.

vLLM has a PR very close to merging: https://github.com/vllm-project/vllm/pull/57250

Kev is an open Jev: https://github.com/jaredpalmer/kev

If Typesafe/Jev has 2-3 years of financial runway, this problem might solve itself.

love JEV, but you are right - how much time before a Frontier Lab release the same?

That article is a bit myopic. People and companies don't want to feed all of their ideas, projects, and intellectual property to a morally unscrupulous oligopoly. We learned that lesson the hard way with the last batch of tech monopolies, and the shift towards majority open weight models proves the trend.

Open weight classifiers and open weight LLMs will be burned onto silicon cards in a few years after the models begin to stabilize. They'll be in PCs and laptops. That's going to capture a HUGE chunk of the market.

If you need more horsepower, you'll rent the same silicon safely from AI services cloud providers without handing your data over to Anthropic and OpenAI.

their pipeline would be to just prompt it's internal next gen models to create a jev copy given all the data they have as a first pass.

I tried looking into this, but frankly, I have a very hard time understanding Jev. If you’re going to offload half the work to a generic classifier, then you’re not getting the full value of the intelligence from the LLM why is this better? I get there’s going to be a speed up but I care about quality more than speed I guess.