Hacker News

A few years is hardly forever and the state of the world here indicates a lot of low hanging fruit still exists.

An LLM can certainly be cheaper than grep, because it’s an approximation, while a grep is deterministic and must examine every byte in what can be a relatively complex state machine for a regex based grep. There are other scales to consider like the scale of your local hardware vs the highly multitenant and high end hardware of the hyper scale inference providers.

There are already high volume models for coding inference where the reasoning time is crazy low and cheap per token where it can build reasonably simple software so blindingly fast it isn’t implausible the bottleneck is the latency in tools and networks. I find them hard to use at times because I don’t have time to think through the next turn by the time it’s done.

Regardless I wouldn’t be surprised to see a world where tokens are so cheap it’s not worth metering them but charging licensing feels with meter tiers at the far horizons to prevent abuse, charge outliers. Subscription models already set this stage well.

The other side to consider is bountiful capacity will also drive tokens to near zero price. The data center build out is barely underway and as it materializes, as hardware efficiencies improve, as techniques and model science and technology improves, harnesses, methodologies , etc improve, the economics flip from load shedding to trying to keep the data centers utilized. The economics lead to the world where tokens are not a unit of measurement for cost for anyone other than the inference providers to manage their utilization.

Yeah I bumped on that too. If it's possible to make llms cheaper than current grep, then it is also almost certainly possible to make grep cheaper.

LLM is spicy memoizing, so it can potentially be faster than a tool call. But people will spend a month tweaking and testing to ensure they have the level of determinism they need, which means it's more expensive, and that they should have used actual memoization in the first place.

> The author observes that a call to GPT-5.6 Luna is only 4-5 orders of magnitude more expensive than grep, and then predicts that at current rates of progress, calling an LLM will soon be cheaper than a grep.

At some future point where LLM hardware is cheaper than simply running grep, then grep equivalent would benefit from those selfsame hardware improvements and be cheaper to run as well, probably still by the same ratio.

4-5 orders of magnitude is huge. Assuming an order of base 10, it's 10000x-100000x. So a call to grep may return in 1s on a typical PC. That means a GPT call takes equivalent energy of 10000-100000 PCs to do the same in 1s. That's a difference that can't be equalized with scaling. It would require a revolutionary breakthrough.

I also don't understand where the idea that frontier models are getting better efficiency comes from. The results are certainly improving, but that comes from feedback and multiplexing requests, which cost more.

Well, think that statement through a bit:

Grep reads through the entire file looking for patterns.

An LLM scans its neural net (in ways that I don't understand) which is kinda-sorta like having a huge index.

You can improve over Grep if you have an index; and the LLM has an index.

Thus, it's plausible that an LLM can be more efficient at reading its neural net (IE, index) than Grep reading the whole file.

Right. And some hardware improvements will speed up both grep and Luna, which won't close the gap.

What if the tool is more advanced, like an optimizing compiler: `g++ -O3 -march=native -x c++ - <<EOF ... EOF`

If the compiler invocation is sufficiently slow, the llm could consider outputting a binary directly?

For all we know matrix multiplications are a faster way to generate optimized machine code than branchy sequential compiler code with tons of heuristics and passes.

It might never beat out grep, but it could beat some more expensive to call tools, similar to how heuristics will often be faster than exact answers. Rust Analyzer can be slow at times, I could see an AI tool taking over a subset of its work.

A classic case of someone projecting out to infinity from just after the first bend of the S curve.

But grep is just a tool in a pipeline between a question in your brain and an answer you are searching for.

what if an LLM finds the answer early?

grep might continue to read everything, doing the wrong thing correctly.

> Barring a true breakthrough in something like quantum computing or room-temperature superconductors

Won't that also help grep and then move the asymptote down more?

> calling an LLM will soon be cheaper than a grep

From a computational standpoint this is obviously nonsense, but from an attentional one I'm not so sure. It may already be more attentionally expensive to use grep in some cases, such the moment you need to remember a non standard arg. And if this applies for performing a simple http operations, then it certainly applies going up the complexity chain.

room-temperature superconductors, sure, but I fail to see how quantum computing will disrupt – in the medium term (25 years or so) – classical computing in any meaningful way

Is running LLMs (or some other ML workload) on/with quantum computers expected to bring efficiency gains?

"Did you know that disco record sales were up 400% for the year ending 1976, if these trends continue...AY!"

I think the article's analysis is basically right in a vacuum. That is, I think it's clear that inference is a viable business model. But what isn't clear is whether it will be such a profitable business model for any given company that it will justify the investment that company has taken. I kind of think the winners might be a follow-on generation of companies that focus on this commodity inference business model instead of the invent-machine-god-first "business model" and thus are wiser about their level of investment and capital costs.

The problem source is that the "cost" of tokens are taken at face value from business that are losing money at record speeds. E.g. https://artificialanalysis.ai says "doing task A costed us $10 using OpenAI", and that is the "cost" the OP used as basis for "tokens are cheap". Meanwhile OpenAI is losing $19 for each $1 in revenue... So right now OpenAI should be charging around $200 to do task A just to break even, but that would mean their use base would collapse.

They are already turning profits and inference has shown to be a cash cow. And they've already secured compute for the next several years.

Think of it like spending $1 trillion to become the next Google. That hardware itself may never turn a profit. But if 10 years for now you're the software provider that owns the ecosystem around too cheap to meter tokens you've got a money printing machine.

Exactly. "Cost-to-distill" is a critical parameter. Right now usage of frontier models for all tasks is both subsidized and irrationally popular even at the subsidized price. Deepseek would solve most tasks faster and 10x cheaper. I agree with the author that just as Deloitte exists, frontier labs will exist. But not because their products are proprietary technical marvels or gods, but rather because of branding.

It's a deliberate reference/meme that is basically used to acknowledge the precedent of overly exuberant predictions of cost in an emerging technology but argue "however, this time it's true".

Of course, perilous territory for future irony depending on how your prediction plays out.

It's really quite unfortunate that the promise was not delivered, mostly for political reasons. I hope that a new wave of reactors and the dire need for clean energy restarts the nuclear race.

https://www.wired.com/2008/02/ff-free/

"Between digital economics and the wholesale embrace of King's Gillette's experiment in price shifting, we are entering an era when free will be seen as the norm, not an anomaly. How big a deal is that? Well, consider this analogy: In 1954, at the dawn of nuclear power, Lewis Strauss, head of the Atomic Energy Commission, promised that we were entering an age when electricity would be "too cheap to meter." Needless to say, that didn't happen, mostly because the risks of nuclear energy hugely increased its costs. But what if he'd been right? What if electricity had in fact become virtually free? The answer is that everything electricity touched—which is to say just about everything—would have been transformed. Rather than balance electricity against other energy sources, we'd use electricity for as many things as we could—we'd waste it, in fact, because it would be too cheap to worry about."

... What Mead understood is that a psychological switch should flip as things head toward zero. Even though they may never become entirely free, as the price drops there is great advantage to be had in treating them as if they were free. Not too cheap to meter, as Atomic Energy Commission chief Lewis Strauss said in a different context, but too cheap to matter. Indeed, the history of technological innovation has been marked by people spotting such price and performance trends and getting ahead of them."

My issue is that datacenter energy costs are being prioritized for commerce over residential use, so the average consumer is paying more for electricity, because a datacenter needs more electricity and they're getting tax breaks. Assuming this all improves efficiency for new products like automated robotics, there is a debatable benefit. Jevon's Paradox has no ceiling, except the environment, and people's 401ks.

On the other hand, this did work out in other areas. I pay a flat monthly rate for all-I-care-to-eat internet access, for example. My email provider has limits on storage space but I don’t get charged per email sent or received. It’s not a crazy concept on its face, nuclear power just didn’t work out as well as was hoped.

Sound critique. I'll add that the Artificial Analysis intelligence index is not considered a good metric for intelligence anymore. Most of the benchmarks that it comprises are saturated or considered low signal today.

LLM hardware wants to be shared. It's significantly more efficient economically to have expensive hardware be better utilized.

> Nvidia will sell a lot of Rubin CPX's, but their margin on that will be a lot smaller than B200 because there is so much more competition in that space.

Given the shrinking margins, I wonder whether Nvidia will still think it's worth competing in that price-performance corner in the long term.

If we're all using distilled open-weight models in ASICs in our own systems the energy cost will come way down. The question is when that becomes a reasonable solution for a broad set of use cases.

I think this is a case where just drawing a "line goes up" extrapolation is incredibly misleading because there is _tremendous_ economic pressure to get costs down, and costs are very tightly tied to energy use. All of these systems are incredibly inefficient right now and have a lot of room to go down in energy use. I'd guess that the absolute _floor_ is burning model weights directly to silicon and that's like a 90+% reduction in energy use.

No it'll be fine as long as you do your part and not drive a car, or have AC, or eat meat, or have children, or live in detached housing, or...

They're not secretly subsidizing, they're openly subsidizing.

Token pricing was a small minority of customers up until this year, when all the labs started trying to force customers onto token-based billing. Within the last week, Anthropic repriced my team's plan from a temporary "50% extra tokens" to 25%: https://support.claude.com/en/articles/15910845-claude-code-...

The fact that all this is ongoing within such a short timeframe should make you suspicious of any analysis that claims to be observing "statistical trends" like they've discovered a new Moore's Law out of 6 months of pricing data from 2 companies.

The cost is decreasing quickly, mostly because the labs stopped competing on quality. At the same time, the costs are enormous, and all labs are very openly subsidizing usage hoping to get enough scale to be profitable (while 1 of them has suspicious unity numbers and can be hiding negative marginal income).

Also, it's impossible that they become cheaper than specialized software. Or even as cheap as them. It's still possible that they become cheap enough that it doesn't matter.

I find it difficult to believe the inference only providers (Baseten, Fireworks, Digitalocean, etc) are all selling tokens at a loss.

Asking Claude for a rough estimate based on publicly available throughput and cost data for open weight models on modern GPUs suggests serverless, pay-as-you-go inference is profitable on owned GPUs with reasonable utilization (30-50%).

The labs themselves when they openly say that they’re subsidizing tokens, I’d imagine.

> AI is not a bubble

When people say "AI is a bubble", they mean economically as a whole, which includes data centers.

Perhaps we need better terminology for "product useful; numbers nonsensical"

I disagree. I own over 1TB of vram at home. I can tell you that it's not a bubble. From my builds, I would rather have cloud, cloud is easier. From running small models like Qwen3.8-27B to large models like Qwen3.8-2.4T. I can tell you that small models will never be enough or match up. Everyone will want the smartest model, not just a good enough model.

Jevon's paradox says that if data centers can serve a lot more tokens per dollar or watt there will be increased demand for data centers.

Seems like 3-6 years is a pretty conservative timeline to me. Memory and inference chip production could ramp up massively in that amount of time.

The article observes that the cost of frontier intelligence from 2025 has fallen 100x in the last year. It also notes that the energy to run models is also collapsing. Consumer hardware is borked right now because these new algorithms are revolutionizing the utility of a computer. Computing is technology who's cost has been collapsing for 90 years, and its a safe prediction that it will decrease again.

> Yeah okay bud, anyone checked in with the state of consumer hardware recently? Not the author, evidently.

RAM prices will crash when demand drops even a little. They'll probably crash to a lower (inflation adjusted) level than before. This has happened before.

Industrial scaling in general often looks like a sawtooth: price spike, capacity investment, crash, repeat.

Part of what's keeping prices high a little longer is that everyone knows this and is a little reluctant to plow resources into chip fabs for fear of having the bottom fall out before they recoup or sell that to someone else to hold that bag.

Graph the average compute and RAM in a mid-high end laptop at an inflation adjusted price point for the past 40 years. It's very exponential and hasn't slowed down much.