Hacker News

Right here is the video demo of what I built - https://streamable.com/e49cgt

You mention cost in one of the replies. Can you elaborate on the cost profile (ballpark) for various problem types? I would also be curious to understand the strategies employed and what the costs look like across each.

Definitely interested, would love to see a video :)

So what harness are you using? And what LLM’s

I think it's more a consequence of pushing for the biggest valuation/IPO. Rumoured profits on inference are north of 70%.

Taking SpaceX as an example, they have increased prices across all their consumer products over the past six months. But they definitely aren't short on money with Alphabet and Anthropic combined paying them over $2 billion per month.

Microsoft/GitHub lost out here as they were just repacking other people's products.

The github example is also a bit of an outlier because they made a recent change to their pricing so that's why its such a drastic jump.

Also I mean prices in generally for all things are based on underlying factors, that doesn't make them arbitary (i.e. github executives using a random number generator for token pricing would be arbitary)

If input tokens dominate the cost to that extent, this implies that major gains are possible by making better use of caching. You could basically ask the model to do a one-time "compaction" step including a dump of the relevant portions of the code, and use that as the cached prefix for a large amount of "swarm" subagent calls.

Did you experiment with giving agent better tools to navigate and document the codebase? Asts, language servers and so on?

A million tokens (not cached) sounds like a lot.

And they like to burn a ton of tokens writing and debugging tests that are semantically corrupt.

Unit tests are a type of dynamic testing. As opposed to static testing which is linting/typechecking etc.

If you want a difference kind of dynamic testing besides unit tests, have you tried writing it in as a requirement during the planning/PRD phase?

And AWS heavily pushes a complex lambda solution stringing together as many chargeable AWS services as possible for a simple requirement

Their interests are often not your interests. In this case they want you to unnecessary money on useless work (let's stop the euphemism of "tokens" btw)

you can just tell them to do more dynamic testing. I think dynamic testing is partly frowned upon because it slows things down & can take down software where you wouldn't expect

I hope this horrible time will soon be over when cheaper NPUs come available from more hardware companies, and also when model size get optimized down further.

I wonder what hyperscaled compute farms and models will be good for at that running cost when most AI needs can be fulfilled by on-prem and on-device hardware and models. Probably only customer left are big governments. So in the end the tax payer has to pay for those billions of investments by the AI cartel.

AIshittification

Code review could also be run as an unattended/batched task though, possibly with at least some use of on-prem inference (which excels at this). That would be a major saving compared to the usual cloud inference scenario.

That assumes Tokens will remain a meaningful expense. I’m not sure developers will find uses for ever more tokens nearly as quickly as the prices fall.

I know how to drop a company’s token costs to zero: treat tokens as a utility same as internet and make engineers pay for it.

Tokenomics had been already used by marijuana enthusiasts for a long time.

cryptocurrency economics = cryptonomics

You're welcome! =)

New fad. Forget about the old fad. This one will be old soon, you better get on board before its too late!

Crypto was already a term before cryptocurrencies made it about them. Web 3.0 was already a thing before crypto bros made web 3 about cryptocurrencies.

So what? Terms are reused in different contexts all the time. And most people have moved on from cryptocurrencies anyway, so there’s little chance it’ll confuse anyone.

I don't think business is interested in any sustainability of anything. There's zero incentives for that for anyone.