How the Claude Code team thinks about verification (claude.dev)

181 points by craigmart 8 hours ago

OtherShrezzing 7 hours ago

This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.

There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.

But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.

xnorswap 7 hours ago

I agree completely, there was a much better talk on this by Lauren Tan: https://www.youtube.com/watch?v=O_zcc6Xl12g

A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.

fg137 5 hours ago

The talk makes that sound something new, but every team I know has already adopted that before AI. I really don't get what value that talk provides -- I learned nothing after watching it.

xnorswap 5 hours ago

dominotw 4 hours ago

verelo 3 hours ago

I wish my reaction to seeing this post was "oh wow Anthropic never write stuff like this, I should make time to read it". My gut instinct was to say "this is going to be long, over complicated and not worth my time". I really think this is the influence of the 90's - 00's SEO. Write lots of stuff, hope Google cares...get more business.

simianwords 7 hours ago

You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification.

Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?

xnorswap 6 hours ago

sceptic123 6 hours ago

Creamsicle47 6 minutes ago

Lauren Tan is a professional larper

rolosa 6 hours ago

That developer brags about slopping out 2000+ PRs a month, how is anything they have to say useful?

xnorswap 5 hours ago

svnt 3 hours ago

It feels like that because it is marketing to improve distant-customer results while also increasing their token use to grow revenue.

pu_pe 6 hours ago

> It’s got some very expensive practices, where near-free practices would produce the same or better results.

I think tokens are near-free already, even for those of us outside Anthropic.

More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.

Grimburger 5 hours ago

> tokens are near-free already

How long do you think that will last?

Am over 11 billion tokens this month on a subscription. In API usage its close to $5000 and there's still 13 days left in the month.

Eventually the music in musical chairs stops. Already many anecdotal reports of the always opaque limits being lowered across the board. On a gemini pro account for personal use I can use a weeks worth of Argon in about 20 minutes on agy in the last month.

ChatGPT Pro I can use the weekly limit in under 2 days, this is with significant context clearing and handoffs.

The cost to provide frontier models isn't cheap and it's entirely subsidised by a winner-takes-all mentality from investors which in all likelihood won't come about.

olkre 4 hours ago

skohan 5 hours ago

It's great to have that option available, but it's incredibly inefficient to just throw GPU hours at a problem that can be solved deterministically.

pydry 5 hours ago

castis 5 hours ago

> Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.

No one is debating this. The debate is that an agent performing validation work via its own chrome plugin and internal reasoning will always cost more than performing that same validation work with deterministic tools.

pu_pe 5 hours ago

chrisweekly 5 hours ago

jgalt212 6 hours ago

As far as timelines go both property based tests and fuzzers predate this token maxing strategy.

bayindirh 6 hours ago

NewJazz 2 hours ago

Great, I'll mail you a quarter and you give me your Anthropic API key. Deal?

skydhash 6 hours ago

> you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case

No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite.

oceanplexian an hour ago

> This article feels like the result of having unlimited tokens at your disposal.

Gong to be honest where do you think this is going?

Do you think tokens are going to magically become more expensive or models are going to become less efficient?

Of course you should be tokenmaxxing, you can already tokenmax with a model that competes with the SOTA 1 year ago on a moderately powerful laptop you can walk out of an Apple store with.

throw10920 an hour ago

To be a bit more grounded and expand on your point:

I, personally, have already experienced situations where I was employing cost-optimization optimization strategies for models 8 months ago, that are borderline-useless now because the Pareto frontier has advanced both more than I thought and also in different ways than I thought (cheap models got smarter, better at following instructions, and more clever about tool calls).

That is, the strategies decreased in effectiveness disproportionately to how much the model improved.

It's worth considering that strategies that seem insanely expensive now may be economically viable in a year or two. Or possibly significantly sooner than that.

It's also possible that, like how jet engines get more efficient as they get larger (leading to the 737 MAX incidents, which seems spiritually significant) corporations with budgets orders of magnitude greater than ours benefit from very different strategies than we do.

ar_lan an hour ago

This seems like a very strange comment to come from someone I am assuming is a SWE. When have we, ever, been happy with non-deterministic, objectively more expensive, solutions?

You're assuming a lot of costs that should just be considered ok in your stance:

* Why pay anything multiple times when you can replicate deterministically for free? * Why require always-online models when you can run computation locally? * Inversely, why use power-hungry/PU-hungry models locally when you can have a deterministic software that can be optimized to not* be memory/PU/power hungry to do validation?

I think tokenmaxxing can be a great start* to get things going but when it comes to productizing I want determinism and cheap optimizations, and using frontier models to build these cheap optimizations is what we've been doing at our company over this past year to great success.

LelouBil 7 hours ago

I had a developer friend who wanted to automate gym reservations at night, his first idea was to make use of ChatGPT Computer use, but he encountered refusal of the AI to enter his password, and it was difficult to convince it to do it without approval as well.

I just told him to make the AI write a small selenium python script to do the clicking and it worked amazingly.

abirch 6 hours ago

It seems like a new stack where you try to push things down as low as possible for efficiency.

    [    AI    ]
    [ Software ]
    [ Kernel/OS]
    [ Hardware ]

saagarjha 5 hours ago

Fun fact: Codex computer use is trained to not enter passwords. When it was being designed this was considered to be a very cheap way to ensure the model would not go poking around places it shouldn’t be, instead of constantly trying to check whether the action being performed was authorized or not. People have spent a lot of time marking out their UIs for sensitive actions and it’s worth using it :)

arbll 7 hours ago

What makes you think they are not using Playwright ?

For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.

bismanawaz 2 hours ago

Full app testing by agents would mean that each agent needs its own running app and browser use. So machines would cost too, not just the tokens in my opinion.

urbsgpw 5 hours ago

I got this feeling a couple of times this year with their sharing of best practices. They seemed theoretically interesting, but not if you're cost-conscious. It seemed like thinking with an unimited budget. And dont get me wrong, maybe that's the way to go since you assume costs of compute will fall down eventually (and these systems will become more optimized), but it would be completely untenable to run if you don't have a similar budget constraint (or lack thereof).

dcss_gardener 3 hours ago

It's really interesting to me to contrast the code and publications of deepseek and anthropic over the last year. Anthropic's stuff has some ideas that can be extended or applied but everything they do assumes basically unlimited frontier model inference and so is nearly unusable to me in the form they publish it.

Where deepseek prices inference the way systems programmers price latency. As a resource to be spent carefully towards specific goals, with systems in place to ensure you're within budget and getting the value you expect from it.

And then this also has second order effects on how each company treats human attention. Having fable xhigh do 3-4 passes over a change can legitimately reduce the need for human attention on it in a way that deepseek flash just can't really. So then they price operator attention in a similar way. With ant nearly considering human intervention a failure mode, while ds again prices it as a budget to be designed within.

And neither one is "right" speaking generally. I have my aesthetic preferences and my own constraints that make one a more appealing approach personally. But both are building towards the goals they have with consideration of the resources at their disposal, it's good engineering either way.

radlad 3 hours ago

Yes. Anecdotally, I saved a ton of tokens (and bigger: time!) switching from having Claude drive Android emulator VMs (even with detailed skills and helpers) to using Roboelectric tests.

bitlad 7 hours ago

They are living in the future.

I think future AI cost will be as cheap as doing an API call in classic sense.

tpetry 6 hours ago

Many API calls can be answered by something as underpowered as an Raspi. But AI needs very expensive GPUs to run.

So no way it will be as cheap. That is just wishfull thinking.

ben_w 6 hours ago

Quite possibly. But, a thought I had some years back:

In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task.

Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition.

Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API.

brabel 6 hours ago

slowhorse 6 hours ago

So the future is about replacing deterministic procedures with stochastic procedures?

On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too.

mleo 4 hours ago

hypendev 6 hours ago

Exactly.

If you're doing this, the agents should create deterministic tests from its process rather than Agent tests or just writes deterministic tests for start. This is a bit of an absurd approach.

The issue I find with these articles is that they are basically the same boat as "Facebook Engineering" articles were 20 years ago.

People will be "oh because Anthropic does it it must be the way", then a bunch of influencers will retweet it as "anthropic kills writing tests", which some CEO's and team leads will parrot and get their team to absorb these practices even tho they are not the best practices.

Some of these things are just reinventing the wheel, like this is a basic script called "run_simulator.sh" and a pre-commit test hook, not a skill and "commit gate" that needs to check session transcript:

>To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes.

>We've also created a commit gate that can block a commit if the session did not contain /code-review or /verify. We've set it up with off, observe, and enforce modes.

baxtr 7 hours ago

Wait, are you saying a company that makes money from selling tokens generated a content piece recommending people solve their problems by using a lot of tokens?

Scandalous.

pluc 7 hours ago

Here's how to properly use XYZ, yes it'll cost you more but we manufacture it so to us it's no issue.

Do you not see the propaganda through the pseudo helpfulness?

hackyhacky 7 hours ago

It's the same reason they give you a recipe for Chex Mix on the back of every box of Chex.

ben_w 6 hours ago

> Do you not see the propaganda through the pseudo helpfulness?

I find it amusing that people phrase bog-standard "what can our product do" documents as "propaganda", even when it's a dogfooding exercise.

Like, I guess? In the sense that all marketing can be seen as capitalist propaganda?

pluc 6 hours ago

seiru 3 hours ago

e2e tests (which you describe at the end of your post) are literally the first thing the article talks about though.

empath75 2 hours ago

> This article feels like the result of having unlimited tokens at your disposal

I think it's worth designing around that because tokens will only get cheaper over time.

cyanydeez 7 hours ago

Chinese gonna beat anthropic because they gluttonous use of AI.

simianwords 7 hours ago

Its almost become a meme at this point: well actually you can achieve the same using X so why waste tokens?!

The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.

Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.

Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.

I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.

LelouBil 6 hours ago

The better thing is do use both, use claude to convert the invariant to deterministic code, and review said code to be sure it matches your natural language idea of it.

The issue with only having the natural language specification is that it is non-deterministic.

fingerlocks 6 hours ago

That’s a terrible example because you can verify that at runtime by attaching thread watchdog daemons. This is invariant across all languages natural or otherwise, invariant to the binary, invariant to tokens or context, invariant to network access, and you can set the assertion, making it invariant to the thing your trying to accomplish.

I expect a lot of no-beard man-child programmer wannabes will brain rot early in their careers and find themselves wallowing in the hopeless despair of sitting in front of a computer arguing with a chatbot for decades on end, living a life trapped under the weight of endless mediocrity, unknowingly desperate for a modicum intellectual stimulation that you’ll never find because you can’t prompt for that, bro.

simianwords an hour ago

SecretDreams 7 hours ago

You don't get a nice juicy IPO valuation by trying to save money or be efficient with your token usage!

catchnear4321 7 hours ago

It behooves the token merchant to espouse the benefits of having more tokens.

Not that it is an explicit, conscious direction, but it very much is one.

You’re not wrong, in that free flowing tokens are enough to convince such a group it’s simply good advice.

adamddev1 7 hours ago

The more you use AI to write your codebase, the more you need AI to manage your codebase.

The more you use product X the more product X you need to buy. Nice to be selling product X.

I'm not working for a manager, so I don't need to use it at all.

catchnear4321 7 hours ago

tshaddox 3 hours ago

Is Claude Code considered to be a reliable piece of software these days?

I stopped using as my daily driver around 9 months ago, and at that time it felt very heavy/janky and seemed to have very regular bugs and breaking UI changes.

I’d be happy to hear that it has gotten better!

kevincox 3 hours ago

My experience of Claude Code is a buggy mess. The only thing that seems to reliably work is the main chat (when they model APIs are online, but that is a separate topic) all of the other features such as file browser, diff viewer, model switcher, and whatever else I didn't think of seem to be working only on occasion.

I definitely wouldn't take their word on verification as a particularly good source.

jrflo 2 hours ago

The desktop app is actually great now, I moved to it from the CLI recently. It has a live view of the context window and what is filling it, integrated terminal, diff, markdown viewer, file tree, browser, etc. If you're embracing agentic development, it's a great environment. The only bug I've encountered is sometimes it lags for ~1s when switching to a long conversation.

couchdb_ouchdb an hour ago

I still prefer the terminal. I used the UI the other day, and it was like really really verbose compared to the terminal filling up the screen with JIRAs and all sorts of things.

ellyagg an hour ago

Reliable enough. Take care not to conflate reliability with overall value. Reliability is one metric, and easy to misread, like all individual metrics. Claude Code is very powerful and valuable overall.

timedude 3 hours ago

What do you use now? And how does Claude Code compare to OpenCode if you have any experience with that?

asmvolatile 3 hours ago

The Claude code team doesn’t think about jack sh*t. Interacting with Anthropic teams makes it clear that nobody at that company has any idea how their software works.

ausbah an hour ago

i mean yeah, they’re very public that no one writes much less reads any code anymore. everything they do is via thousands of agents and burning quadrillions of tokens. their blogs and other public facing comms are full of “best practices” reiterating the above and how much more code their pushing everyday

most of that is prob for internal tooling and model related work, but seeing their public facing software makes me think that all this work hasn’t really translated to all that much more than 3 years ago

girvo 8 hours ago

> With them, you can run dozens or hundreds in parallel.

Im already spending my $2k usd per month in tokens with just one or two agents. How do they expect me to do that?

simianwords 7 hours ago

I don't know why you are comparing your personal spend with how a profitable enterprise company works.

I have worked in companies that spend millions a month on some random infra. I'm obviously not going to complain that its not sustainable?!

Edit: can't reply but Token maxxing is not over. We are just getting started. And please keep in mind two things are happening

1. AI costs are reducing by 2x every few months

2. This causes more usage because of Jevons Paradox

3. Also, running verification tests on releases is not that frequent and it happens only once for a whole team of 5-6 people

cannonpalms 4 hours ago

This article describes verification testing that's being done on every PR before merge.

viklove 38 minutes ago

Anthropic isn't profitable

surgical_fire 3 hours ago

> how a profitable enterprise company works.

If they spend tokens like Anthropic suggests, they won't be profitable for long.

duskdozer 7 hours ago

Why aren't you spending $12k on tokens? Or better, $200k? Coding is solved, you can now infinite money glitch; just give your current money, which will be useless in the Agentic Future anyway, to Anthropic now.

thatsabadlook 7 hours ago

You should fire yourself and have your boss go all in on anthropic duh. If you are the boss then you either shouldn't have gone down this path, accept your code won't be verified, or shut your doors.

4lb3rn 7 hours ago

Exactly this. I build large systems like erp, ipaas just from a Claude team with two premium licences. everything builds, verifies, deploys itself. I can track everything and implement features in a days. security and big fixing is background work. I would love to know what 2000$ worth of tokens do in your business.

philipwhiuk 5 hours ago

Just IPO for a $1 trillion then you'll have loads for tokens?

varispeed 7 hours ago

This is probably aimed at gullible governments to burn tokens using tax payer money.

Narann 7 hours ago

The more I read articles like this, the more I realize Claude is good for process prototyping, but you need to "bake" the process with flat code/script/organization to decrease costs.

The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.

koolba 7 hours ago

> Nobody can read every change, so we use failures as one way to decide what needs a person.

Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.

fhd2 6 hours ago

The "right mind" of companies is a bit different from that of individuals. Generally they care about three things: Revenue, costs and risks.

To companies, some risks are worth taking, if it can increase revenue or lower costs. Many companies happily do not-quite-legal things, taking into account the potential of being caught and the size of the fine.

In the case of Anthropic, from that perspective, it almost seems like a no-brainer that they would want to push an "our AI can do it all, no humans needed" narrative in everything they do. They need hype like we need oxygen.

I would agree that for many others, the risk is too high, and the reward too low. So they'd indeed have to be out of their mind to deploy code nobody understands. But depends on the situation of course.

Just trying to say, in a really lengthy and possibly patronising fashion, that "good engineering" is unfortunately not something many (most?) companies prioritise.

ForHackernews 7 hours ago

> Nobody in their right mind performs non-mechanical changes of 100s of files.

We're in a brave new world, many many big name companies are doing this.

testtestmetest 7 hours ago

Their status pages reflect that.

IshKebab 7 hours ago

> Nobody in their right mind performs non-mechanical changes of 100s of files.

Not 6 months ago, sure. Today? It's not totally crazy. In a year it will be routine.

r0m4n0 2 hours ago

The rate at which software engineering has changed is staggering. We don’t fully understand what is the right or wrong way to be doing things. It’s cool that they share what they are doing but that’s probably about the only thing most people should take from this. Jury is still out on whether this is sustainable budget wise, can be done better differently, whether this sustainable maintenance or quality wise, etc

FartyMcFarter 7 hours ago

> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.

If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).

If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.

hliyan 6 hours ago

In my opinion, the output of a stochastic process can only be truly verified through a deterministic process. Tests already do this for functional accuracy. Linters can do this for line-level code, with code treated as text. But there is currently no such method of deterministic verification for the structure of the entire program, or more importantly the viability of the architecture when you throw in connectivity and IOPS.

Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.

How to do this for architecture: unsure.

Dashadower 34 minutes ago

Its called formal verification. If you do try a refinement proof against a specification the proof architecture is just following the implementation's code and performing case anslysis for each branch possible.

chamomeal 4 hours ago

> a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules

Isn’t that basically a linter?? If you go too far outside a regular ol’ linter (even a real nice type aware one like typescript-eslint) you hit a ceiling where some intelligent being needs to interpret things.

Though I will say I have seen interesting projects in clojure world like rewrite-clj, which makes it really easy to do structural analysis and modification of clojure programs. I’ve definitely thought there could be more room for stuff like that in the age of LLMs. I’ve also made more use out of ts-morph than ever before.

Also in my box of purely delusional ideas, I’ve thought about how cool it would be to encode LLM-derived semantics at the AST level, which then could be run through a regular deterministic linter. Like a switch case or pattern match could have “this is the happy path” or “this is an edge case” encoded into each branch. Then you could do stuff like collapse all non-happy-path lines, or highlight all edge case handling.

gregwebs 6 hours ago

After I created this skill for verification, it completely changed my level of trust with AI coding.

https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...

The implementation process that I am using also looks for verifications during planning and reviewing and makes sure they are completed. It does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.

https://github.com/gregwebs/skills-sdlc/blob/main/skills/imp...

nelox 6 hours ago

We are well on the way to recursive self improvement

holoduke an hour ago

What we need are ultra fast testing suites. Specially e2e tests are notoriously slow. Runtime is big and its often difficult to isolate just a few tests. I want 1000 e2e tests in a few seconds instead of 10 minutes. It's also a mayor blocker for an agent that cannot work fast because of the constant waiting. At the same time you need the test feedback loop in order for the LLM to do proper shotgun programming.

sashank_1509 3 hours ago

Take 10-20 skilled programmers, the kind that made web browsers, game engines etc. Give them a year. They will end up with a superior working version of Claude Code without thousands of bugs and issues, that keep getting fixed and unfixed and trillions of tokens spent by AI.

They won’t even do all this verification nonsense and their end product will be of superior quality.

anshumankmr an hour ago

Honestly, the amount of bullshit unit tests Claude keeps pushing nowadays. I took a few hours the past few weeks, some with AI, some by myself to hand remove some really dumb fucking test cases. God knows how long the CI must be running when these guys push their code.

AlienRobot 23 minutes ago

I wonder what was it even trained on?

Who ever had the time and will to write unit tests that mock every single function/class being called, checks the strings of thrown errors, etc., for every single function, for several tests per function?

The instant you change the implementation the tests break because it's testing how many times each method has been called of an object that was passed as argument.

That's just an insane way of writing tests. But for AI that's normal for some reason.

It feels like it's making unit tests the "textbook" way that in practice you wouldn't do because in practice half the things it's testing are inconsequential.

nandanadileep29 7 hours ago

excellent content from the team. very concise and clear

dotdi 7 hours ago

> https://github.com/anthropics/claude-code/issues has over 16k open issues

this u?

laichzeit0 7 hours ago

And 90k closed. Looks like it's doing a pretty good job.

ahsillyme 7 hours ago

Issues are automatically closed after being inactive for too long. It's also true that many issues that are filed there have nothing to do with Claude Code proper. To conclude, these numbers (either many open, or many closed, or any relationship between them) a-priori mean nothing either way.

orbital-decay 7 hours ago

Opening and closing GitHub issues does not necessarily mean doing some job.

catlifeonmars 7 hours ago

A better signal would be average issue age.

Ampersander 7 hours ago

Why is the team thinking? Their product should be doing that for them

simianwords 7 hours ago

HN comments love to use the curmudgeonly tone but did you not stop to think that they are going as far as possible to let model do what it can and do everything else?

Did you forget that they literally publish benchmarks that show how much of the task can be done independently?

Goto80 7 hours ago

Why is there a team? Their product should be the team. /s

saagarjha 5 hours ago

This is just a depressing read, and I’m not even sure why the team decided to post it because it just looks horrible. Like, you’re telling me that the team’s approach to verification is that they post Claudeslop blog posts talking about the most unimaginative ways to write tests? That their idea of making things work is this?

> You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again.

There are people out there who are doing all sorts of crazy neat things in the verification space using the tools you build and yet the people who could be helping seem to have no good ideas on how to help :/

titzer 5 hours ago

I see a lot about adding skills to make Claude test things.

How about making Claude generate scripts that you can run?

Algorithms are better than agents. In fact, the whole point of AI is to reduce thinking to algorithmics. I didn't see a lot of evidence that the team understands this. If you need a magic box to test your other magic box, you have magic boxes all the way down.

duskdozer 3 hours ago

Scripts that you can run on your own CPU don't cost tokens, so these companies are disincentivized from encouraging them.

TheRoque 6 hours ago

I can totally see that they don't review their product at all, Claude code is probably the thrashiest TUI I've ever used.

lavendermon 4 hours ago

At what point in the future could the title become: "How the Claude Code team's Claude Code thinks about verification"?

camgunz 2 hours ago

Congratulations you have invented testing.

inamorty 7 hours ago

Can they not afford someone with an engineering background?

FartyMcFarter 7 hours ago

They poached a lot of people from companies like Google that have solid engineering practices. I imagine it's a culture issue if anything, not a knowledge/background issue.

thatsabadlook 6 hours ago

Googles engineers are good in those types of environments. Most environments aren't Google like at all. Ive seen so many small and medium companies hire googlers and burn money because of it. I actually watched a start up die because of a few key middle management/staff engineer hires from Google. They just could not operate for anyone other then themselves and were oblivious to the scale they worked on.

FartyMcFarter 5 hours ago

whamp 4 hours ago

This whole thing read like google not anthropic.

matltc 6 hours ago

Formal vibefication

jordemort 4 hours ago

we think about verification

we don’t actually do any of the work but we think about it

ForHackernews 7 hours ago

Ironic that this page loads partially styled with broken CSS:

"The resource at “https://claude.dev/_next/static/chunks/241gnzjfqbp9f.css” preloaded with link preload was not used within a few seconds. Make sure all attributes of the preload tag are set correctly."

dude250711 8 hours ago

> ...On the Claude Code team, the vast majority of our code (~90%) is written and reviewed by Claude...

I am more interested in the remaining 10%.

TobTobXX 8 hours ago

Apparently, the remaining 10% too, but not attributed correctly, so they are careful of claiming it as Claude-written. (Can't remember where I've read that.)

thevilish 7 hours ago

Boris has said it when asked, such as here;

“Also Claude, but we're fixing up the attribution to be super confident before claiming it”

https://x.com/bcherny/status/2111207278653399460?a=CQxYPj8Va...

123ajhgF 6 hours ago

Is the perfect verification the reason that Claude is down so often?

More snake oil and theatrical agent setups described in insufferable Claudish. I think the target group of Claude is people with an IQ below 100.

mococa 7 hours ago

So a lot of slop...