Hacker News

When I was really early in my career, a mentor told me that code review is not about catching bugs but spreading context (i.e. increasing bus factor.) Catching bugs is a side effect, but unless you have a lot of people review each pull request, it's basically just gambling.

The more expensive and less sexy option is to actually make testing easier (both programmatically and manually), write more tests and more levels of tests, and spend time reducing code complexity. The problem, I think, is people don't get promoted for preventing issues.

Expert reviews are just about the only thing that makes AI generated code viable, though doing them after the fact is a bit sketchy, to be efficient you kinda need to keep an eye on what the model is doing as its working.

Unchecked, AI models output code that is as buggy as it is inefficient. In smaller green field contexts, it's not so bad, but in a large code base, it's performs much worse as it will not have access to the bigger picture.

In my experience, you should be spending something like 5-15X the time the model takes to implement a feature on reviewing and making it fix its errors and inefficiencies. If you do that (with an expert's eye), the changes will usually have a high quality and will be correct and good.

If you do not do that due dilligence, the model will produce a staggering amount of low quality code, at a rate that is probably something like 100x what a human could output in a similar timespan. Unchecked, it's like having a small army of the most eager junior devs you can find going completely fucking ape in the codebase.

> requires an amount of time approaching the time spent if they had just done it themselves

It's actually often harder to fix something sloppy than to write it from scratch. To fix it, you need to hold in your head both the original, the new solution, and calculate the difference, which can be very confusing. The original solution can also anchor your thinking to some approach to the problem, which you wouldn't have if you solve it from scratch.

Seniors are going to need to hold Juniors to a high bar for understanding and explaining what they are committing. Otherwise it will become totally soul destroying to have a bunch of juniors submitting piles of nonsense and claiming they are blocked on you all the time.

I.e. senior review is valuable, but it does not make bad code good.

I suspect that isn't the goal.

Review by more senior people shifts accountability from the Junior to a Senior, and reframes the problem from "Oh dear, the junior broke everything because they didn't know any better" to "Ah, that Senior is underperforming because they approved code that broke everything."

Right, code reviews should already have been happening with human written junior code.

If AI is a productivity boost and juniors are going to generate 10x the PRs, do you need 10x the seniors (expensive) or 1/10th the juniors (cost save).

A reminder that in many situations, pure code velocity was never the limiting factor.

Re: idiot prooofing I think this is a natural evolution as companies get larger they try to limit their downside & manage for the median rather than having a growth mindset in hiring/firing/performance.

This is also why I think we will enter a world without Jr's. The time it takes for a Senior to review the Jr's AI code is more expensive than if the Sr produced their own AI code from scratch. Factor in the lack of meetings from a Sr only team, and the productivity gains will appear to be massive.

Whether or not these productivity gains are realized is another question, but spreadsheet based decision makers are going to try.

It could create the right sort of incentives though. If I'm a junior and I suddenly have to take my work to a senior every time I use AI, I'm going to be much more selective about how I use it and much more careful when I do use it. AI is dangerous because it is so frictionless and this is a way to add friction.

Maybe I don't have the correct mental model for how the typical junior engineer thinks though. I never wanted to bug senior people and make demands on their time if I could help it.

Why only AI generated code? I wouldn’t let a junior or mid level developer’s code go into production without at least verifying the known hotspots - concurrency, security, database schema, and various other non functional requirements that only bite you in production.

I’m probably not going to review a random website built by someone except for usability, requirements and security.

Deming's point 3 (of 14): Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.

I seriously doubt that they think senior reviewers will meticulously hunt down and fix all the AI bugs. Even if they could, they surely don't have the time. But it offers other benefits here:

1. They can assess whether the use of AI is appropriate without looking in detail. E.g. if the AI changed 1000 lines of code to fix a minor bug, or changed code that is essential for security.

2. To discourage AI use, because of the added friction.

What a statement at the end. You are absolutely right.

I hear “x tool doesn’t really work well” and then I immediately ask: “does someone know how to use it well?” The answer “yes” is infrequent. Even a yes is often a maybe.

The problem is pervasive in my world (insurance). Number-producing features need to work in a UX and product sense but also produce the right numbers, and within range of expectations. Just checking the UX does what it’s supposed to do is one job, and checking the numbers an entirely separate task.

I don’t many folks that do both well.

The goal of Sr code review is not to make the code better, it's to make the author better.

>requires an amount of time approaching the time spent if they had just done it themselves.

I would actually say having at least 2 people on any given work item should probably be the norm at Amazon's size if you also want to churn through people as Amazon does and also want quality.

Doing code reviews are not as highly valued in terms of incentives to the employees and it blocks them working on things they would get more compensation for.

The unwritten thing is that if you need seniors to review every single change from junior and mid-level engineers, and those engineers are mostly using Kiro to write their CRs, then what stops the senior from just writing the CRs with Kiro themselves?

Senior reviews are useful, but as I understand it, Amazon has a fairly high turnover rate, so I wonder just how many seniors with deep knowledge of the codebase they could possibly have.

Other than “don’t hire idiots”, what is the solution to this problem? I agree with you, and this particular systems management issue is not constrained to software.

What stops the senior from using AI to review the AI generated code the junior published?

the outcome of the review isn't just that the code gets shipped, it's knowledge transfer from the senior engineer to the junior engineers that then creates more senior engineers

LGTM

Who said PR reviews need to solve all the things and result in proof against idiots?

So you're saying that peer reviews are a waste of time and only idiots would use/propose them?

The article claims:

>He asked staff to attend the meeting, which is normally optional.

Is that false? It also discusses a new policy:

>Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added.

Is that inaccurate? It is good context that this is a regularly scheduled meeting. But, regularly scheduled meetings can have newsworthy things happen at them.

It didn't seem to make the news but at least in NYC the entire Amazon storefront was broken all afternoon on Friday.

Items weren't displaying prices and it was impossible to add anything to your cart. It lasted from about 2pm to 5pm.

It's especially strange because if a computer glitch brought down a large retail competitor like Walmart I probably would have seen something even though their sales volume is lower.

I am not in that specific meeting but it made me chuckle that a weekly ops meeting will somehow get media attention. It's been an Amazon thing forever. Wait until the public learns about CoEs!

Every "AI did this bad thing" will be a source of celebration and comfort for many. They forget that "AI" itself is a moving target and won't be the same 6 months from now even if it is bad at current thing.

> Feels like the media is making a mountain out of a mole hill here.

That's been their job ever since cable news was invented.

> This meeting happens literally every week, and has for years. Feels like the media is making a mountain out of a mole hill here.

Are you completely missing the point of the submission? It's not about "Amazon has a mandatory weekly meeting" but about the contents of that specific meeting, about AI-assisted tooling leading to "trends of incidents", having a "large blast radius" and "best practices and safeguards are not yet fully established".

No one cares how often the meeting in general is held, or if it's mandatory or not.

Who is the media you're accusing here? This is a twitter post. As far as I can tell they do not work a media company.

What is worth being pointed out is how quickly people blame "The Media" for how people use, consume and spread information on social networks.

The core message of the article is that Amazon has been having issues with AI slop causing operational reliability concerns, and that seems to be 100% accurate.

I believe it is by group - AWS started the weekly operations meeting, effectively every service's oncall from the last week had to attend. Then it grew massive, so they made it optional. Alexa had a similar meeting that tried to replicate what AWS did. A lot of time spent reviewing load tests getting ready for holiday season, prime day, and the superbowl (super bowl ads used to cause crazy TPS spikes for Alexa). And a lot of finger pointing if there was an outage from one team. While it probably did help raise the operational bar, so much time wasted by engineers on busywork/paperwork documenting an error or fix vs improving the actual service.

Juniors could just code things the old fashioned way. It isn't hard. And if they do find it too hard, they aren't cut out for this job.

Accelerate a person speed toward being burned out..

I'm sorry what? Junior engineers can't learn anything without using AI assistants (or is the implication that having seniors review their code makes them incapable of learning?) and senior engineer would hate their jobs reviewing more code from their teammates? What reality do people live in now?

Not only is having too many comments on your PRs bad for you, but so is not leaving comments on other people's PRs. Both are metrics used

Indeed. My view as a CEO is, if you are still reviewing the code yourself then what use is it that you can produce a bunch of text at a faster rate?

I'd prefer people wrote good quality code and checked it as they went along... whilst allowing room for other stuff they didn't think of to come to the front. The production process of using LLMs is entirely different, in its current state I don't see the net benefit.

E.g. if you have a very crystalised vision of what you want, why would I want an engineer to use an LLM to write it, when the LLM can't do both raw production and review? Could this change? Sure. But there's no benefit for me personally to shift toward working that way now - I'd rather it came into existence first before I expose myself to incremental risk that affects business operations. I want a comprehensive solution.

This is what I don't understand about this policy. There's no way a senior has enough spare capacity to be the gate keeper on every PR made by AI below them. So now we are just making it so the senior people use more AI to keep up but now they're to blame for letting it happen.

It sounds like a piss poor deal for seniors unless senior engineer now means professional code reviewer.

> Senior leadership doesn’t know this, though.

Well, you'd think senior leadership should know how their business and their people work.

Surely they know all this. They're worried about AI code degrading codebase quality, so they're putting on the brakes.

Thanks for the links. Strangely I cannot get past the arhive.is "I am not a robot" wall. I click it, then it refreshes, I click it again, and then it asks me to find Traffic Lights, and then "I am not a robot," repeat.

Maybe I need a bot to do this for me...

It's the same as the offshoring episode of the early 2000's. There is such a massive financial incentive to somehow make the low quality code work. And they will try to resist the reality that it's a huge net negative for as long as they can.

So will it turn out that actually writing code was never the time sink in the first place?

That has always been my feeling. Once I really understand what I need to implement, the code is the easy part. Sure it takes some time, but it's not the majority. And for me, actually writing the code will often trigger some additional insight or awareness of edge cases that I hadn't considered.

My prediction is a concorde-like incident is going to shatter trust and make people re-think their expectations of the capabilities of LLMs and their abilities of the present.

Essentially something big has to happen that affects the revenue/trust of a large provider of goods, stemming from LLM-use.

They wont go away entirely. But this idea that they can displace engineers at a high-rate will.

a) what happens if there is change that hasn't been encountered yet so it's not in .agentnotallowed? b) is there a guarantee that something described in these files won't be touched? I've seen examples when agents directly violate these rules, profusely apologising after they get caught on it.

Interesting. How would it be an early step towards an apprenticeship system?

You shouldn't be surprised.

How else would they train the LLM PR reviewers to their standards?

I've never personally been in the position, because my entire career has been in startups, but I've had many friends be in the unenviable position of training their replacements. Here's the thing though, at least they knew they were training their replacements. We could be looking at a potential future where an employee or contractor doesn't realize s/he is actually just hired to generate training data for an LLM to replace them, and then be cut.

LLMs have been garbage for real work until very recently. Doesn't this show they were adopted too soon at amazon?

Right? If they're using some sort of tool, there's always another tool to fool the tool.

Create the problem and then create the solution.

"Why don't they just make the plane out of the black box?"

This is incredibly circular lol...

You mean like what Anthropic announced yesterday ? Code Review can review your code for $15 - $25 per review.

/s

So now, you can speed up using Claude Code and use Code Review to keep it in check.

I used Kiro IDE and really liked it. The all you can eat model of LLM usage is very tempting compared to say Cursor. The features in the editor are basically the same.

Haven't tried Kiro CLI.

makes me want to vomit. I am not spending more time reviewing code than the "author" spent creating it. Ill just leave the industry if that happens.

A lot of juniors only graduated using these tools. Good luck taking it away from them.