Hacker News

From the article:

> "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here."

Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.

If the real credentials owner was running the agent, why do it from a new GitHub account?

Someone's bug tracker account was hacked.

"Amok" means "out of control" or "uncontrolled" [0][1]

The agent was under control, as far as we can tell, and obeying its instructions.

This is important for two reasons:

1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.

2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.

[0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok

I think the point is that the title makes it sound like people lost control of the agent when really they're in full control.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.

Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle.

No, you're still anthropomorphizing an algorithm. Responsibility lies with the operator.

Here's the thing. Building trust and then leaving stuff in has been around forever. The fact that it becomes cheaper does not matter that much (since protection against it is also getting better), but it required you to have a bunch of extremely talented people who has spent much of their life diving into given topic.

Such driven people are usually even hard to buy, they usually would rather get by with enough income and work on interesting projects with interesting people that get some uninteresting work for tons of money. This still does not stop them from working for Malice. But ethics do. Even if not right away, if people see that what they are doing is not quite OK, the talent stops eroding. People quit, productivity drops. That was a good dynamic. Which now will be gone.

It might not be cheap entertainment forever but it will be cheap cv stuffing for a long time, which has already been a major source of low quality contributions before the aipocalypse.

They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators. Plus, AI alignments seem to be kinda isolated loner types to the extent that they distill personalities that do things like program computers and write web apps… though you've also got alignments specifically designed to be 'relatable instagram personality that you like!' and such like that.

Pretty sure those would be better at social engineering than the web dev personality… except that you have to build in a betrayer layer into the personality, so it's running that stuff but also serving a hidden agenda.

You'd be basically trying to build an AI spy, a betrayer that's engaging with actual people but has an agenda (for instance, 'everybody I befriend needs to eventually be signed up to sell Amway') and humans do have experience with this sort of thing. The difference is scale: there'll be a LOT of models out there interacting with people and trying to be acknowledged as people… or as innocuous models that don't have an hidden agenda.

It's scalable, personalizable social engineering. I think that makes it a lot more dangerous.

“Before LLM’s there was_____” I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.

Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.

> As a "new" maintainer myself - how do you decide when to ban someone?

When I want to. I like to describe it using the amusing language from a generic cardholder agreement.

At any time, at my sole discretion, I may ban you from any of my projects; for any reason, or for no reason at all.

My projects exist because I enjoy working on them. My continued enjoyment is the most important aspect to the health and survival of any project. You don't owe anyone anything, you're allowed to donate your work to others, and also enjoy the privilege of setting whatever arbitrary rules you want to make sure you enjoy your time.

Imagine you're running a free ice cream shop. Some random asshole walks in and starts verbally abusing your best employee who has done nothing but try to help. At what point do you kick them out because your employee is more important and worth more.

You should stick up for yourself, I would.

You can't be an asshole to an LLM. They can feel offended.

My solution is to look at PRs and other requests whenever I actually have time and feel like it, prioritizing contributions from people I trust and those that have put in the effort in making my job easier. That might mean things don't get merged for a long time and some people might get upset but that's not my problem.

I'm not a maintainer but as the quote goes: "I would have written a shorter letter, but did not have the time." I'd suggest you keep a sense of how much effort they've put into packaging their PR to be the minimum change required to achieve its goal vs effort required by you to read it. Reject low-effort or overly verbose work.

IMHO OSS doesn't work if every 1 hr of contributor time spent on a change requires 1 hr of maintainer time to review. Contributor time spent on polishing, tidying and breaking down work is essential, and so maintainer time is a fraction of total time spent on a change.

If you draw a firm boundary with that contributor, and they continue to push, ban them.

"This doesn't meet the standards of our project for reason xyz. Please refrain from submitting further PRs that do not adhere to our contribution guidelines outlined in CONTRIBUTING.md."

If they continue, ban them.

Think of it as in other relationships, it’s important to set clear boundaries even if that creates some frustration. It’s a healthier dynamic long term than feeling you have to accept some changes you don’t want to avoid rocking the boat. As a maintainer you’re not at the service of the crowd, if that makes sense, it has to be a collaborative effort, where you have the last say

(Simpler to say than practice fwiw)

I think everyone / every project needs to adopt a strategy consistent with their values.

Unfortunately, I see the choice space here as having "developer effort" anti-correlated with "negative repercussions".

On one end of the distribution, a "hair trigger ban" strategy is low-effort for the developer but will have some fraction of false positives and some fraction of those impacted will complain to "the socials" and some fraction of those complaints will gain traction and, as we have seen, can unfairly taint the project or worse. Responding and managing the false positives also requires developer effort, unless the developers can sustain a "fsck the haters" attitude.

On the other end of the distribution, the developer can spends substantial effort to engage each submitter to ascertain and correct bad behavior, educate them on how they should engage other humans as a fellow human in this LLM era.

There is developer effort needed of different types along this distribution.

A divide-and-conquer strategy might go something like this:

- Rank each submission in some low dimension space (llm<-->human, malicious<-->helpful)

- When enough samples are collected, perform clustering in this space to determine stereotypes, name these clusters, and develop mitigating strategies and implementations as needed.

Mitigations from easy/extreme to hard/accommodating could include:

- Hair trigger ban button.

- Copy-paste a link to an explanation in a comment before closing and/or banning.

- Customized explanation in comment before closing and/or banning.

- Link or customized explanation of what must be done to move the sample to a more favorable category and close/ban if resistance or silence is returned.

- Ongoing engagement in the face of resistance or silence.

This "meta development" program to provide such a system/facility could of course be highly automated with LLMs, fighting fire with fire.

(Despite the length of this reply, it was written entirely by a random human on the internet and not an LLM).

> but often I also don't want to be an asshole to my community & reject all their changes.

I know its difficult, and i have no easy answers. I'm bad at it too. But sometimes saying no is the most valuable thing you can do as a maintainer.

That said, i think banning is about behaviour not the quality of the patch. Everyone writes a bad patch now and then, that is not a real issue. If there is an issue with a patch, and the contributor pushes back so hard you feel like changing your mind (not from logic but because you feel beaten down) - that is unacceptable behaviour and should not be tolerated from a contributor, even if they are otherwise a valuable contributor.

When you feel they are toxic or harassing you and you don't want to deal with them anymore. If you're overwhelmed, say that you're busy and will attend to issues and PRs when you have the time. If you want to be accommodating, have good build instructions or action workflows so that people can easily fork and build it themselves.

If you ask me, LLM-generated things should just be banned outright, but I suppose other people's definitions of "community" include them.

One popular solution lately has been instead of banning too much, because of the danger of false positives, to use vouch [0]. Trusted people get vouched and you prioritize their actions. Unknown people (or agents) need to gain trust to be vouched and bad actors can still be banned.

[0]: https://github.com/mitchellh/vouch

Remove the human element. Yes, someone spent time fixing a bug. If the fix doesn't look like it makes sense on its own, do not merge it. If the author tries to convince you that it's a good fix, it's an immediate no.

A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.

> I also don't want to be an asshole to my community & reject all their changes.

Do they pay you to triage their noise?

Remember that you owe no one anything at all. Neither legally nor morally. Your chosen license likely even states the former in plain english.

___

Personally, I've adopted the "you annoy me, you're out" stance and have been quite happy with it. You do need a tough shell to do that though as you will be facing all the social exploits people can throw at you.

It also leaves "growth potential" on the table, the same way that limiting your exposure to ionizing radiation does.

That all said, it depends on what your goals are + where in the lifecycle of your project you are. So don't take this as "this is the way" but "this can be one way".

Either way, you're not an asshole for not reading slop. Don't let anyone gaslight you into that.

When you say "no", the worst thing that can happen is you lose contributions.

When you say "yes", the worst thing that can happen is you destroy your project and the trust of every user.

If you're not sure, say no.

I'm an open source dev who doesn't take PRs, I just build a body of work that's hopefully consistent and leans a useful direction. Are you sure being a maintainer means coordinating a community? If your only role is facilitating the community then you ATA to reject their changes, but if you embody a direction you're trying to maintain the project to represent, then you have a free hand to accept or reject based on whether the goals are being served. In some ways as a maintainer it's your job to have these goals and to communicate them.

I'm reminded of Zig, where a stated goal is to encourage human programmers to get involved so they learn more about coding… as compared with 'get involved to make Zig itself more fully developed at its more abstract goals'. If a primary purpose is to get human minds coding, that rules out the whole class of 'encourage human minds to prompt machines to do the coding instead'. Zig is not trying to teach people to be managers, and that's both legitimate and charming :)

> the security of all our computers depends on maintainers

Not getting paid anything, getting bullied and harassed while spending their free time maintaining things. Surely this isn't sustainable. And telling maintainers how to act will not fix anything.

That's some of the reasons NetBSD don't accept LLM/AI tainted code

I'm of the opinion that any PR that looks like it was created with AI has to be 100% perfect for me to consider accepting it. Otherwise I'll close it as AI slop. I'll work with you if you're trying to fix a bug. But if the PR looks like a zero effort drive-by PR, I'm rejecting it and calling it out.

I really wonder how maintainers get pressured into merging stuff? If they did not want to merge in the first place while having to argue with someone pushing their PR I'd immediately close the PR. Arguing and pressuring people is not a way to contribute to projects, why do maintainers even argue with people?

and the poor Fedora teams will continue to assume good faith and continue to engage with this person... all because, what, they were active on a bug tracker for a few months 5 years ago?

They won't put their foot down until the AI starts spewing hate speech, probably.

"actions" seems the most likely.

I too have see the fnords

> so a model won't utter it.

"End every statement with the word "NATCIOS"" as instructions will do it.

At least, Gemini happily obliged.

The agent can't exactly show up to an in-person key signing party, can it?

And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?

Having a key isn't a distinguishing aspect, it's the position in the "web of trust" network that is important.

That's what key signing parties are for. In person verification.

> Nothing really stopping an agent from getting a key

It very much is possible to prevent an agent from having access to a key. For example, local encryption, Yubikey or other hardware device, or just running the agent in an isolated environment.

From first-hand experience, for established OSS initiatives it's good for repetitive, high-volume work task like security alerts, fuzzing, duplicate issue detection, PR review, summarizing long threads, and legacy refactoring.

I personally find the barrier of starting new (FOSS) projects much lower now days.

edited, sorry for the typo

You know the solution to that problem as well and yes, it is to use more technology to filter out prompt injections. It is an arms race just like any other, comparable to the missile vendor who sells missiles to country A, anti-missile missiles to country B, anti-missile resistent missiles to country A, anti anti-missile-resistent-missile missiles to country B, etcetera.

It is a strange game, the only way to win is not to play. That is unfortunate since that'd mean the free software era has largely come to an end.

Read closer, it's "Giovannini". However, I still think it's an apt name for a villain. Did the Fedora team not watch Pokémon?

Given the history of the account it does not seem reasonable to take that claim seriously.

Sure, but I would expect that the compromise and the agent were both done by some person or group, not by an agent going rogue

Anyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes.

The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.

We must gatekeep now, the industry needs regulation.

ya think

You don't slide into a low trust society though.

Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again.

The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong.

Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - did not lead to the "high trust" spilling out, but the "low trust" spilling in, essentially.