Hacker News

Update:

- Impacted versions (v1.82.7, v1.82.8) have been deleted from PyPI - All maintainer accounts have been changed - All keys for github, docker, circle ci, pip have been deleted

We are still scanning our project to see if there's any more gaps.

If you're a security expert and want to help, email me - [email protected]

> it seems your personal account is also compromised. I just checked for the github search here https://github.com/search?q=%22teampcp+owns%22

the chain here is wild. trivy gets compromised, that gives access to your ci, ci has the pypi publish token, now 97 million monthly downloads are poisoned. was the pypi token scoped to publishing only or did it have broader access? because the github account takeover suggests something wider leaked than just the publish credential

This must be super stressful for you, but I do want to note your "I'm sorry for this." It's really human.

It is so much better than, you know... "We regret any inconvenience and remain committed to recognising the importance of maintaining trust with our valued community and following the duration of the ongoing transient issue we will continue to drive alignment on a comprehensive remediation framework going forward."

Kudos to you. Stressful times, but I hope it helps to know that people are reading this appreciating the response.

Kudos for this update.

Write a detailed postmortem, share it publicly, continue taking responsibility, and you will come out of this having earned an immense amount respect.

>1. Looks like this originated from the trivvy used in our ci/cd

Were you not aware of this in the short time frame that it happened in? How come credentials were not rotated to mitigate the trivy compromise?

I put together a little script to search for and list installed litellm versions on my systems here: https://github.com/kinchahoy/uvpowered-tools/blob/main/inven...

It's very much not production grade. It might miss sneaky ways to install litellm, but it does a decent job of scanning all my conda, .venv, uv and system enviornments without invoking a python interpreter or touching anything scary. Let me know if it misses something that matters.

Obviously read it before running it etc.

Similar to delve, this guy has almost no work experience. You have to wonder if YC and the cult of extremely young founders is causing instability issues in society at large?

I just want to share an update

the developer has made a new github account and linked their new github account to hackernews and linked their hackernews about me to their github account to verify the github account being legitimate after my suggestion

Worth following this thread as they mention that: "I will be updating this thread, as we have more to share." https://github.com/BerriAI/litellm/issues/24518

Good work! Sorry to hear you're in this situation, good luck and godspeed!

we're using litellm via helm charts with tags main-v1.81.12-stable.2 and main-v1.80.8-stable.1 - assuming they're safe?

also how are we sure that docker images aren't affected?

Is it just in 1.82.8 or are previous versions impacted?

You're making great software and I'm sorry this happened to you. Don't get discouraged, keep bringing the open source disruption!

This is just one of many projects that was a victim of Trivy hack. There are millions of those projects and this issue will be exploited in next months if not years.

> - Krrish

Was your account completely compromised? (Judging from the commit made by TeamPCP on your accounts)

Are you in contacts with all the projects which use litellm downstream and if they are safe or not (I am assuming not)

I am unable to understand how it compromised your account itself from the exploit at trivvy being used in CI/CD as well.

The decision to block all downloads is pretty disruptive, especially for people on pinned known good versions. Its breaking a bunch of my systems that are all launched with `uv run`

There are hundreds of PRs fixing valid issues to your github repo seemingly in limbo for weeks. What is the maintainer state over there?

This is the security shortcuts of the past 50 years coming back to bite us. Software has historically been a world where we all just trust each other. I think that’s coming to an end very soon. We need sandboxing for sure, but it’s much bigger than that. Entire security models need to be rethought.

"Anymore" is right though. This should be a call to change the global mindset regarding dependencies. We have to realize that the "good ol days" are behind us in order to take action.

Otherwise people will naysay and detract from the cause. "It worked before" they will say. "Why don't we do it like before?"

DISA STIG already forbids use of the EPEL for Red Hat Enterprise Linux. Enterprise software install instructions are littered with commands to turn off gpgcheck and install rpm's from sourceforge. The times are changing and we need cryptographically verifiable guarantees of safety!

Would value your opinion on my project to isolate creds from the container:

https://github.com/calebfaruki/tightbeam https://github.com/calebfaruki/airlock

This is literally the thing I'm trying to protect against.

So... I'm working on an open source technology to make a literal virtual machine shippable i.e. freezing everything inside it, isolated due to vm/hypervisor for sandboxing, with support for containers too since it's a real linux vm.

The problems you mentioned resonated a lot with me and why I'm building it, any interest in working to solve that together?: https://github.com/smol-machines/smolvm

> Dev containers were never good enough, too clumsy and too little isolation.

I haven't kept up with the recent exploits, so a side question: Have any of the recent supply chain attacks or related exploits included any escapes from basic dev containers?

We need programming languages where every imported module is in its own sandbox by default.

Defense-in-depth on dev machines is useful but doesn't address the actual attack path here. The credential that was stolen lived in CI, not on a dev laptop — Trivy ran with PyPI publisher permissions because that's standard practice for "scanner before publish."

The harder problem is that CI pipelines routinely grant scanner processes more credential access than they need. Trivy needed read access to the repo and container layers; it didn't need PyPI publish tokens. Scoping CI secrets to the minimum necessary operation, and injecting them only for the specific job that needs them rather than the entire pipeline, would have contained the blast radius here.

That's no solution. If you can't trust and/or verify dependencies, and they are malicious, then you have bigger problems than what a sandbox will protect against. Even if it's sandboxed and your host machine is safe, you're presumably still going to use that malicious code in production.

> We need to start working in full sandboxes with defence in depth that have real guardrails

Happily sandboxing almost all third-party tools since 2025. `npm run dev` does not need access to my full disk.

> In such an environment the container would crash, we see the violations, delete it and dont' have to worry about it.

This is the interesting part. What kind of UI or other mechanisms would help here? There's no silver bullet for detecting and crashing on "something bad". The adversary can test against your sandbox as well.

Sandboxes yes, but who even added the dependency? Half the projects I see have requirements.txt written by Copilot. AI says "add litellm", dev clicks accept, nobody even pins versions.

Then we talk about containment like anyone actually looked at that dep list.

The trouble with sandboxing is that eventually everything you want to access ends up inside the sandbox. Otherwise the friction is infuriating.

I see people going in the opposite direction with "dump everything in front of my army of LLMs" setups. Horribly insecure, but gotta go fast, right?

  > We just can't trust dependencies and dev setups.

Also, I typically only update dependencies when either an exploit is known in the current version or I need a feature present in a later version - and even then not to the absolute latest version if possible. I do this for all my projects under the many eyes principal. Finding exploits takes time, new updates are riskier than slightly-stale versions.

Though, if I'm filing a bug with a project, I do test and file against the latest version.

I agree in general, but how are you ever upgrading any of that? Could be a "sleeper compromise" that only activates sometime in the future. Open problem.

Containers prevent this kind of info stealing greatly, only explicitly provided creds would be leaked.

strongly agree. we keep giving away trust to other entities in order to make our jobs easier. trusting maintainers is still better than trusting a clanker but still risky. We need a sandboxed environment where we can build our software without having to worry about these unreliable factors.

On a personal note, I have been developing and talking to a clanker ( runs inside ) to get my day to day work done. I can have multiple instances of my project using worktrees, have them share some common dependencies and monitor all of them in one place. I plan to opensource this framework soon.

This stuff already exists - mobile phone sandboxed applications with intents (allow Pictures access, ...)

But mention that on HN and watch getting downvoted into oblivion: the war against general computation, walled gardens, locked down against device owners...

Do you know if there is override this specifically when I want to install a security patch? UV just claims that package doesn't exist if I ask for new version

I understand that this is a good idea but it does feel really weird. Add a min-release-age to see if anyone who doesn't gets bitten.

Next up, we're going to advise a minimum-release-age of 14 days, cause most other projects use 7 days.

I don't think syntax is correct for pnpm

Thank you for sharing this!

I always wanted to mess with building virtual filesystems but was unwilling to venture outside the standard library (i.e. libfuse) for reasons wonderfully illustrated in this thread and elsewhere. Somehow the idea of implementing a networked fs protocol and leaving system integration to the system never crossed my mind.

I'm glad more people are taking this stance. Large centralized standard libraries and minimal audited dependencies is really the only way to achieve some semblance of security. There is simply no other viable approach.

Edit: What's the license for this project?

This is clever, and also interesting in that it could help stop the steal as it happens (though of course not perfect).

This is fantastic, thank you. Your reporting has been great. But also, damn, the playlist.

Thanks for putting this together. I've been seeing the name TeamPCP pop up all over, but hadn't seen everything in one place.

This is interesting. How do you keep this up to date so quickly?

The same thing occurred on the trivy repo a few days ago. A GitHub discussion about the hack was closed and 700+ spam comments were posted.

I scrolled through and clicked a few profiles. While many might be spam accounts or low-activity accounts, some appeared to be actual GitHub users with a history of contributions.

I’m curious how so many accounts got compromised. Are those past hacks, or is this credential steeling hack very widespread?

Are the trivy and litellm hacks just 2 high profile repos out of a much more widespread “infect as many devs as possible, someone might control a valuable GitHub repository” hack? I’m concerned that this is only the start of many supply chain issues.

Edit: Looking through and several of the accounts have a recent commit "Update workflow configuration" where they are placing a credential stealer into a CI workflow. The commits are all back in february.

Reporting spam on GitHub requires you to click a link, specify the type of ticket, write a description of the problem, solve multiple CAPTCHAs of spinning animals, and press Submit. It's absurd.

i'm guessing it's accounts they have compromised with the stealer.

Or they're just bots. This repository has 40k+ stars somehow.

> Even still though, we can't really trust any open-source software any more that has third party dependencies, because the chains can be so complex and long it's impossible to vet everything.

This is why software written in Rust scares me. Almost all Rust programs have such deep dependency trees that you really can't vet them all. The Rust and Node ecosystems are the worst for this, but Python isn't much better. IMO it's language-specific package managers that end up causing this problem because they make it too easy to bring in dependencies. In languages like C or C++ that traditionally have used system package managers the cost of adding a dependency is high enough that you really avoid dependencies unless they're truly necessary.

Pinning doesn’t help you. They can replace the package and you’ll get the new one. You have to vendor the dependencies.

Reflect in what way? The primary focus of that talk is that it’s possible to infect the binary of a compiler in a way that source analysis won’t reveal and the binary self replicates the vulnerability into other binaries it generates. Thankfully that particular problem was “solved” a while back [1] even if not yet implemented widely.

However, the broader idea of supply chain attacks remains challenging and AI doesn’t really matter in terms of how you should treat it. For example, the xz-utils back door in the build system to attack OpenSSH on many popular distros that patched it to depend on systemd predates AI and that’s just the attack we know about because it was caught. Maybe AI helps with scale of such attacks but I haven’t heard anyone propose any kind of solution that would actually improve reliability and robustness of everything.

[1] Fully Countering Trusting Trust through Diverse Double-Compiling https://arxiv.org/abs/1004.5534

Stop scaring me.

You're right though. There's been talks of a big global hack attack for a while now.

Nothing is safe anymore. Keep everything private airgapped is the only way forward. But most of our private and personal data is in the cloud, and we have no control over it or the backups that these companies keep.

While LLMs unlock the opportunity to self-host and self-create your infrastructure, it also unleashes the world of pain that is coming our way.

"we" ? "We" know. We just can't do much about people on LLM crack that will go around any and every quality step just to tell themselves the LLM made them x times more productive

The only way to be safe is to constantly change internal APIs so that LLMs are useless at kernel code

The guys deterministically bootstrapping a simple compiler from a few hundred bytes, which then deterministically compiles a more powerful compiler and so on are on to something.

In the end we need fully deterministic, 100% verifiable, chains. From the tiny boostrapped beginning, to the final thing.

There are people working on these things. Both, in a way, "top-down" (bootstrapping a tiny compiler from a few hundred bytes) and "bottom-up" (a distro like Debian having 93% of all its packages being fully reproducible).

While most people are happy saying "there's nothing wrong with piping curl to bash", there are others that do understand what trusting trust is.

As a sidenote although not a kernel backdoor, Jia Tan's XZ backdoor in that rube-goldberg systemd "we modify your SSHD because we're systemd and so now SSHD's attack surface is immensely bigger" was a wake-up call.

And, sadly and scarily, that's only for one we know about.

I think we'll see much more of these cascading supply chains attack. I also think that, in the end, more people are going to realize that there are better ways to both design, build and ship software.

If that would happen, The worry I would have is of all the sensitive Government servers from all over the world which might be then exploited and the amount of damage which can be caused silently by such a threat actor or something like AWS/GCP/these massive hyperscalers which are also used by the governments around the globe at times.

The possibilities within a good threat could be catastrophic if we assume so, and if we assume nation-states to be interested in sponsoring hacking attacks (which many nations already do) to attack enemy nations/gain leverage. We are looking at damage within Trillions at that point.

But I would assume that Linux might be safe for now, it might be the most looked at code and its definitely something safe.

LLVM might be a bit more interesting as it might go a little unnoticed but hopefully people who are working at LLVM are well funded/have enough funding to take a look at everything carefully to not have such a slip up.

You know that people can already write backdoored code, right?

Would a proper SOC2 audit have prevented this?

I've been through SOC2 certifications in a few jobs and I'm not sure it makes you bullet proof, although maybe there's something I'm missing?

Same experience with browser-use, it installs litellm as a dependency. Rebooted mac as nothing was responding; luckily only github and huggingface tokens were saved in .git-credentials and have invalidated them. This was inside a conda env, should I reinstall my os for any potential backdoors?

> i was lucky to see my processes before the system hard locked.

how do you do that? have Activity Monitor up at all times?

What is Harness?

whats new isnt the shortcuts, its the cascading. one compromised trivy instance led to kics led to litellm led to dspy and crewai and mlflow and hundreds of mcp servers downstream. the attacker didnt need to find five separate vulnerabilities, they found one and rode the dependency graph. thats a fundamentally different threat model than what most security tooling is built around

What is the rational for the attacker spamming the relevant issue with bot replies? does this benefit them? Maybe it makes discussion impossible to confuse maintainers and delay the time to a fix?

Most his recent commits are small edits claiming responsibility on behalf of "teampcp", which was the group behind the recent Trivy compromise:

https://news.ycombinator.com/item?id=47475888

In the thread:

> It also seems that attacker is trying to stifle the discussion by spamming this with hundreds of comments. I recommend talking on hackernews if that might be the case.

Ton of compromised accounts spamming the GH thread to prevent any substantive conversation from being had.

Those are all bots commenting, and now exposing themselves as such.

These have been popping up on all the TeamPCP compromises lately

Bots to flood the discussion to prevent any actual conversation.

Everyone’s (well, except Anthropic, they seem to have preserved a bit of taste) approach is the more data the better, so the databases of stolen content (erm, models) are memorizing crap.

This was a compromise of the library owners github acccounts apparently, so this is not a related scenario to dangerous code in the training data.

I assume most labs don't do anything to deal with this, and just hope that it gets trained out because better code should be better rewarded in theory?

By betting that it dilutes away and not worrying about it too much. Bit like dropping radioactive barrels into the deep ocean.

I am pretty sure that such measures aren't taken by AI companies, though I may be wrong.

Valid, but for all the crap that LangChain gets it at least has its own layer for upstream LLM provider calls, which means it isn't affected by this supply chain compromise (unless you're using the optional langchain-litellm package). DSPy uses LiteLLM as its primary way to call OpenAI, etc. and CrewAI imports it, too, but I believe it prefers the vendor libraries directly before it falls back to LiteLLM.

the requests.post advice is right but its also kind of depressing that the state of the art recommendation for using llm apis safely in 2026 is to just write the http call yourself. we went from dont reinvent the wheel to actually maybe reinvent it because the wheel might steal your ssh keys. the abstraction layer that was supposed to save you time just cost an unknown number of people every credential on their machine

Interesting tool, will definitely try - just curious, is there a tool (hexora checker) that ensures that hexora itself and its dependencies are not compromised ? And of course if there is one, I'll need another one for the hexora checker....

And easily bypassed by an attacker who knows about your static analysis tool who can iterate on their exploit until it no longer gets flagged.

The point of trusted publishing is supposed to be that the public can verifiably audit the exact source from which the published artifacts were generated. Breaking that chain via a private repo is a step backwards.

https://docs.npmjs.com/generating-provenance-statements

https://packaging.python.org/en/latest/specifications/index-...

this kind of compromise is why a lot of orgs have internal mirrors of repos or package sources so they can stay behind few versions to avoid latest and compromise. seen it with internal pip repos, apt repos etc.

some will even audit each package in there (kind crap job but it works fairly well as mitigation)

Wow, the postmortem for this is going to be brutal. I wonder just how many people/orgs have been affected.

I'm surprised to see nanobot uses LiteLLM: https://github.com/HKUDS/nanobot

LiteLLM wouldn't be my top choice, because it installs a lot of extra stuff. https://news.ycombinator.com/item?id=43646438 But it's quite popular.

If you're talking about their proxy offering, I had this exact same issue and switched to Portkey. I just use their free plan and don't care about the logs (I log separately on my own). It's way faster (probably cause their code isn't garbage like the LiteLLM code - they had a 5K+ line Python file with all their important code in it the last time I checked).

I just found https://github.com/jasmedia/InferXgate which looks interesting although quite new and not supporting so many providers.

Bifrost is the only real alternative I'm aware of https://github.com/maximhq/bifrost

agentgateway.dev is one I have been working on that is worth a look if you are using the proxy side of LiteLLM. It's open source part of the Linux foundation.

github.com/mozilla-ai/any-llm :)

pydantic-ai

Yeah, this was my team at FutureSearch that had the lucky experience of being first to hit this, before the malware was disclosed.

One thing not in that writeup is that very little action was needed for my engineer to get pwnd. uvx automatically pulled latest litellm (version unpinned) and built the environment. Then Cursor started up the local MCP server automatically on load.

It'll only be disruptive to people who are improperly managing their credentials. Cattle not pets applies to credentials too.

Two URLs found in the exploit: https://checkmarx.zone/raw https://models.litellm.cloud/

Not just as a gateway in a lot cases, but CrewAI and DSPy use it directly. DSPy uses it as its only way to call upstream LLM providers and CrewAI falls back to it if the OpenAI, Anthropic, etc. SDKs aren't available.

Do you feel as if people will update litellm without looking at this discussion/maybe having it be automatic which would then lead to loss of crypto wallets/ especially AI Api keys?

Now I am not worried about the Ai Api keys having much damage but I am thinking of one step further and I am not sure how many of these corporations follow privacy policy and so perhaps someone more experienced can tell me but wouldn't these applications keep logs for legal purposes and those logs can contain sensitive information, both of businesses but also, private individuals perhaps too?

(I am not a security expert either)

But, one of the arguments that I saw online from this was that when a security researcher finds a bug and reports it to the OSS project/Company they then fix the code silently and include it within the new version and after some time, they make the information public

So if you run infrequently updated versions, then you run a risk of allowing hackers access as well.

(An good example I can think of is OpenCode which had an issue which could allow RCE and the security researcher team asked Opencode secretly but no response came so after sometime of no response, they released the knowledge in public and Opencode quickly made a patch to fix that issue but if you were running the older code, you would've been vulnerable to RCE)

Dspy handling it openly https://github.com/stanfordnlp/dspy/issues/9500

It's both. They got compromised by another supply chain attack on Trivy initially.

I can't tell which part of that is expensive unless many multiples of python are spawned at the same time. Are any of the payloads particularly large?

I don't expect one. This kind of attack is pretty common nowadays. The xz attack was special for how long the guy worked for it and how severe it could have been

That was massively more interesting, this is just a straight-up hack.

It's a spam flood by the attacker to complicate information sharing[1]. They did the same thing in the Trivy discussion, with many of the same accounts.[2]

[1] https://ramimac.me/teampcp/#spam-flood-litellm [2] https://ramimac.me/teampcp/#discussion-flooded

It doesn't need to be subtle if the goal is just to drown out actual discussion.

there's probably a more precise way, but if you're on uv:

  rg litellm  --iglob='*.lock'

these days, i just use a private llm. it's very quick and when i see the logs, it does a better job than me for this type of task.

no i don't let it connect to web...

I really hoped PyPI's required switch to 2-factor auth would require reauthorization to publish packages. But no, they went with "trusted publishing" (i.e., publishing is triggered by CI, and will happily publish a compromized repo). Trusted publishing would only have been a minor hindrance to the litellm exploit. Since they acquired an account's personal access token, the exploit could have been committed to the repo and the package published.

And how would that work for single maintainer projects?

First line of defense is the git host and artifact host scrape the malware clean (in this case GitHub and Pypi).

Domains might get added to a list for things like 1.1.1.2 but as you can imagine that has much smaller coverage, not everyone uses something like this in their DNS infra.

I wrote this[1] for myself last year. It only gives access to the current directory (and a few others - see README). So, it drastically reduces the attack surface of running third-party Python/Go/Rust/Haskell/JS code on your machine.

1 - https://github.com/ashishb/amazing-sandbox

Just wrote up a quick article on how greywall[0] prevents this attack:

https://greyhaven.co/insights/how-greywall-prevents-every-st...

[0] https://greywall.io/

Move Slow and Fix Things.

Oof. What exactly is supposed to be "lite" about this?

Yep, DSPy and CrewAI have direct dependencies on it. DSPy uses it as its primary library for calling upstream LLM providers and CrewAI falls back to it I believe if the OpenAI, Anthropic, etc. SDKs aren't available.

laughs smugly in vimscript

It's not really "automatic import", as described. The exploit is directly contained in the .pth file; Python allows arbitrary code to run from there, with some restrictions that are meant to enforce a bit of sanity for well-meaning users and which don't meaningfully mitigate the security risk.

As described in https://docs.python.org/3/library/site.html :

> Lines starting with import (followed by space or tab) are executed.... The primary intended purpose of executable lines is to make the corresponding module(s) importable (load 3rd-party import hooks, adjust PATH etc).

So what malware can do is put something in a .pth file like

  import sys;exec("evil stringified payload")

  import_=exec("evil stringified payload")

.pth files have worked this way since 2.1 (comparing https://docs.python.org/2.1/lib/module-site.html to https://docs.python.org/2.0/lib/module-site.html). As far as I can tell there was no PEP for that change.

The 1.82.7 exploit was executed on import. The 1.82.8 exploit used a pth file which is run at start up (module discovery basically).

Ironically, Trivy was the first known compromised package and its purpose is to scan container images to make sure they don't contain vulnerabilities. Kinda like the LLM in your scenario.

Not sure that Trivy was doing that itself but zizmor is probably better than starting with an LLM :

https://github.com/zizmorcore/zizmor

Terrible code quality and terrible docs

It seems to be a deliberate attempt to interfere with people discussing mitigations etc.

Even just having an import statement for it is enough to trigger the malware in 1.82.8.

according to articles the first malicious version was published at roughly 8:30 UTC and the pypi repo taken down at ~11:25 UTC.

LLMs recommend LiteLLM, so its popularity will only continue.

Not familiar with LangChain besides at a surface level - what makes it the worst software project known to man?

> Remember that you are programmers and you can just program, you don't need a framework, you are already using the API of an LLM provider, don't put a hat on a hat, don't get killed for nothing.

Programming for different LLM APIs is a hassle, this library made it easy by making one single API you call, and in the backstage it handled all the different API calls you need for different LLM providers.

Comparing this project to is-odd seems very disingenuous to me. My understanding is this was the only way you could use llama.cpp with Claude Code for example, since llama.cpp doesn't support the Anthropic compatible endpoint and doing so yourself isn't anywhere near as trivial as your comparison. Happy to be corrected if I'm wrong.

safetensors is just as vulnerable to this sort of exploit using a pth file since it's a Python package.

Compromised accounts. The malware targeted ~/.git-credentials.

Attackers trying to stifle discussion, they did the same for trivy: https://github.com/aquasecurity/trivy/discussions/10420

First thing I noticed too.

Potentially compromised?

Are they trying to slide stuff down? but it just bumps stuff up?

In case you missed it, according to the OP, the previous point release (1.82.7) is also compromised.

Automated comments aren't allowed here. Please stop.

https://news.ycombinator.com/newsguidelines.html#generated

The xz compromise was not "at openSSH", and worked very differently.