Railway Security Disclosure (blog.railway.com)

295 points by passenger 3 hours ago

vldszn 2 hours ago

Crazy day for cybersecurity: PHP RCE, eMudhra certificates, and now Railway.

And it’s all just getting started imo.

https://blog.cloudflare.com/emudhra-certificate-compromise

https://www.wiz.io/blog/php2shell-php-rce

netvarun 2 hours ago

I think the pingcap hack might be the biggest of them all, somehow it has gone under the radar.

[0] https://news.ycombinator.com/item?id=50173716

llama052 31 minutes ago

I find it wild how much they have downplayed this in their statement. https://www.pingcap.com/trust-hub/security/security-update-o...

”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”

hmm, I don't think that's true.

jeremyjh an hour ago

I guess I picked the wrong week to stop sniffing glue.

yungbeto 33 minutes ago

corford 2 hours ago

Jesus that one is bad

tom1337 2 hours ago

vdfs 2 hours ago

Also Shopify, they have been delisting a lot of apps without giving a reason, most lickely a supply chain attacked that spoofed/stol checkout data

vldszn 2 hours ago

Madness..

SlightlyLeftPad 2 hours ago

And about 50+ sev 9+ CVEs for the linux kernel

bokchoi 2 hours ago

And litellm too

vldszn 2 hours ago

true =(

theturtletalks 3 hours ago

Postiz founder posted on X that his service is being brought down, his keys were already used and Railway was quiet about this until he spoke up:

https://x.com/wickedguro/status/2113015432974405658

lifeisloving 2 hours ago

Why is a 300k mrr company (his tiwtter bio) running on raleway?

jeremyjh an hour ago

You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.

lifeisloving 43 minutes ago

rco8786 2 hours ago

Lots of companies run on railway

SkyPuncher an hour ago

Why not?

posabsolute 2 hours ago

Why are you even commenting? And it's Railway not raleway.

halfmatthalfcat 2 hours ago

fbrncci 3 hours ago

Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…

jazzyjackson 2 hours ago

Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?

jjulius 2 hours ago

Prioritize your mental health, go enjoy your vacation.

fbrncci an hour ago

Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.

boringg an hour ago

bryancoxwell 2 hours ago

Strongly concur

Liuser 2 hours ago

Not your problem. Enjoy your vacation. Bus factor of 1 is poor planning of leadership.

balderdash an hour ago

Either a) your company will reimburse you for your costs incurred and give you a different week off, or b) they don’t need you there that badly.

remywang 21 minutes ago

Services you were responsible for

oofdere 2 hours ago

I say this as a Railway customer, go take your vacation. The only time you should even reconsider is a massive bonus plus more time off for staying.

eecks 2 hours ago

Correct me if I am wrong - if the API keys are compromised, then a large bill could be racked up?

dansquizsoft 2 hours ago

reassess_blind an hour ago

None of this matters if you zoom out.

MattIPv4 3 hours ago

Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.

numbsafari 2 hours ago

Environment variables are a terrible place to store secrets.

I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.

The 12 Factor manifesto was 100% wrong on this point.

Use environment variables to point to where secrets are located, not to store their values.

atherton94027 2 minutes ago

How does this help? If your container gets compromised it's trivial to exfiltrate the secrets.

jrochkind1 an hour ago

It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.

skinfaxi 2 hours ago

> Use environment variables to point to where secrets are located, not to store their values.

Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.

tptacek 2 hours ago

Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.

IanCal 2 hours ago

zaik 2 hours ago

systemd makes it easy: https://systemd.io/CREDENTIALS/

martinald 3 hours ago

I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.

Fwiw it sounds similar to the hack over the weekend that opencode suffered.

0x1F8B 2 hours ago

The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.

Fordec 3 hours ago

I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.

My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.

munificent 3 hours ago

> hot patching will need to itself become fully automated.

Great, even less control over the things we use and more power ceded to corporations.

Fordec 3 hours ago

stabbles 2 hours ago

Hot patching is itself a liability

Fordec 2 hours ago

xtracto 2 hours ago

The internet is over. We are going back to static htmls for public pages, and Point to Point VPN bases connections for B2B service provision.

B2C? too risky to publish services online haha.

mosura an hour ago

Still some industries where you cable colocated servers in a shared DC together where neither server has internet access.

yakz 3 hours ago

It's only just starting. It'll get worse before it gets better.

reconnecting 3 hours ago

I think the only way out is to go back to on-prem for everything, like in the 00s.

LAN cables will go up in price soon.

seanw444 2 hours ago

Ha, Cyberpunk 2077 is becoming reality. We're proposing the Blackwall.

soulofmischief an hour ago

cyberax 2 hours ago

I think we'll see a reduction in complexity and an increase in transparency, rather than just on-prem.

One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.

Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.

It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.

That's it.

You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.

mosura an hour ago

solyin 3 hours ago

How does on-prem help if your product is a SaaS.

lawrenceduk 3 hours ago

codeulike 2 hours ago

80fj0348i 2 hours ago

Sivart13 3 hours ago

surely once there's been enough patches the code will be perfect though?

alpineman 3 hours ago

Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.

fyzix 3 hours ago

How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...

tossstone 2 hours ago

I've already taken my applications offline and will be migrating to a new host...

frio 2 hours ago

Yep. What an incredibly frustrating notice. I like Railway and I guess their hands are tied here, but wow.

eecks 2 hours ago

I rotated mine. If you just change the keys and hit deploy it works

xmcp123 an hour ago

Saw someone on Twitter indicating that they did this, and new keys got compromised.

AJRF 3 hours ago

Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?

Aboutplants 2 hours ago

Not a bad idea

tom1337 2 hours ago

It is… for their business.

DrawTR 3 hours ago

Jeez, what a rough past 24 hours for all security folk. Hang in there

jacquesm 3 hours ago

It's going to be a rough decade.

rectang 3 hours ago

I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.

The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.

faizshah an hour ago

jacquesm 2 hours ago

aaron_m04 2 hours ago

kilroy123 2 hours ago

It already _is_ a rough decade. I have hated the 20s so damn much.

cortesoft 3 hours ago

It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.

rectang 3 hours ago

jacquesm 2 hours ago

cyberpunk 2 hours ago

llama052 2 hours ago

I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.

Then you have real security incidents that you need to tune into constantly.

merlindru 2 hours ago

job security!

minraws 3 hours ago

how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.

Like did someone vibe code too hard, how do env secrets leak for everyone...

9cb14c1ec0 3 hours ago

Easy, all the bad guys need to do is access whatever system stores the decryption keys.

1-6 2 hours ago

It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.

ksajadi 38 minutes ago

someone managed to get the environment variables AND the encryption keys for those? i feel for the secops engineers over there right now. brutal

dylanz an hour ago

I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.

gbnwl 3 hours ago

The past two days have been awful as someone who uses Railway.

eecks 3 hours ago

Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36

MattIPv4 3 hours ago

barelysapient an hour ago

Ugh. Love Railway.

shinycode 2 hours ago

I’m lucky I moved away from it to host my own db with coolify

dgellow 2 hours ago

That’s coming for all platforms unfortunately

alpineman 11 minutes ago

Take coolify admin off the web, put it on a private address only accessible via tailscale

arbayi 2 hours ago

i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.

wateralien 3 hours ago

Is this our new AI daily reality?

pizzafeelsright 8 minutes ago

script kiddies now have access to frontier models. bad

iririririr 3 hours ago

companies probably have php and Python applications exposed to the web and nobody knows.

the speed of irresponsible deployments is infinite now, by design

paxy 3 hours ago

It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.

tokioyoyo 3 hours ago

Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.

It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.

Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.

nrmitchi 2 hours ago

> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”

This isn't in the article? They do not at all claim "the attacked used AI!".

The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.

arcanemachiner 3 hours ago

> your own red teaming should have found them before the attacker did

Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.

gkoberger 3 hours ago

Unless they edited the post, they never said this. You should delete this; it's false information.

mbreese 3 hours ago

I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.

Or did I miss something?

barrrrald 3 hours ago

This one doesn’t?

sergiotapia 3 hours ago

If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.

sergiotapia 2 hours ago

Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.

What is the best VPS to use today? Is it still hetzner or ovh?

lijok 3 hours ago

Damn I guess all those l33t hiring practices were for nothing

rvz 3 hours ago

Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.

Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.

Keloran 3 hours ago

Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not

esseph 3 hours ago

Fuck, I thought this was about trains

goforbg 2 hours ago

the worst service I've signed up for has to be railway. What a terrible product.

torben-friis 2 hours ago

>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.

Is that an accusation? As in, major AI players taking out competitors ?

visch 40 minutes ago

This is exactly why we should own our deployment stacks. Going to push up the launch here as pdt is a railway alternative thttps://github.com/autoidm/pdt for scheduled jobs

AI makes this so much easier to do now, we should all own our own code, and deployment infrastructure (use a cloud instead of someone wrapping aws for you)

skwirl 36 minutes ago

Do you own argcomplete, backoff, certifi, cffi, charset-normalizer, cryptography, google-auth, google-auth-oauthlib, idna, markdown-it-py, mdurl, msal, oauthlib, pyasn1, pyasn1-modules, pycparser, pygments, pyjwt, python-dotenv, pyyaml, requests, requests-oauthlib, rich, rich-argparse, shellingham, and urllib3?

visch 27 minutes ago

No, and this isn't about a supply chain attack on Python libs. This is about running your service(s) on a service that runs in front of a cloud provider. My argument is simply that it's needless extra risk

You do have an argument to be made about vendoring our dependencies, or removing them in general. Not a bad idea