> Why did Railway not notify customers of potentially decrypted keys sooner? ... we wanted to ensure the attacker was ejected from our running infrastructure before telling users to roll their credentials, to prevent the credentials from becoming immediately compromised again.
The way this was handled is messed up to the point it's starting to sound hilarious, they could've easily let people know ASAP that the keys were compromised so they can disable them early and Railway could temporarily block them from updating secrets to keep them secure all around in the meantime
This was an awful write up. I want to know how, over multiple days worth of mitigations, the attacker remained in the system. I want to know exactly, from a detailed technical perspective, what flaws there were, what actions have been taken to fix them, and how you can be positive an attacker won’t gain access in the same manner. This post is so high level that it sounds like railway doesn’t even know the answer to them.
> Forensic investigation evidence indicates the attacker was likely looking for AI and platform API keys, for distilling model weights.
It’s impressive that AI API keys are so valuable that attackers are willing to devote resources to hacking a hosting company just to gain access to them.
Another thing: based the actions they have undertaken, it doesn’t sound like they’ve figured out how they were compromised.
> Why did Railway not notify customers of potentially decrypted keys sooner? ... we wanted to ensure the attacker was ejected from our running infrastructure before telling users to roll their credentials, to prevent the credentials from becoming immediately compromised again.
The way this was handled is messed up to the point it's starting to sound hilarious, they could've easily let people know ASAP that the keys were compromised so they can disable them early and Railway could temporarily block them from updating secrets to keep them secure all around in the meantime
This was an awful write up. I want to know how, over multiple days worth of mitigations, the attacker remained in the system. I want to know exactly, from a detailed technical perspective, what flaws there were, what actions have been taken to fix them, and how you can be positive an attacker won’t gain access in the same manner. This post is so high level that it sounds like railway doesn’t even know the answer to them.
That article's tl;dr seems to be: "we recommend you migrate off of Railway". Happy to oblige, already taken care of.
Just curious, where have you migrated to? Plenty of competitors.
> Forensic investigation evidence indicates the attacker was likely looking for AI and platform API keys, for distilling model weights.
It’s impressive that AI API keys are so valuable that attackers are willing to devote resources to hacking a hosting company just to gain access to them.
Another thing: based the actions they have undertaken, it doesn’t sound like they’ve figured out how they were compromised.
What an absolute mess. Nothing but stress with railway over the past year.