Skip to writing
Peter Cellino — The Working Journal
Notes

It Would Be Prudent

Railway lost control of the place where its customers' keys are kept, then told me it would be prudent to change everything. I spent Friday afternoon doing it. Then I left.

PC
Peter Cellino· Publisher, The Charlotte Mercury
||3 min read

It would be prudent.

You learn a lot about a company from the mood of its verbs. Nobody says "it would be prudent" about something they plan to do themselves. It is the voice of advice, and advice is cheap because the person giving it is not the one who pays.

Railway, which hosts other people's software, said it to me by email on Friday. Someone had gotten into the place where its customers' passwords and keys are kept. The company could not say whether mine were among them. It would be prudent, it said, to change everything. It was sorry for the burden it created.

So I spent Friday afternoon changing everything, account by account, and then I shut down what I had running there. That is the whole story. But the paper trail behind that email, two posts and an earlier email over three days, is a small masterpiece of a particular form, and the form deserves a closer look. It is the art of telling a customer that your problem is now his.

The first notice went up Wednesday, Oct. 21, and a copy reached my inbox that night. An unauthorized third party "may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents." Environment variables, in plain language, are the keys. Customers should "immediately roll all environment variables, including any API keys."

Look at who gets the adverb. The company lost control of the keys. The customer is the one told to do something immediately.

A locksmith who loses the master key to your building does not send a notice saying it would be prudent to change your locks. He changes them. He does it that day, and he does not bill you, because the alternative is not being a locksmith anymore. Railway did some of this. It called the vendors it could name and had those keys revoked. Then it told every customer to do the rest, immediately. This is not a high standard. It is the standard that comes with holding other people's keys, and it is the entire reason anyone pays you to hold them.

Thursday brought the preliminary report. The attack ran from Oct. 20 to Oct. 22. Up to "about 100,000 workspaces" may have been exposed. The attacker "was likely looking for AI and platform API keys."

The thief went to the place where the keys are kept, looking for keys. That is the finding.

Then Friday, and the email. The subject line was "Urgent Railway security update: Your workspace may be at risk." Urgent, and may be. The two words that matter in that line are arguing with each other. Inside: "We cannot confirm whether credentials in your workspace were accessed."

This is true, and it is the worst kind of true. The one party with the logs is telling you the logs do not settle it. They are handing you uncertainty, and they are the only people who could have taken it off your hands.

Then the sentence I started with. Then the apology, for "the burden it created."

Give the apology its due. Wednesday's notice had been sorry for "this inconvenience," meaning the paused builds. Friday's is sorry for the incident, and for what it cost me, and it says so. Read it once more, though. The burden it created. It. The incident did the creating. The company that had the incident appears in the sentence only as the party that is sorry. It is a package set on my porch with the sender's name left off.

I do not know whether anyone took anything of mine. I am not going to guess, because the people in a position to know have said they cannot, and their guess would be worth more than mine. What I know is the afternoon. Every password, every key, every account, one at a time. I know which of us spent it.

The report, to be fair, has a section called "Railway actions taken," and it is a real list: keys rolled, systems rebuilt, outside investigators brought in. The company can write in the past tense, with a subject in front of each verb, when the subject is itself. The email to me was a different document. Its list was what I should do, under a line that said it would be prudent.

Railway chose the conditional. It would be prudent. It may have been. We cannot confirm. The burden it created.

I chose the indicative. I moved out.

Topics: Editorial · Platform Thesis

PC
Peter Cellino

Publisher, The Charlotte Mercury

Peter Cellino is the publisher of The Charlotte Mercury and founder of Mercury Local, the platform that runs it. He writes on agentic AI, platform economics, and the future of local journalism.

More in Notes