Every engineer has a version of this nightmare. Mine started with a single click in the AWS console: "Empty bucket." Then "Delete bucket." The bucket in question held the Terraform state for roughly twenty different projects, in one shared environment, and I deleted it myself, by accident, on a Tuesday.
If you don't work with IaC (Infrastructure as Code), here's the short version of why that's bad. Terraform state is not your actual infrastructure. Your servers, databases, and load balancers are all still sitting there in AWS, completely unbothered. State is just Terraform's memory of what it built and where. Delete the state, and none of your real infrastructure goes anywhere, but Terraform forgets it ever existed. It's less "the library burned down" and more "somebody stole the card catalog." Every book is still on every shelf. Nobody has any idea which shelf.
I did not know, in the moment, whether this distinction would save me. I found out the hard way that it mostly does, and that "mostly" is doing a lot of work in that sentence.
The Five Stages of Realizing You Broke Something
There's a specific kind of dread that sets in when you realize what you just did. Mine went, roughly: surely it's fine, surely there's a recycle bin, surely AWS support can just... no. No recycle bin. The bucket had versioning enabled, which normally means deleted objects are recoverable. But "Empty bucket" in the console doesn't soft-delete anything. It reaches in and permanently purges every version, on purpose, because that's the whole point of the button. I had, with great efficiency, defeated my own safety net.
So that was gone. Actually gone, not oops check the trash gone.
What wasn't gone: every single resource those twenty projects had ever created. Lambda functions still running. Databases still serving traffic. Load balancers still balancing. The infrastructure didn't notice anything had happened, because nothing had happened to it. Only the bookkeeping was missing.
That reframe mattered more than I expected. Panic is a "the building is on fire" feeling. This was a "I lost the keys to a building that is, itself, completely fine" feeling. Annoying, embarrassing, and solvable, in that order.
Re-Cataloging Twenty Libraries by Hand
The fix for missing Terraform state has a name: terraform import. You tell Terraform "here is a resource address from your config, and here is the real-world ID of the thing it's supposed to represent," and it goes and reads that resource's actual attributes back into state. Do that for every resource in every project, and you've rebuilt the card catalog by walking every shelf yourself and writing each card by hand.
I worked through this with an AI agent, which honestly changed the whole shape of the day. Not because it did anything magic, but because the work was extremely repetitive and extremely unforgiving of small mistakes, which is exactly the combination where having a second set of eyes (or in this case, a very patient collaborator that doesn't get bored on resource four hundred) matters most.
We didn't start with the biggest project. We started with the smallest one, on purpose. Twelve resources, no secrets, nothing fragile. Get the process right at a scale where a mistake costs nothing, before doing it at a scale where a mistake costs a lot. That project became the template: read the actual code, don't guess, ask Terraform itself what it thinks is missing rather than trying to reconstruct the list by eye, and only ever generate the exact command needed, never something typed from memory.
Most resources turned out to be simple. If a Lambda function is named something-staging-orders, and the code builds that name directly from the environment, then the name is the ID. No detective work required. A handful of resource types needed one extra lookup: an assigned version number, an autogenerated ID, something Terraform couldn't have known just from reading the config. And a very small number turned out to be flatly ambiguous, where two live things matched one address, and the only honest way through was finding some other piece of live evidence to break the tie rather than guessing and hoping.
By the end we'd worked through several projects, one running past a thousand resources on its own, all imported back into state, verified clean against a fresh plan, logged in enough detail that anyone could see exactly which command fixed which resource and why.
The Part Where I Almost Did It Again
Here's the part I'd leave out if I were writing this to look competent instead of writing it honestly.
One of the larger projects had a Terraform quirk: importing resources one at a time, the normal way, kept failing, because Terraform insisted on evaluating the entire configuration graph after every single import, and one small piece of that graph couldn't resolve until other things existed first. The practical fix was to switch to Terraform's newer bulk-import mechanism, which resolves a whole batch of imports together in one pass instead of one at a time.
That part worked. What I didn't fully think through was that applying a bulk import batch doesn't limit itself to just the imports. It processes the entire plan, including a few resources that weren't part of my batch at all, because they hadn't been created yet. Three of those were secrets. Database credentials, an API key, the kind of thing you really do not want touched carelessly.
Since I'd deliberately left those three out of the import batch, Terraform did the only other thing it knows how to do with something not yet in state: it created them fresh, using the placeholder values sitting in a config file meant only for testing. For about a minute, three real, live secrets held the word "placeholder" instead of anything real.
I noticed almost immediately, because I checked. That's really the whole lesson, condensed: I checked. AWS Secrets Manager, mercifully, doesn't delete old secret values the moment a new one is written. It just relabels which version is "current" and keeps the previous one sitting right there, recoverable, until you tell it otherwise. So the fix was three commands, pointing "current" back at the version that actually held real data, followed by a longer moment of sitting very still and being grateful for boring, well-designed default behavior in a piece of infrastructure I hadn't thought about twice before that day.
What Actually Held Up
None of this is a story about being careful enough to never make mistakes. It's a story about a process that assumes you will, and catches most of them anyway.
A few things that genuinely mattered, in hindsight:
Read-only until proven necessary. Every lookup, every existence check, every plan preview happened before anything touched real state. The only steps allowed to actually change something were the ones we'd deliberately walked up to and confirmed.
Small before large. The twelve-resource project taught us the pattern cheaply. The thousand-resource project would have taught us the same lessons expensively, with real secrets in the blast radius from the very first mistake instead of the fifteenth.
Verify, don't assume. "It looked like it worked" and "I checked and it worked" turned out to be very different sentences, more than once that day.
Never apply broader than you meant to. The one moment that actually went sideways happened because a batch operation touched more than I'd scoped it to touch. Scope creep isn't just a project management problem. It's very much an infrastructure problem too, and it's the one I'd flag hardest for anyone reading this who's about to run something in bulk.
The bucket is still gone, permanently, no version history to restore. But everything it was supposed to be keeping track of is not gone, was never gone, and by the end of the day knew exactly where it stood again. I'd call that a good trade, given the alternative was finding out the hard way what "the library burned down" actually looks like.