Blog
Ben Johnson
Stop Replaying Your Logs
Data replay made sense when archives were dead. They aren't anymore.
Ben JohnsonData replay made sense when archives were dead. They aren't anymore.
Ben JohnsonI saw another log replay setup today. I am so tired of seeing this.
Here was a system that went through all the trouble of collecting raw logs, summarized them until they could no longer answer real questions, dumped the originals into an archive, and then built machinery to replay them when the summary inevitably failed.
A Rube Goldberg machine for querying your own data.
I understand how we got here. A decade ago, searchable storage was expensive and archives were genuinely cold. The archive was where data went to stop costing money and, for all practical purposes, stop existing. If you wanted to use it again, you had to drag it back into a real database.
Fine. Data replay made sense then.
But it's 2026. Modern columnar engines can query enormous amounts of cheap object storage directly. Pick whichever one you want. The storage we used to call an archive is now fast enough to query. It's really good now.
The cold storage is hot now. Or at least hot enough that replay no longer makes sense.
The whole strategy is backwards. You keep the raw logs because the summaries aren't always enough, then make engineers restore them before they can search them.
Your engineers hate this. Of course they do. They're halfway through an investigation, trying to understand what production just did, and now they have to pick a time range, start a replay job, wait, discover they asked for the wrong slice, and try again.
What a miserable way to use a logging platform.
Vendors still love the arrangement. They get to sell you the hot tier, the archive, and the toll booth between them. You get an incomplete copy you can search and a complete copy you can't.
None of this means you should keep every garbage log forever. Garbage should be dropped. Sampling can be sensible. Bad instrumentation should be fixed.
But if data is worth keeping, it should be queryable. Storing the truth somewhere your engineers can't reach without a restoration job is not a logging strategy. It's an old constraint that no longer exists.
Data replay made sense when archives were dead.
They aren't anymore. Search the logs where they live.
