Blog
Ben Johnson
We Can Understand It All Now
The waste question forced a harder realization: production telemetry can be continuously understood.
Ben JohnsonThe waste question forced a harder realization: production telemetry can be continuously understood.
Ben JohnsonIn January, I wrote a post called The Question Your Observability Vendor Won't Answer.
The question was simple:
how much of your observability data is waste?
That question came from a decade in observability and a few years very close to the metal with Vector. I watched teams build absurdly complex pipelines, configs, filters, sampling rules, storage tiers, archives, regex lists, and budget rituals around one basic fact: they were sending too much low-value data into expensive systems, and nobody could tell them what was safe to change.
I thought waste was the biggest problem in observability.
I was wrong.
Waste is real. We answered the question. Most environments carry 40-60% telemetry waste. If you are spending seven figures on observability, that is not a cute optimization. That is a serious operating problem.
But I care less about the number now than I care about what it took to get it.
To answer the waste question, we had to solve a much harder problem: we had to build a system that could continuously understand production telemetry at scale.
By understand, I do not mean sampling a stream, summarizing a few rows, watching a handful of alerts, or letting an agent poke around after something already hurts. I mean building and maintaining a finite map of the telemetry estate: what events exist, what they mean, how they change, what value they carry, and where the raw evidence lives.
That is the part that changed everything for me. It broke an assumption I did not realize I had stopped questioning.
The observability industry was built around an assumption that used to be true:
you cannot continuously understand all of the data.
There is too much of it. It is too messy. It is too repetitive. It is too expensive. It lacks context. It changes constantly. A single log line is rarely enough to know whether something matters, and the surrounding context is spread across services, deploys, teams, business paths, time ranges, and raw records.
So the industry did the reasonable thing and built around the assumption. We built faster storage because the pile kept growing. We built metrics because raw events were too much to reason over directly. We built traces because requests needed shape across services. We built dashboards for known questions. We built alerts for known failure modes. We built search for after-the-fact investigation. We built pipelines, filters, sampling rules, storage tiers, archives, and data lakes because the evidence was too valuable to throw away and too expensive to keep in one place.
That model was not stupid. It was the right model for a world where you had more data than you could continuously understand, but could not afford to throw away.
But it was still built around a limitation: no one could understand it all.
The waste question sounds like a cost question, but it forced us into something more basic.
You cannot decide whether telemetry is waste by counting bytes. You cannot decide it by looking at volume alone. You cannot hand one log line to a model and ask if it is valuable. That fails for the same reason a new engineer cannot join your company, read one line from one service, and reliably tell you whether deleting it will hurt you later.
Value is contextual. You need to know what operation the event belongs to, what state it marks, what service emitted it, what that service does, how the event behaves over time, what kind of evidence it carries, whether it helps explain failure, performance, audit, or compliance, whether nearby events already tell the same story better, and whether its cost and storage path match its actual use. A log can be valuable and still be wrong: too noisy, too expensive, redundant in context, or carrying far more payload than the evidence requires.
Telemetry has to be understood before you can decide what should happen to it.
We had to compress raw telemetry into durable context: event types, representative examples, volume, change over time, semantic meaning, service context, ground-truth facts, and links back to raw evidence. We had to use AI where it actually mattered, not as a mascot bolted onto a UI, but as part of a system that could maintain understanding across data that used to be too large to reason about.
We built it to find waste, and then other things started falling out.
The first thing we found was waste. A lot of it.
That was satisfying, because it proved the original thesis. Companies really are paying for enormous amounts of telemetry that does not help humans or agents understand production. Sometimes they ingest it only to drop it later. Sometimes it is debug noise left in a hot path. Sometimes it is repeated low-value events or oversized payloads or the same story told ten different ways.
The surprise was that the system did not stop at cost. Once we understood the data well enough to say whether an event was earning its place, we had already built much of what was needed to find other problems teams usually discover too late.
We found sensitive data: payment fields, tokens, secrets, request payloads. The kind of thing that sneaks in through an innocent debug line or payload inspection, then sits there until someone finally sees it with examples, owners, and destinations attached.
We found reliability risk: retry storms that never quite page anyone, database connection errors hidden by retries until traffic grows, dependency timeouts that look harmless one at a time and obvious across weeks, error paths affecting a small percentage of users every day.
We found security issues for the same reason. Not because we set out to build five separate scanners, but because once the data is understood, these things are already there.
This is where the old model started to feel wrong to me. It is one thing to talk about better telemetry in the abstract. It is another thing to connect to real production data and watch issues appear with examples, owners, impact, and raw evidence attached.
The logs had been saying it the whole time. We just were not continuously reading them.
Telemetry is not passive because it lacks answers. It is passive because the industry never had a good way to continuously understand it.
Once you can continuously understand the data, it starts handing you work: this event is wasteful, this field is sensitive, this dependency is degrading, this service owns the issue, this evidence supports the claim, this action path is safe enough to review.
That feels very different from waiting for a dashboard to turn red.
When that assumption breaks, observability does not look like the same stack with a chatbot attached. The center of the system moves.
Telemetry stops being passive evidence and starts becoming work: issues with evidence, owners, raw records, and safe paths forward. Some work still goes to humans. Some goes to agents. Some becomes a policy, a pull request, a ticket, a runbook step, or a review. The point is not reckless automation. The point is that production work starts upstream, while the evidence is still cheap to act on.
That changes what the stack is for.
Storage still matters, but storage is no longer the center. Companies should be able to put telemetry wherever it makes sense: ClickHouse, S3, CloudWatch, a lakehouse, or whatever internal store they want to use. The value moves to the layer above storage: the layer that understands the data, finds the work, and gives agents and teams enough context to act.
As engineering work moves through agents, that layer becomes the operating surface. Agents can query raw stores, but raw access keeps them in the old model. They need context, evidence, ownership, and safe paths to act.
Dashboards, alerts, and search do not disappear overnight. But they stop being the main event. Dashboards become views the system can generate when needed. Alerts become safety rails. Search becomes a way to inspect evidence, not the place where understanding begins.
The loop is different: understand the telemetry, surface the issue, attach the evidence, route the work, give agents a safe path to act, and measure what changed.
That is the world Tero is building.
