Blog
Jacob Aronoff
Policies are Officially Part of OpenTelemetry
The Policy OTEP has been merged into OpenTelemetry!
Jacob AronoffThe Policy OTEP has been merged into OpenTelemetry!
Jacob AronoffI’m happy to announce that after a long review cycle in OpenTelemetry, the Policies OpenTelemetry Enhancement Proposal (OTEP) has officially been merged!
Policies are a new standard for teams to manage logs, traces and metrics which evolved out of community discussions at KubeCon NA ‘25 about common problems when using OpenTelemetry at scale. Everyone shared stories around how configuration often doesn’t work as expected, changing configuration is too slow and difficult, and there is zero portability between components like Vector and OpenTelemetry Collector. All of this is happening while overall telemetry volume is growing faster than ever. Josh Suereth and I discussed a few ways the ecosystem was working to solve these challenges, but we weren’t convinced that any of them address all of our concerns. The OTEP came from these issues and I began building prototypes at Tero.
Policies introduce a novel method of configuration for telemetry that enables high-throughput filtering and transformation. It’s small enough to be run in an SDK or be run in a central gateway like an OpenTelemetry Collector, which can run thousands of these policies concurrently with no loss in performance versus using existing solutions (read more about that here). Policies are able to do this because they’re designed to be independent (one policy does not depend on another), this allows us to match the policies concurrently through the use of hyperscan, a high-performance regular expression engine. You can read more about our design for Tero’s policy engine here.
One giant config, one fragile pipeline
processors:
filter/logs:
logs:
log_record:
- 'IsMatch(body, ".*/health.*")'
- 'severity_number < 9'
transform/pii:
log_statements:
- context: log
statements:
- replace_pattern(body, ...)
filter/metrics: { ... }
service:
pipelines:
logs: [filter/logs, transform/pii, ...]break one step, break everything downstream
Small, independent rules - matched all at once
id: drop-gateway-route-hops
log:
match:
- log_field: event
equals: middleware_request
keep: noneindependent Hyperscan-matched no perf loss
The next steps for Policies will be integrating them into the OpenTelemetry ecosystem. We’re beginning by forming a project group around the effort to coordinate the workstreams. This group will work on formalizing the specification proposed in the OTEP, integrating policies into the OpenTelemetry collector, and specing what integration in an SDK through declarative config would look like. Before they are upstreamed into OpenTelemetry, you can begin using Policies through our open source tools like the edge, policy-go, or our policy processor.
Policies are an exciting new step for OpenTelemetry and the observability ecosystem. Configuration for telemetry can feel like whack-a-mole at times: you find a pesky log, you add a new rule in a pipeline, and another pops up a day later. Managing these systems can be even more frustrating; needing to handle huge volume at scale gets expensive as you need more compute in your environment and your vendor charges you even more. Policies help you solve this, Tero puts this on autopilot. We intelligently discover problems in your telemetry and proactively generate policies to solve them.
A MASSIVE thank you to Josh Suereth, Raphael Menderico and Jack Shirazi for coauthoring. Thank you to Jack Berg, Reiley Yang, Tigran Najaryan, David Ashpole, Josh MacDonald, Liudmila Molkova, Braydon Kains, Andrzej Stencel, James Thompson, Christoph Heger, Florian Lehner, and Antoine Toulme for their comments, discussions and reviews.