b7610a72a7
I am looking into https://promlabs.com/blog/2025/07/17/why-i-recommend-native-prometheus-instrumentation-over-opentelemetry/#comparing-counter-increment-performance, which seems to suggest the OTel metrics SDK performs poorly when a counter is incremented concurrently. It is potentially a bit of an artificial benchmark, but does suggest there is some contention beyond just the fact that they are incrementing an atomic integer... Original benchmarks from the blog post: https://github.com/promlabs/prometheus-otel-benchmarks/blob/main/otel_test.go ``` $ go test -run=xxxxxMatchNothingxxxxx -cpu=24 -test.benchtime=1s -bench=BenchmarkSyncMeasure/NoView/ goos: linux goarch: amd64 pkg: go.opentelemetry.io/otel/sdk/metric cpu: Intel(R) Xeon(R) CPU @ 2.20GHz BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/0-24 3946789 313.2 ns/op BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/1-24 3420992 374.4 ns/op BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/10-24 574608 1745 ns/op BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/0-24 3996166 281.1 ns/op BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/1-24 3091573 367.1 ns/op BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/10-24 705693 1660 ns/op BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/0-24 4098727 296.4 ns/op BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/1-24 3029276 355.4 ns/op BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/10-24 605174 1803 ns/op BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/0-24 4057765 298.6 ns/op BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/1-24 3384812 366.9 ns/op BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/10-24 714900 1742 ns/op BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/0-24 3274644 364.3 ns/op BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/1-24 3780115 316.1 ns/op BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/10-24 1294364 993.5 ns/op BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/0-24 3543817 343.2 ns/op BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/1-24 3523102 335.8 ns/op BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/10-24 1329352 956.3 ns/op PASS ok go.opentelemetry.io/otel/sdk/metric 27.504s ``` ``` $ go test -run=xxxxxMatchNothingxxxxx -cpu=1 -test.benchtime=1s -bench=BenchmarkSyncMeasure/NoView/ goos: linux goarch: amd64 pkg: go.opentelemetry.io/otel/sdk/metric cpu: Intel(R) Xeon(R) CPU @ 2.20GHz BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/0 9905773 121.3 ns/op BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/1 4079145 296.5 ns/op BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/10 781627 1531 ns/op BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/0 10017988 120.2 ns/op BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/1 4055418 296.4 ns/op BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/10 761139 1540 ns/op BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/0 10017126 121.1 ns/op BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/1 4037232 295.3 ns/op BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/10 757010 1539 ns/op BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/0 10122925 119.0 ns/op BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/1 4070942 293.8 ns/op BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/10 788176 1542 ns/op BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/0 10794142 110.8 ns/op BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/1 5929494 201.0 ns/op BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/10 1449292 825.4 ns/op BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/0 10875385 110.1 ns/op BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/1 5903116 202.4 ns/op BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/10 1459578 827.4 ns/op PASS ok go.opentelemetry.io/otel/sdk/metric 25.688s ``` Results are significantly worse (almost > 2x in some cases) with parallelism, but don't initially seem as bad as the blog post suggests. I only have 24 cores, so I can't test higher numbers. Do we want to have parallel benchmarks in addition to our current non-parallel ones?
OpenTelemetry-Go
OpenTelemetry-Go is the Go implementation of OpenTelemetry. It provides a set of APIs to directly measure performance and behavior of your software and send this data to observability platforms.
Project Status
| Signal | Status |
|---|---|
| Traces | Stable |
| Metrics | Stable |
| Logs | Beta1 |
Progress and status specific to this repository is tracked in our project boards and milestones.
Project versioning information and stability guarantees can be found in the versioning documentation.
Compatibility
OpenTelemetry-Go ensures compatibility with the current supported versions of the Go language:
Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release.
For versions of Go that are no longer supported upstream, opentelemetry-go will stop ensuring compatibility with these versions in the following manner:
- A minor release of opentelemetry-go will be made to add support for the new supported release of Go.
- The following minor release of opentelemetry-go will remove compatibility testing for the oldest (now archived upstream) version of Go. This, and future, releases of opentelemetry-go may include features only supported by the currently supported versions of Go.
Currently, this project supports the following environments.
| OS | Go Version | Architecture |
|---|---|---|
| Ubuntu | 1.24 | amd64 |
| Ubuntu | 1.23 | amd64 |
| Ubuntu | 1.24 | 386 |
| Ubuntu | 1.23 | 386 |
| Ubuntu | 1.24 | arm64 |
| Ubuntu | 1.23 | arm64 |
| macOS 13 | 1.24 | amd64 |
| macOS 13 | 1.23 | amd64 |
| macOS | 1.24 | arm64 |
| macOS | 1.23 | arm64 |
| Windows | 1.24 | amd64 |
| Windows | 1.23 | amd64 |
| Windows | 1.24 | 386 |
| Windows | 1.23 | 386 |
While this project should work for other systems, no compatibility guarantees are made for those systems currently.
Getting Started
You can find a getting started guide on opentelemetry.io.
OpenTelemetry's goal is to provide a single set of APIs to capture distributed traces and metrics from your application and send them to an observability platform. This project allows you to do just that for applications written in Go. There are two steps to this process: instrument your application, and configure an exporter.
Instrumentation
To start capturing distributed traces and metric events from your application it first needs to be instrumented. The easiest way to do this is by using an instrumentation library for your code. Be sure to check out the officially supported instrumentation libraries.
If you need to extend the telemetry an instrumentation library provides or want to build your own instrumentation for your application directly you will need to use the Go otel package. The examples are a good way to see some practical uses of this process.
Export
Now that your application is instrumented to collect telemetry, it needs an export pipeline to send that telemetry to an observability platform.
All officially supported exporters for the OpenTelemetry project are contained in the exporters directory.
| Exporter | Logs | Metrics | Traces |
|---|---|---|---|
| OTLP | ✓ | ✓ | ✓ |
| Prometheus | ✓ | ||
| stdout | ✓ | ✓ | ✓ |
| Zipkin | ✓ |
Contributing
See the contributing documentation.