Fixes#6979
Towards #6887
## What
- Documentation includes a section on cardinality limits.
- Adding guidance around what cardinality is, why it is important to
consider, how to control it, and what defaults the SDK provides
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
* Check that IDs round-trip between binary and hex string formats.
`TestIDsRoundTrip` is patterned after `TestNewIDs`.
* Check that bad values give the expected errors.
* Use larger values in `TestWithIDGenerator`. Previously, nearly all the
bits were zero so a mistake in encoding/decoding higher bits could be
missed.
Start with arbitrary values with more bits set.
(Span ID still has top half as zero due to taking an uint64)
* Modify `testIDGenerator` so this ^^ change runs on 32-bit platforms.
The idea for more tests arose while considering #6791.
Does not need a CHANGELOG entry - test only.
---------
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
Fixes#6980
Towards #6887
## What
- Any references to sdk/metric/x in documentation, tests, or examples
are updated to redirect users to use the cardinality limit feature in
the SDK.
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Fixes https://github.com/open-telemetry/opentelemetry-go/issues/7005
Adds `otel.sdk.processor.span.queue.size`,
`otel.sdk.processor.span.queue.capacity`, and
`otel.sdk.processor.span.processed.count` metrics to the trace batch
span processor.
These are defined in
cb11bb9bac/docs/otel/sdk-metrics.md,
and are experimental. Because of this, metrics are behind the
OTEL_GO_X_SELF_OBSERVABILITY feature gate.
Given the feature is experimental, it always uses the global
meterprovider when enabled.
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
I am looking into
https://promlabs.com/blog/2025/07/17/why-i-recommend-native-prometheus-instrumentation-over-opentelemetry/#comparing-counter-increment-performance,
which seems to suggest the OTel metrics SDK performs poorly when a
counter is incremented concurrently. It is potentially a bit of an
artificial benchmark, but does suggest there is some contention beyond
just the fact that they are incrementing an atomic integer...
Original benchmarks from the blog post:
https://github.com/promlabs/prometheus-otel-benchmarks/blob/main/otel_test.go
```
$ go test -run=xxxxxMatchNothingxxxxx -cpu=24 -test.benchtime=1s -bench=BenchmarkSyncMeasure/NoView/
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/metric
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/0-24 3946789 313.2 ns/op
BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/1-24 3420992 374.4 ns/op
BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/10-24 574608 1745 ns/op
BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/0-24 3996166 281.1 ns/op
BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/1-24 3091573 367.1 ns/op
BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/10-24 705693 1660 ns/op
BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/0-24 4098727 296.4 ns/op
BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/1-24 3029276 355.4 ns/op
BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/10-24 605174 1803 ns/op
BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/0-24 4057765 298.6 ns/op
BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/1-24 3384812 366.9 ns/op
BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/10-24 714900 1742 ns/op
BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/0-24 3274644 364.3 ns/op
BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/1-24 3780115 316.1 ns/op
BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/10-24 1294364 993.5 ns/op
BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/0-24 3543817 343.2 ns/op
BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/1-24 3523102 335.8 ns/op
BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/10-24 1329352 956.3 ns/op
PASS
ok go.opentelemetry.io/otel/sdk/metric 27.504s
```
```
$ go test -run=xxxxxMatchNothingxxxxx -cpu=1 -test.benchtime=1s -bench=BenchmarkSyncMeasure/NoView/
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/metric
cpu: Intel(R) Xeon(R) CPU @ 2.20GHz
BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/0 9905773 121.3 ns/op
BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/1 4079145 296.5 ns/op
BenchmarkSyncMeasure/NoView/Int64Counter/Attributes/10 781627 1531 ns/op
BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/0 10017988 120.2 ns/op
BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/1 4055418 296.4 ns/op
BenchmarkSyncMeasure/NoView/Float64Counter/Attributes/10 761139 1540 ns/op
BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/0 10017126 121.1 ns/op
BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/1 4037232 295.3 ns/op
BenchmarkSyncMeasure/NoView/Int64UpDownCounter/Attributes/10 757010 1539 ns/op
BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/0 10122925 119.0 ns/op
BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/1 4070942 293.8 ns/op
BenchmarkSyncMeasure/NoView/Float64UpDownCounter/Attributes/10 788176 1542 ns/op
BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/0 10794142 110.8 ns/op
BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/1 5929494 201.0 ns/op
BenchmarkSyncMeasure/NoView/Int64Histogram/Attributes/10 1449292 825.4 ns/op
BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/0 10875385 110.1 ns/op
BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/1 5903116 202.4 ns/op
BenchmarkSyncMeasure/NoView/Float64Histogram/Attributes/10 1459578 827.4 ns/op
PASS
ok go.opentelemetry.io/otel/sdk/metric 25.688s
```
Results are significantly worse (almost > 2x in some cases) with
parallelism, but don't initially seem as bad as the blog post suggests.
I only have 24 cores, so I can't test higher numbers.
Do we want to have parallel benchmarks in addition to our current
non-parallel ones?
This PR changes `DefaultExemplarReservoirProviderSelector` in
`go.opentelemetry.io/otel/sdk/metric` to use `runtime.GOMAXPROCS(0)`
instead of `runtime.NumCPU()` for the `FixedSizeReservoirProvider`
default size.
**Motivation**
The [runtime.NumCPU](https://pkg.go.dev/runtime#NumCPU) returns the
number of logical CPUs usable by the current process. In Kubernetes, the
method will return the total number of cores on the node, not the actual
number of cores available to the container. Since it is common practice
to allocate only a small fraction of the total CPU on a node to a
container, the current implementation may result in an excessive
`FixedSizeReservoir` size, which in turn negatively impacts the
resources consumed.
---------
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
Co-authored-by: Tyler Yahn <codingalias@gmail.com>
Fixes#6976
Towards #6887
## What
- A new configuration field for cardinality limits is added to the SDK.
- Users can set cardinality limits via code
- Default value of 0(no limit) is applied if no value is provided.
- Unit tests validate the configuration behavior.
## Additional
- Issue created regarding the missing environment variable for the
cardinality limit:
[opentelemetry-specification#4586](https://github.com/open-telemetry/opentelemetry-specification/issues/4586)
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: Damien Mathieu <42@dmathieu.com>
Co-authored-by: Cijo Thomas <cithomas@microsoft.com>
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
The unexported `version()` function in `sdk/trace` is not used anywhere.
Not mentioning that the value it is returning is wrong.
There is already a `Version` function in `sdk` (which is in the same Go
module).
Closes#5133
This couldn't be added as an option on a processor, as that would
involve moving all the attribute deduplication. logic outside of the
record type. Instead this PR provides the same functionality but it is
set when creating the log provider
The below benchstat report shows the performance improvement when
`allowDupKeys` is set
```
goos: darwin
goarch: arm64
pkg: go.opentelemetry.io/otel/sdk/log
cpu: Apple M2 Pro
│ withoutDedup.txt │ withDedup.txt │
│ sec/op │ sec/op vs base │
SetAddAttributes/SetAttributes-12 141.3n ± 2% 167.4n ± 5% +18.51% (p=0.000 n=10)
SetAddAttributes/AddAttributes-12 117.5n ± 2% 124.8n ± 5% +6.17% (p=0.000 n=10)
geomean 128.9n 144.5n +12.17%
│ withoutDedup.txt │ withDedup.txt │
│ B/op │ B/op vs base │
SetAddAttributes/SetAttributes-12 48.00 ± 0% 48.00 ± 0% ~ (p=1.000 n=10) ¹
SetAddAttributes/AddAttributes-12 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
│ withoutDedup.txt │ withDedup.txt │
│ allocs/op │ allocs/op vs base │
SetAddAttributes/SetAttributes-12 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) ¹
SetAddAttributes/AddAttributes-12 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
```
### Description
OpenTelemetry uses `DefaultScheduleDelay` and `DefaultExportTimeout`
values as milliseconds but Go time package will understand them as
nanoseconds.
I understand that this is a stable library and that those value will
probably never change, so can we at least clarify their usage?
Right above the defaults declaration it says `// Defaults for
BatchSpanProcessorOptions.` which is confusing.
We used `trace.DefaultScheduleDelay` as a fallback value for our tracing
setup.
This confusion led to high CPU usage due to the frequent batch exports.
### Confusing behavior
```go
processor := trace.NewBatchSpanProcessor(exporter,
// set timeout to 5000 ns instead of the expected 5000 ms
trace.WithBatchTimeout(trace.DefaultScheduleDelay),
// set timeout to 30000 ns instead of the expected 30000 ms
trace.WithExportTimeout(trace.DefaultExportTimeout),
)
```
### Correct way to use those values
```go
processor := trace.NewBatchSpanProcessor(exporter,
trace.WithBatchTimeout(trace.DefaultScheduleDelay * time.Millisecond),
trace.WithExportTimeout(trace.DefaultExportTimeout * time.Millisecond),
)
```
---------
Co-authored-by: Damien Mathieu <42@dmathieu.com>
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Commit 575e1bb270 deprecated the Library
type in favor of Scope, but did not add an empty line before the
deprecation comment. Go's formatting rules require an empty line;
omitting the empty line can cause some tools to not detect the
deprecation.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Co-authored-by: Robert Pająk <pellared@hotmail.com>
### Added
- The `go.opentelemetry.io/otel/semconv/v1.33.0` package.
The package contains semantic conventions from the `v1.33.0` version of
the OpenTelemetry Semantic Conventions.
See the [migration documentation](./semconv/v1.33.0/MIGRATION.md) for
information on how to upgrade from
`go.opentelemetry.io/otel/semconv/v1.32.0.`(#6799)
- The `go.opentelemetry.io/otel/semconv/v1.34.0` package.
The package contains semantic conventions from the `v1.34.0` version of
the OpenTelemetry Semantic Conventions. (#6812)
- Add metric's schema URL as `otel_scope_schema_url` label in
`go.opentelemetry.io/otel/exporters/prometheus`. (#5947)
- Add metric's scope attributes as `otel_scope_[attribute]` labels in
`go.opentelemetry.io/otel/exporters/prometheus`. (#5947)
- Add `EventName` to `EnabledParameters` in
`go.opentelemetry.io/otel/log`. (#6825)
- Add `EventName` to `EnabledParameters` in
`go.opentelemetry.io/otel/sdk/log`. (#6825)
- Changed handling of `go.opentelemetry.io/otel/exporters/prometheus`
metric renaming to add unit suffixes when it doesn't match one of the
pre-defined values in the unit suffix map. (#6839)
### Changed
- The semantic conventions have been upgraded from `v1.26.0` to
`v1.34.0` in `go.opentelemetry.io/otel/bridge/opentracing`. (#6827)
- The semantic conventions have been upgraded from `v1.26.0` to
`v1.34.0` in `go.opentelemetry.io/otel/exporters/zipkin`. (#6829)
- The semantic conventions have been upgraded from `v1.26.0` to
`v1.34.0` in `go.opentelemetry.io/otel/metric`. (#6832)
- The semantic conventions have been upgraded from `v1.26.0` to
`v1.34.0` in `go.opentelemetry.io/otel/sdk/resource`. (#6834)
- The semantic conventions have been upgraded from `v1.26.0` to
`v1.34.0` in `go.opentelemetry.io/otel/sdk/trace`. (#6835)
- The semantic conventions have been upgraded from `v1.26.0` to
`v1.34.0` in `go.opentelemetry.io/otel/trace`. (#6836)
- `Record.Resource` now returns `*resource.Resource` instead of
`resource.Resource` in `go.opentelemetry.io/otel/sdk/log`. (#6864)
- Retry now shows error cause for context timeout in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`,
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`,
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`,
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`,
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`,
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#6898)
### Fixed
- Stop stripping trailing slashes from configured endpoint URL in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`.
(#6710)
- Stop stripping trailing slashes from configured endpoint URL in
`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`.
(#6710)
- Stop stripping trailing slashes from configured endpoint URL in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`.
(#6710)
- Stop stripping trailing slashes from configured endpoint URL in
`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`.
(#6710)
- Validate exponential histogram scale range for Prometheus
compatibility in `go.opentelemetry.io/otel/exporters/prometheus`.
(#6822)
- Context cancellation during metric pipeline produce does not corrupt
data in `go.opentelemetry.io/otel/sdk/metric`. (#6914)
### Removed
- `go.opentelemetry.io/otel/exporters/prometheus` no longer exports
`otel_scope_info` metric. (#6770)
This fixes issue #6344, in which an attempted clean shutdown of the
metrics SDK using PeriodicReader's Shutdown method can produce and emit
wildly incorrect data point values that are orders of magnitude too
large.
The root of the issue is that the pipeline produce method changed in
this PR cannot safely return early, which was happening in the callback
loops that were checking for context cancellation. Early return is not
safe since callbacks and aggregations are tightly coupled in practice:
invoking callbacks without also invoking aggregations corrupts internal
data point value accounting.
The linked issue more concretely walks through the sequence of steps
that were causing this issue.
---------
Co-authored-by: Damien Mathieu <42@dmathieu.com>
# Overview
### Fixes
- Use the proper dependency version of
`go.opentelemetry.io/otel/sdk/log/logtest` in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`. (#6800)
- Use the proper dependency version of
`go.opentelemetry.io/otel/sdk/log/logtest` in
`go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#6800)
- Use the proper dependency version of
`go.opentelemetry.io/otel/sdk/log/logtest` in
`go.opentelemetry.io/otel/exporters/stdout/stdoutlog`. (#6800)
