1
0
mirror of https://github.com/open-telemetry/opentelemetry-go.git synced 2026-06-03 18:35:08 +02:00
Files
opentelemetry-go/sdk/metric/exemplar.go
T
David Ashpole 6f81801cd8 Use a DropReservoir when an exemplar.AlwaysOffFilter is provided (#8211)
Fixes https://github.com/open-telemetry/opentelemetry-go/issues/6260,
https://github.com/open-telemetry/opentelemetry-go/issues/6333

The [spec for always
off](https://github.com/open-telemetry/opentelemetry-specification/blob/d500678e4612b56ff2cd5f03e67cd845977d1746/specification/metrics/sdk.md#alwaysoff)
suggests this should "disable" the Exemplar feature:

> An ExemplarFilter which makes no measurements eligible for being an
Exemplar.
Using this ExemplarFilter is as good as disabling the Exemplar feature.

There were a few reports of much higher memory usage when exemplars were
added. I looked into the reports, and it looks like exemplar reservoirs
are expensive in terms of memory compared to the aggregation itself.
Ideally, we shouldn't construct the reservoir when an AlwaysOff exemplar
filter is used.

I couldn't find a super clean way to do this...
`exemplar.AlwaysOffFilter` is a function, so I couldn't easily compare
it without comparing the pointers. I'm open to other suggestions. The
internet says that compiler optimizations could cause this logic to
fail, but worst-case you get the existing behavior.

### Benchmarks

I needed a benchmark that creates a new attribute set for each call to
be able to see how much memory each attribute set uses, so I added a new
one, and ran it against main.

Overall, it reduces the amount of memory used per-attribute-set by
70-88%, which is pretty substantial.

Overall, 

```
                                                  │   old.txt   │               new.txt               │
                                                  │   sec/op    │    sec/op     vs base               │
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24            3.523µ ± 2%   3.745µ ± 29%   +6.32% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24          3.714µ ± 6%   3.879µ ±  5%        ~ (p=0.132 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24      3.811µ ± 8%   3.941µ ±  5%        ~ (p=0.065 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24    3.714µ ± 5%   3.822µ ±  1%        ~ (p=0.132 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24          3.126µ ± 4%   3.333µ ±  4%   +6.61% (p=0.004 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24        3.099µ ± 4%   3.204µ ±  8%        ~ (p=0.093 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24              3.745µ ± 7%   3.902µ ±  4%        ~ (p=0.240 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24            3.746µ ± 7%   3.862µ ±  2%        ~ (p=0.102 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24           3.621µ ± 3%   1.665µ ±  1%  -54.00% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24         3.639µ ± 2%   1.686µ ±  4%  -53.68% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24     3.563µ ± 4%   1.700µ ±  4%  -52.29% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24   3.634µ ± 1%   1.690µ ±  4%  -53.49% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24         2.892µ ± 2%   2.005µ ±  2%  -30.66% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24       2.962µ ± 5%   2.057µ ±  6%  -30.55% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24             3.692µ ± 5%   1.599µ ±  2%  -56.68% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24           3.651µ ± 3%   1.612µ ±  2%  -55.86% (p=0.002 n=6)
geomean                                             3.495µ        2.541µ        -27.30%

                                                  │   old.txt    │               new.txt               │
                                                  │     B/op     │     B/op      vs base               │
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24            3.107Ki ± 0%   3.107Ki ± 0%        ~ (p=0.121 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24          3.108Ki ± 0%   3.107Ki ± 0%        ~ (p=0.177 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24      3.108Ki ± 0%   3.108Ki ± 0%        ~ (p=0.675 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24    3.107Ki ± 0%   3.107Ki ± 0%        ~ (p=1.000 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24          2.622Ki ± 0%   2.622Ki ± 0%        ~ (p=0.394 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24        2.622Ki ± 0%   2.622Ki ± 0%        ~ (p=0.636 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24              3.108Ki ± 0%   3.107Ki ± 0%        ~ (p=0.167 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24            3.107Ki ± 0%   3.108Ki ± 0%        ~ (p=0.182 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24            3182.0 ± 0%     361.0 ± 0%  -88.65% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24          3182.5 ± 0%     360.5 ± 0%  -88.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24      3182.0 ± 0%     359.5 ± 1%  -88.70% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24    3182.0 ± 0%     361.0 ± 0%  -88.65% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24          2684.0 ± 0%     773.0 ± 0%  -71.20% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24        2684.0 ± 0%     773.0 ± 0%  -71.20% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24              3182.0 ± 0%     361.0 ± 1%  -88.65% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24            3182.0 ± 0%     361.0 ± 0%  -88.65% (p=0.002 n=6)
geomean                                             2.978Ki        1.127Ki       -62.17%

                                                  │   old.txt   │               new.txt               │
                                                  │  allocs/op  │ allocs/op   vs base                 │
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24             12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24           12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24       12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24     12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24           13.00 ± 0%   13.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24         13.00 ± 0%   13.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24               12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24             12.00 ± 0%   12.00 ± 0%        ~ (p=1.000 n=6) ¹
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24           12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24         12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24     12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24   12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24         13.000 ± 0%   9.000 ± 0%  -30.77% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24       13.000 ± 0%   9.000 ± 0%  -30.77% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24             12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24           12.000 ± 0%   7.000 ± 0%  -41.67% (p=0.002 n=6)
geomean                                              12.24        9.553       -21.97%
¹ all samples are equal
```

Gemini helped me write this.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-21 16:08:33 -04:00

82 lines
3.4 KiB
Go

// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0
package metric // import "go.opentelemetry.io/otel/sdk/metric"
import (
"reflect"
"runtime"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/sdk/metric/exemplar"
"go.opentelemetry.io/otel/sdk/metric/internal/aggregate"
)
// ExemplarReservoirProviderSelector selects the
// [exemplar.ReservoirProvider] to use
// based on the [Aggregation] of the metric.
type ExemplarReservoirProviderSelector func(Aggregation) exemplar.ReservoirProvider
// reservoirFunc returns the appropriately configured exemplar reservoir
// creation func based on the passed InstrumentKind and filter configuration.
func reservoirFunc[N int64 | float64](
provider exemplar.ReservoirProvider,
filter exemplar.Filter,
) func(attribute.Set) aggregate.FilteredExemplarReservoir[N] {
if reflect.ValueOf(filter).Pointer() == reflect.ValueOf(exemplar.AlwaysOffFilter).Pointer() {
return aggregate.DropReservoir[N]
}
return func(attrs attribute.Set) aggregate.FilteredExemplarReservoir[N] {
return aggregate.NewFilteredExemplarReservoir[N](filter, provider(attrs))
}
}
// DefaultExemplarReservoirProviderSelector returns the default
// [exemplar.ReservoirProvider] for the
// provided [Aggregation].
//
// For explicit bucket histograms with more than 1 bucket, it uses the
// [exemplar.HistogramReservoirProvider].
// For exponential histograms, it uses the
// [exemplar.FixedSizeReservoirProvider]
// with a size of min(20, max_buckets).
// For all other aggregations, it uses the
// [exemplar.FixedSizeReservoirProvider]
// with a size equal to the number of CPUs.
//
// Exemplar default reservoirs MAY change in a minor version bump. No
// guarantees are made on the shape or statistical properties of returned
// exemplars.
func DefaultExemplarReservoirProviderSelector(agg Aggregation) exemplar.ReservoirProvider {
// https://github.com/open-telemetry/opentelemetry-specification/blob/d4b241f451674e8f611bb589477680341006ad2b/specification/metrics/sdk.md#exemplar-defaults
// Explicit bucket histogram aggregation with more than 1 bucket will
// use AlignedHistogramBucketExemplarReservoir.
a, ok := agg.(AggregationExplicitBucketHistogram)
if ok && len(a.Boundaries) > 0 {
return exemplar.HistogramReservoirProvider(a.Boundaries)
}
var n int
if a, ok := agg.(AggregationBase2ExponentialHistogram); ok {
// Base2 Exponential Histogram Aggregation SHOULD use a
// SimpleFixedSizeExemplarReservoir with a reservoir equal to the
// smaller of the maximum number of buckets configured on the
// aggregation or twenty (e.g. min(20, max_buckets)).
n = min(int(a.MaxSize), 20)
} else {
// https://github.com/open-telemetry/opentelemetry-specification/blob/e94af89e3d0c01de30127a0f423e912f6cda7bed/specification/metrics/sdk.md#simplefixedsizeexemplarreservoir
// This Exemplar reservoir MAY take a configuration parameter for
// the size of the reservoir. If no size configuration is
// provided, the default size MAY be the number of possible
// concurrent threads (e.g. number of CPUs) to help reduce
// contention. Otherwise, a default size of 1 SHOULD be used.
//
// Use runtime.GOMAXPROCS instead of runtime.NumCPU to support
// containerized environments that may have less than the total number
// of logical CPUs available on the local machine allocated to it.
n = max(runtime.GOMAXPROCS(0), 1)
}
return exemplar.FixedSizeReservoirProvider(n)
}