Use a DropReservoir when an exemplar.AlwaysOffFilter is provided (#8211)

Fixes https://github.com/open-telemetry/opentelemetry-go/issues/6260, https://github.com/open-telemetry/opentelemetry-go/issues/6333 The [spec for always off](https://github.com/open-telemetry/opentelemetry-specification/blob/d500678e4612b56ff2cd5f03e67cd845977d1746/specification/metrics/sdk.md#alwaysoff) suggests this should "disable" the Exemplar feature: > An ExemplarFilter which makes no measurements eligible for being an Exemplar. Using this ExemplarFilter is as good as disabling the Exemplar feature. There were a few reports of much higher memory usage when exemplars were added. I looked into the reports, and it looks like exemplar reservoirs are expensive in terms of memory compared to the aggregation itself. Ideally, we shouldn't construct the reservoir when an AlwaysOff exemplar filter is used. I couldn't find a super clean way to do this... `exemplar.AlwaysOffFilter` is a function, so I couldn't easily compare it without comparing the pointers. I'm open to other suggestions. The internet says that compiler optimizations could cause this logic to fail, but worst-case you get the existing behavior. ### Benchmarks I needed a benchmark that creates a new attribute set for each call to be able to see how much memory each attribute set uses, so I added a new one, and ran it against main. Overall, it reduces the amount of memory used per-attribute-set by 70-88%, which is pretty substantial. Overall, ``` │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24 3.523µ ± 2% 3.745µ ± 29% +6.32% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24 3.714µ ± 6% 3.879µ ± 5% ~ (p=0.132 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24 3.811µ ± 8% 3.941µ ± 5% ~ (p=0.065 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24 3.714µ ± 5% 3.822µ ± 1% ~ (p=0.132 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24 3.126µ ± 4% 3.333µ ± 4% +6.61% (p=0.004 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24 3.099µ ± 4% 3.204µ ± 8% ~ (p=0.093 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24 3.745µ ± 7% 3.902µ ± 4% ~ (p=0.240 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24 3.746µ ± 7% 3.862µ ± 2% ~ (p=0.102 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24 3.621µ ± 3% 1.665µ ± 1% -54.00% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24 3.639µ ± 2% 1.686µ ± 4% -53.68% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24 3.563µ ± 4% 1.700µ ± 4% -52.29% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24 3.634µ ± 1% 1.690µ ± 4% -53.49% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24 2.892µ ± 2% 2.005µ ± 2% -30.66% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24 2.962µ ± 5% 2.057µ ± 6% -30.55% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24 3.692µ ± 5% 1.599µ ± 2% -56.68% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24 3.651µ ± 3% 1.612µ ± 2% -55.86% (p=0.002 n=6) geomean 3.495µ 2.541µ -27.30% │ old.txt │ new.txt │ │ B/op │ B/op vs base │ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24 3.107Ki ± 0% 3.107Ki ± 0% ~ (p=0.121 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24 3.108Ki ± 0% 3.107Ki ± 0% ~ (p=0.177 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24 3.108Ki ± 0% 3.108Ki ± 0% ~ (p=0.675 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24 3.107Ki ± 0% 3.107Ki ± 0% ~ (p=1.000 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24 2.622Ki ± 0% 2.622Ki ± 0% ~ (p=0.394 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24 2.622Ki ± 0% 2.622Ki ± 0% ~ (p=0.636 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24 3.108Ki ± 0% 3.107Ki ± 0% ~ (p=0.167 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24 3.107Ki ± 0% 3.108Ki ± 0% ~ (p=0.182 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24 3182.0 ± 0% 361.0 ± 0% -88.65% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24 3182.5 ± 0% 360.5 ± 0% -88.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24 3182.0 ± 0% 359.5 ± 1% -88.70% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24 3182.0 ± 0% 361.0 ± 0% -88.65% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24 2684.0 ± 0% 773.0 ± 0% -71.20% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24 2684.0 ± 0% 773.0 ± 0% -71.20% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24 3182.0 ± 0% 361.0 ± 1% -88.65% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24 3182.0 ± 0% 361.0 ± 0% -88.65% (p=0.002 n=6) geomean 2.978Ki 1.127Ki -62.17% │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Counter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Counter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64UpDownCounter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64UpDownCounter-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Histogram-24 13.00 ± 0% 13.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Histogram-24 13.00 ± 0% 13.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Int64Gauge-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOn/Float64Gauge-24 12.00 ± 0% 12.00 ± 0% ~ (p=1.000 n=6) ¹ BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Counter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Counter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64UpDownCounter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64UpDownCounter-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Histogram-24 13.000 ± 0% 9.000 ± 0% -30.77% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Histogram-24 13.000 ± 0% 9.000 ± 0% -30.77% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Int64Gauge-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) BenchmarkMeasureNewAttributeSet/AlwaysOff/Float64Gauge-24 12.000 ± 0% 7.000 ± 0% -41.67% (p=0.002 n=6) geomean 12.24 9.553 -21.97% ¹ all samples are equal ``` Gemini helped me write this. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-06-19 21:45:50 +02:00 · 2026-04-21 16:08:33 -04:00
parent 8365fb9660
commit 6f81801cd8
8 changed files with 159 additions and 7 deletions
@@ -36,6 +36,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - `ErrorType` in `go.opentelemetry.io/otel/semconv` now unwraps errors created with `fmt.Errorf` when deriving the `error.type` attribute. (#8133)
 - `go.opentelemetry.io/otel/sdk/log` now unwraps error chains created with `fmt.Errorf` when deriving the `error.type` attribute from errors on log records. (#8133)
 - `Set.MarshalLog` method in `go.opentelemetry.io/otel/attribute` now uses `Value.String` formatting following the [OpenTelemetry AnyValue representation for non-OTLP protocols](https://opentelemetry.io/docs/specs/otel/common/#anyvalue). (#8169)
+- Optimize `go.opentelemetry.io/otel/sdk/metric` to return a drop reservoir when `exemplar.AlwaysOffFilter` is configured. (#8211)

 ### Deprecated

@@ -732,3 +732,128 @@ func appendAttributes(kvs []attribute.KeyValue, number int) []attribute.KeyValue
 		panic("unknown number of attributes")
 	}
 }
+
+func BenchmarkMeasureNewAttributeSet(b *testing.B) {
+	ctx := b.Context()
+
+	instruments := []struct {
+		name string
+		make func(b *testing.B, meter metric.Meter) func(int)
+	}{
+		{
+			name: "Int64Counter",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Int64Counter("int64-counter")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Add(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Float64Counter",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Float64Counter("float64-counter")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Add(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Int64UpDownCounter",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Int64UpDownCounter("int64-up-down-counter")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Add(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Float64UpDownCounter",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Float64UpDownCounter("float64-up-down-counter")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Add(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Int64Histogram",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Int64Histogram("int64-histogram")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Record(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Float64Histogram",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Float64Histogram("float64-histogram")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Record(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Int64Gauge",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Int64Gauge("int64-gauge")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Record(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+		{
+			name: "Float64Gauge",
+			make: func(b *testing.B, meter metric.Meter) func(int) {
+				cnt, err := meter.Float64Gauge("float64-gauge")
+				assert.NoError(b, err)
+				return func(n int) {
+					cnt.Record(ctx, 1, metric.WithAttributes(attribute.Int("id", n)))
+				}
+			},
+		},
+	}
+
+	for _, filterName := range []string{"AlwaysOn", "AlwaysOff"} {
+		var filter exemplar.Filter
+		if filterName == "AlwaysOn" {
+			filter = exemplar.AlwaysOnFilter
+		} else {
+			filter = exemplar.AlwaysOffFilter
+		}
+
+		b.Run(filterName, func(b *testing.B) {
+			for _, inst := range instruments {
+				b.Run(inst.name, func(b *testing.B) {
+					var record func(int)
+
+					b.ReportAllocs()
+					b.ResetTimer()
+
+					for n := 0; n < b.N; n++ {
+						if n%10000 == 0 {
+							b.StopTimer()
+							rdr := NewManualReader()
+							provider := NewMeterProvider(
+								WithReader(rdr),
+								WithExemplarFilter(filter),
+							)
+							meter := provider.Meter("BenchmarkMeasureNewAttributeSet")
+							record = inst.make(b, meter)
+							b.StartTimer()
+						}
+						record(n)
+					}
+				})
+			}
+		})
+	}
+}
@@ -4,6 +4,7 @@
 package metric // import "go.opentelemetry.io/otel/sdk/metric"

 import (
+	"reflect"
 	"runtime"

 	"go.opentelemetry.io/otel/attribute"
@@ -22,6 +23,9 @@ func reservoirFunc[N int64 | float64](
 	provider exemplar.ReservoirProvider,
 	filter exemplar.Filter,
 ) func(attribute.Set) aggregate.FilteredExemplarReservoir[N] {
+	if reflect.ValueOf(filter).Pointer() == reflect.ValueOf(exemplar.AlwaysOffFilter).Pointer() {
+		return aggregate.DropReservoir[N]
+	}
 	return func(attrs attribute.Set) aggregate.FilteredExemplarReservoir[N] {
 		return aggregate.NewFilteredExemplarReservoir[N](filter, provider(attrs))
 	}
@@ -11,6 +11,8 @@ import (

 	"github.com/stretchr/testify/require"

+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/sdk/metric/exemplar"
 	"go.opentelemetry.io/otel/sdk/metric/metricdata"
 )

@@ -59,3 +61,23 @@ func TestFixedSizeExemplarConcurrentSafe(t *testing.T) {
 	cancel()
 	wg.Wait()
 }
+
+func TestReservoirFuncAlwaysOff(t *testing.T) {
+	var invoked bool
+	provider := func(attribute.Set) exemplar.Reservoir {
+		invoked = true
+		return nil
+	}
+
+	f := reservoirFunc[int64](provider, exemplar.AlwaysOffFilter)
+	_ = f(*attribute.EmptySet())
+
+	require.False(t, invoked, "ReservoirProvider should not be invoked when AlwaysOffFilter is used")
+
+	// Test non-off filter
+	invoked = false
+	f = reservoirFunc[int64](provider, exemplar.AlwaysOnFilter)
+	_ = f(*attribute.EmptySet())
+
+	require.True(t, invoked, "ReservoirProvider should be invoked when AlwaysOnFilter is used")
+}
@@ -36,8 +36,8 @@ type Builder[N int64 | float64] struct {
 	// ReservoirFunc is the factory function used by aggregate functions to
 	// create new exemplar reservoirs for a new seen attribute set.
 	//
-	// If this is not provided a default factory function that returns an
-	// dropReservoir reservoir will be used.
+	// If this is not provided a default factory function that returns a
+	// DropReservoir reservoir will be used.
 	ReservoirFunc func(attribute.Set) FilteredExemplarReservoir[N]
 	// AggregationLimit is the cardinality limit of measurement attributes. Any
 	// measurement for new attributes once the limit has been reached will be
@@ -54,7 +54,7 @@ func (b Builder[N]) resFunc() func(attribute.Set) FilteredExemplarReservoir[N] {
 		return b.ReservoirFunc
 	}

-	return dropReservoir
+	return DropReservoir
 }

 type fltrMeasure[N int64 | float64] func(ctx context.Context, value N, fltrAttr attribute.Set, droppedAttr []attribute.KeyValue)
@@ -74,7 +74,7 @@ func (c *clock) Register() (unregister func()) {
 }

 func dropExemplars[N int64 | float64](attr attribute.Set) FilteredExemplarReservoir[N] {
-	return dropReservoir[N](attr)
+	return DropReservoir[N](attr)
 }

 func TestBuilderFilter(t *testing.T) {
@@ -10,8 +10,8 @@ import (
 	"go.opentelemetry.io/otel/sdk/metric/exemplar"
 )

-// dropReservoir returns a [FilteredReservoir] that drops all measurements it is offered.
-func dropReservoir[N int64 | float64](attribute.Set) FilteredExemplarReservoir[N] {
+// DropReservoir returns a [FilteredExemplarReservoir] that drops all measurements it is offered.
+func DropReservoir[N int64 | float64](attribute.Set) FilteredExemplarReservoir[N] {
 	return &dropRes[N]{}
 }

@@ -18,7 +18,7 @@ func TestDrop(t *testing.T) {
 }

 func testDropFiltered[N int64 | float64](t *testing.T) {
-	r := dropReservoir[N](*attribute.EmptySet())
+	r := DropReservoir[N](*attribute.EmptySet())

 	var dest []exemplar.Exemplar
 	r.Collect(&dest)