This fixes issue #6344, in which an attempted clean shutdown of the
metrics SDK using PeriodicReader's Shutdown method can produce and emit
wildly incorrect data point values that are orders of magnitude too
large.
The root of the issue is that the pipeline produce method changed in
this PR cannot safely return early, which was happening in the callback
loops that were checking for context cancellation. Early return is not
safe since callbacks and aggregations are tightly coupled in practice:
invoking callbacks without also invoking aggregations corrupts internal
data point value accounting.
The linked issue more concretely walks through the sequence of steps
that were causing this issue.
---------
Co-authored-by: Damien Mathieu <42@dmathieu.com>
```go
type interInst struct {
x int
}
type inter interface {
}
var sink []inter
func BenchmarkX(b *testing.B) {
sink = make([]inter, b.N)
for i := 0; i < b.N; i++ {
sink[i] = &interInst{}
}
clear(sink)
sink = sink[:0]
runtime.GC()
var ms runtime.MemStats
runtime.ReadMemStats(&ms)
b.Log(b.N, ms.Frees) // Frees is the cumulative count of heap objects freed.
}
```
```
clear:
ioz_test.go:35: 1 589
ioz_test.go:35: 100 711
ioz_test.go:35: 10000 10729
ioz_test.go:35: 1000000 1010750 <-- 1m+ freed
ioz_test.go:35: 16076874 17087643
ioz_test.go:35: 19514749 36602412
```
```
no clear:
ioz_test.go:35: 1 585
ioz_test.go:35: 100 606
ioz_test.go:35: 10000 725
ioz_test.go:35: 1000000 10745 <-- some "overhead" objects freed, not the slice.
ioz_test.go:35: 16391445 1010765
ioz_test.go:35: 21765238 17402230
```
This is documented at https://go.dev/wiki/SliceTricks:
> NOTE If the type of the element is a pointer or a struct with pointer
fields, which need to be garbage collected, the above implementations of
Cut and Delete have a potential memory leak problem: some elements with
values are still referenced by slice a’s underlying array, just not
“visible” in the slice. Because the “deleted” value is referenced in the
underlying array, the deleted value is still “reachable” during GC, even
though the value cannot be referenced by your code. If the underlying
array is long-lived, this represents a leak.
Followed by examples of how zeroing out the slice elements solves
the problem. This PR does the same.
Follow-up to
https://github.com/open-telemetry/opentelemetry-go/pull/5861. This is an
attempt to:
* Limit the API surface of the aggregate package
* Try to use predefined types (e.g. exemplar.Filter) over custom
functions where possible.
* Avoid using nil, and use No-Ops where it makes sense
This makes `aggregate.NewFilteredExemplarReservoir` no longer exported,
removes the `aggregate.FilteredExemplarReservoir` interface, and removes
the `aggregate.dropReservoir`.
Resolve https://github.com/open-telemetry/opentelemetry-go/issues/5249
### Spec
> exemplar_reservoir: A functional type that generates an exemplar
reservoir a MeterProvider will use when storing exemplars. This
functional type needs to be a factory or callback similar to aggregation
selection functionality which allows different reservoirs to be chosen
by the aggregation.
> Users can provide an exemplar_reservoir, but it is up to their
discretion. Therefore, the stream configuration parameter needs to be
structured to accept an exemplar_reservoir, but MUST NOT obligate a user
to provide one. If the user does not provide an exemplar_reservoir
value, the MeterProvider MUST apply a [default exemplar
reservoir](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#exemplar-defaults).
Also,
> the reservoir MUST be given the Attributes associated with its
timeseries point either at construction so that additional sampling
performed by the reservoir has access to all attributes from a
measurement in the "offer" method.
### Changes
In sdk/metric/exemplar, add:
* `exemplar.ReservoirProvider`
* `exemplar.HistogramReservoirProvider`
* `exemplar.FixedSizeReservoirProvider`
In sdk/metric, add:
* `metric.ExemplarReservoirProviderSelector` (func Aggregation ->
ReservoirProvider)
* `metric.DefaultExemplarReservoirProviderSelector` (our default
implementation)
* `ExemplarReservoirProviderSelector` added to `metric.Stream`
Note: the only required public types are
`metric.ExemplarReservoirProviderSelector` and
`ExemplarReservoirProviderSelector` in `metric.Stream`. Others are for
convenience and readability.
### Alternatives considered
#### Add ExemplarReservoirProvider directly to metric.Stream, instead of
ExemplarReservoirProviderSelector
This would mean users can configure a `func() exemplar.Reservoir`
instead of a `func(Aggregation) func() exemplar.Reservoir`.
I don't think this complies with the statement: `This functional type
needs to be a factory or callback similar to aggregation selection
functionality which allows different reservoirs to be chosen by the
aggregation.`. I'm interpreting "allows different reservoirs to be
chosen by the aggregation" as meaning "allows different reservoirs to be
chosen **based on the** aggregation", rather than meaning that the
aggregation is somehow choosing the reservoir.
### Future work
There is some refactoring I plan to do after this to simplify the
interaction between the internal/aggregate and exemplar package. I've
omitted that from this PR to keep the diff smaller.
---------
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Refactor exemplars to not use generic argument
* Update internal/aggregate
* Update metric SDK
* Test exemplar value type
* Add TestCollectExemplars
* Fix lint
---------
Co-authored-by: Sam Xie <sam@samxie.me>
* Add the experimental exemplar feature
* Add exemplars to EXPERIMENTAL.md
* Add changelog entry
* Fix hist buckets > 1 detection
* Collect instead of Flush res about to be deleted
* Add e2e test
* Do not pre-alloc ResourceMetrics
This only has a single use.
* Fix grammatical error in comment
* Add test cases
Default and invalid OTEL_METRICS_EXEMPLAR_FILTER.
Test sampled and non-sampled context for trace_based.
* Comment nCPU
* Doc OTEL_METRICS_EXEMPLAR_FILTER
* Add agg limiting func
* Add unit test for limitAttr
* Add limiting to aggregate types
* Add internal x pkg for experimental feature-flagging
* Connect cardinality limit to metric SDK
* Replace limitAttr fn with limiter type
The Attribute method is still inlinable.
* Use x.CardinalityLimit directly
* Simplify limiter test
* Add limiter benchmark
* Document the AggregationLimit field
* Test sum limits
* Test limit for last value
* Test histogram limit
* Refactor expo hist test to use existing fixtures
The tests for the exponential histogram create their own testing
fixtures. There is nothing these new fixtures do that cannot already be
done with the existing testing fixtures used by all the other aggregate
functions. Unify the exponential histogram testing to use the existing
fixtures.
* Test the ExponentialHistogram limit
* Fix lint
* Add docs
* Rename aggregation field to aggLimit
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Add allow/deny attr filters
* Revert "Replace `Stream.AttributeFilter` with `AllowAttributeKeys` (#4288)"
This reverts commit 1633c74aea.
* Rename new attr filter funcs
Do not include the term "Attribute" in a creation function of the
"attribute" pkg.
* Update the AttributeFilter field documentation
* Add tests for filter creation funcs
* Add change to changelog
* Apply feedback
* Use NewDenyKeysFilter for allow-all and deny-list filters
* Remove links from field docs
These links do not render.
---------
Co-authored-by: Chester Cheung <cheung.zhy.csu@gmail.com>
* Deprecate the aggregation pkg
* Decouple the internal/aggregate from aggregation pkg
* Add Aggregation to the metric pkg
* Do not use sdk/metric/aggregation in stdoutmetric exporter
* Update all generated templates
* Update prom exporter
* Fix view example
* Add changes to changelog
* Update CHANGELOG.md
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Rename Sum to AggregationSum
* Fix comments
* Centralize validation of aggregation in pipeline
* Remove validation of agg in manual_reader selector opt
* Fix merge
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Adds Exponential Histograms aggregator
* Added aggregation to the pipeline.
Adjust to new bucket
* Add no allocation if cap is available.
* Expand tests
* Fix lint
* Fix 64 bit math on 386 platform.
* Fix tests to work in go 1.19.
Fix spelling error
* fix codespell
* Add example
* Update sdk/metric/aggregation/aggregation.go
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Update sdk/metric/aggregation/aggregation.go
* Update sdk/metric/aggregation/aggregation.go
* Changelog
* Fix move
* Address feedback from the PR.
* Update expo histo to new aggregator format.
* Fix lint
* Remove Zero Threshold from config of expo histograms
* Remove DefaultExponentialHistogram()
* Refactor GetBin, and address PR Feedback
* Address PR feedback
* Fix comment in wrong location
* Fix misapplied PR feedback
* Fix codespell
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: Chester Cheung <cheung.zhy.csu@gmail.com>
* Detect dup inst for case-insensitive names
Resolve#3835
Detect duplicate instrument registrations for instruments that have the
same case-insensitive names. Continue to return the instruments with
different names, but log a warning. This is the solution proposed in
https://github.com/open-telemetry/opentelemetry-specification/pull/3606.
* Add changes to changelog
* Reset global logger after test
* Use inst ID for agg cache key
Resolve#4201
The specification requires the duplicate instrument conflicts to be
identified based on the instrument identifying fields:
- name
- instrument kind
- unit
- description
- language-level features such as the number type (int64 and float64)
Currently, the conflict detection and aggregation caching are done based
on the stream IDs which include an aggregation name, monotonicity, and
temporality instead of the instrument kind.
This changes the conflict detection and aggregation caching to use the
OpenTelemetry specified fields. This is effectively a no-op given there
is a 1-to-1 mapping of aggregation-name/monotonicity/temporality to
instrument kind (they are all resolved based on the instrument kind).
Additionally, this adds a stringer representation of the
`InstrumentKind`. This is needed for the logging of duplicate instrument
conflicts.
* Add changes to changelog
* Allow histogram for all instruments
Any instrument that can record negative values, do not include a sum in
the produced aggregation (like the specification recommends).
Resolves#4161
* Add changes to changelog
* Fix TestBucketsSum
* Replace Stream AttributeFilter with AttributeKeys
* Rename Stream field AttributeKeys to AllowAttributeKeys
Ensure forward compatibility if a deny-list of attribute keys is ever
added.
* Add change to changelog
* Update PR number in changelog
* Update CHANGELOG.md
Co-authored-by: Damien Mathieu <42@dmathieu.com>
---------
Co-authored-by: Damien Mathieu <42@dmathieu.com>
* Use newPipeline instead of direct construction
The newPipeline function exists to create a new pipeline, but it is not
currently used. This switches to using that as the default creation
method and removes the test for construction of a pipeline directly.
* Update sdk/metric/pipeline.go
* fix aggregation.Default to properly return the default one
* add changelog entry
* default aggregation does not error anymore
* test all instrument kinds
* PoC of embedded private method ifaces
* Rename embed to embedded
* Add an embedded iface for all instruments
* Fix metric/instrument tests
* Fix global and otel
* Fix SDK
* Comment the embedded pkg types
* Update the embedded pkg docs
* Update otel/metric docs about impls
* Update otel/metric type docs on impl
* Update docs in otel/metric/instrument on default
* Add changes to changelog
* Apply suggestions from code review
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Apply feedback on URLs
* Reword based on feedback
* Make it clear we only recommended embedding noop
* Ignore links with godot linter
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Replace Unit from metric/unit with string
Deprecate the units package. This package will not be included in the
metric GA.
* Add changes to changelog
* Merge instrument cache to inserter
The current pipeline resolution path will only add the resolved
aggregators to the pipeline when it creates one (cache miss). It will
not add it if there is a cache hit. This means (since we cache
instruments at the meter level, not the pipeline level) the first reader
in a multiple-reader setup is the only one that will collect data for
that aggregator. All other readers will have a cache hit and nothing is
added to the pipeline. This is causing #3720.
This resolves#3720 by moving the instrument caching into the inserter.
This means aggregators are cached at the reader level, not the meter.
* Rename aggCV to aggVal
---------
Co-authored-by: Chester Cheung <cheung.zhy.csu@gmail.com>
An instrument is defined by a name, description, unit, and kind. The
instrumentID contains more identifying information than these fields.
The additional information it contains relate to what the OTel metric
data-model calls a stream. Match the terminology and remove using the
instrumentID name in case we want to use it for something else (say,
when caching instruments).
* Update RegisterCallback and Callback declerations
RegisterCallback accepts variadic Asynchronous instruments instead of a
slice.
Callback accepts an observation result recorder to ensure instruments
that are observed by a callback.
* Update global, noop, SDK implementations
* Fix examples
* Add changes to changelog
* Test RegisterCallback for invalid observers
* Test callbacks from foreign sources not collected
* Support registering delegating instruments
* Split metric inst config
Instead of having the same configuration for both the Synchronous and
Asynchronous instruments, use specific options for both.
* Use Async/Sync opt for appropriate inst
* Update noop inst providers
* Update internal global impl
* Update sdk
* Remove unused method for callbackOption
* Test instrument configuration
* Lint imports
* Add changes to changelog
* Refactor callbacks and further split opts
Define callbacks to return the value observed. Because of the different
types returned for different observables, the callbacks and options are
move to the sync/async packages.
* Update noop impl
* Fix example_test.go
* Fix internal impl
* Update Callbacks
Return observations for distinct attr sets.
* Refactor common code in sdk/metric inst provider
* Update examples and prom exporter
* Generalize callback
* Update changelog
* Add unit tests for callback
* Add meter tests for cbacks on creation
* Rename Observations to Measurements
* Update Callback to accept an Observer
* Update SDK impl
* Move conf to instrument pkg
* Apply suggestions from code review
* Create metric API Callback type
Document the type according the OTel specification requirements.
* Update all impls of the metric API with new type
* Add changes to changelog
* Update PR number in changelog entry
* Update Meter RegisterCallback method
Return a Registration from the method that can be used by the caller to
unregister their callback.
Update documentation of the method to better explain expectations of
use and implementation.
* Update noop impl
* Update global impl
* Test global Unregister concurrent safe
* Use a map to track reg in global impl
* Update sdk impl
* Use a list for global impl
* Fix prom example
* Lint metric/meter.go
* Fix metric example
* Placeholder for changelog
* Update PR number in changelog
* Update sdk/metric/pipeline.go
Co-authored-by: Aaron Clawson <3766680+MadVikingGod@users.noreply.github.com>
* Add test unregistered callback is not called
Co-authored-by: Aaron Clawson <3766680+MadVikingGod@users.noreply.github.com>
* Replace view usage in sdk/metric
* Replace view use in stdoutmetric
* Replace view use in prometheus exporter
* Replace view use in otlpmetric exporters
* Replace view use in view example
