Closes#5425
Our current log `Processor` interface contains more functionality than
the [OTel
spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/sdk.md#logrecordprocessor-operations).
The additional functionality allows processors to report back to the API
if a Record should be constructed and emitted or not, which is quite
helpful[^1][^2][^3][^4][^5].
This removes the `Enabled` method from the `Processor` type. It adds
this functionality a new optional and experimental `FilterProcessor`
interface type. The logger and provider are updated to check for this
optional interface to be implemented with the configured processors and
uses them to back the `Logger.Enabled` method, preserving existing
functionality.
By making this change:
- The `Processor` interface is now compliant with the OTel spec and does
not contain any additional unspecified behavior.
- All `Processor` implementations are no longer required to implement an
`Enabled` method. The default, when they do not implement this method,
is to assume they are enabled.
### Benchmark
```terminal
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/log
cpu: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
│ old.txt │ new7.txt │
│ sec/op │ sec/op vs base │
LoggerEnabled-8 133.30n ± 3% 32.36n ± 3% -75.72% (p=0.000 n=10)
│ old.txt │ new7.txt │
│ B/op │ B/op vs base │
LoggerEnabled-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ old.txt │ new7.txt │
│ allocs/op │ allocs/op vs base │
LoggerEnabled-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
```
This is a significant performance improvement due to the `Record` no
longer being converted from the API version to the SDK version.
[^1]: https://pkg.go.dev/go.opentelemetry.io/contrib/processors/minsev
[^2]:
https://pkg.go.dev/go.opentelemetry.io/otel/sdk/log#BatchProcessor.Enabled
[^3]:
https://pkg.go.dev/go.opentelemetry.io/otel/sdk/log#SimpleProcessor.Enabled
[^4]:
af75717ac4/bridges/otelslog/handler.go (L206-L211)
[^5]:
d0309ddd8c/bridges/otelzap/core.go (L142-L146)
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: Sam Xie <sam@samxie.me>
Fix#5342
Run of `go test -v -timeout 10s -short -count=10000 ./...
-run='TestBatchProcessor/ForceFlush/ErrorPartialFlush'`
**Before**:
Failed with `panic: test timed out after 10s`
**After**:
Passed
Problem was with `bufferExporter.input` chan.
In test expected:
> 1 export being performed, 1 export in buffer chan, >1 batch
But buffer chan wasn't checked. Now we ensure, that record is in buffer
chan
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
Fix#5384
Run of `go test -count=1000000 -run="TestBatchProcessor/DroppedLogs"`
**Before**:
Failed with either `Condition never satisfied` or panic
**After**:
Passed
First, bytes.Buffer is not thread-safe, so writing log and reading
(`bytes.String()`) caused panic. Added `concurrentBuffer`
Second, fixed flaky test with 4 records:
1. Record goes to `testExporter.Export` function and blocks in this
function because of `ExportTrigger`
(19ee6d4775/sdk/log/exporter_test.go (L87))
2. Record goes to `bufferExporter.input`
(7c5e64cccc/sdk/log/exporter.go (L129))
3. Record goes to `BatchProcessor.q` queue and it could not be enqueued
to export, because `bufferExporter.input` is full
4. Record goes to `BatchProcessor.q` and because of overfill, drops
third record
---------
Co-authored-by: Sam Xie <sam@samxie.me>
Fix#5384
On single-threaded, or in general slow, systems where the log message
may not be emitted, use assert.Everything to allow multiple lookups for
the expected log message.
Co-authored-by: Sam Xie <sam@samxie.me>
* [WIP] Implement the BatchingProcessor
* Add TestExportSync
* Add TestChunker
* Test export error default to ErrorHandler
* Fix lint
* Fix chunk smaller than size error
* Add batch tests
* Fix lint
* Update OnEmit test
Check the len of records in eventually assertion given that is what we
are going to measure.
* Revert unneeded change to BatchingProcessor doc
* Add batch type
* Refactor testing of batching config
The BatchingProcessor is not expected to ultimately contain
configuration fields for queue size or export parameters (see #5093).
This will break TestNewBatchingProcessorConfiguration which tests the
configuration by evaluating the BatchingProcessor directly.
Instead, test the batchingConfig and rename the test to
TestNewBatchingConfig to match what is being tested.
* Implement the BatchingProcessor without polling
* Add TestBatchingProcessor
* Add ConcurrentSafe test
* Expand Shutdown tests
* Test context canceled for ForceFlush
* Refactor batch to queue
* Use exportSync
* Update docs and naming
* Split buffered export to its own type
* Update comments and naming
* Fix lint
* Remove redundant triggered type
* Add interval polling
* Refactor test structure
* Add custom ring implimementation
* Add BenchmarkBatchingProcessor
* Fix merge
* Remove custom ring impl
* Remove BenchmarkBatchingProcessor
* Update dev docs
* Test nil exporter
* Update OnEmit test
Ensure the poll goroutine will completely flush the queue of batches.
* Test RetriggerFlushNonBlocking
* Update ascii diagram
* Fix flaky OnEmit
* Revert unnecessary change to test pkg name
* Use batching term in docs
* Document EnqueueExport
* Return from EnqueueExport if blocked
Do not wait for the enqueue to succeed.
* Do not drop failed flush log records
* Use cancelable ctx in concurrency test
* Fix comments
* Apply feedback
Do not spawn a goroutine for the flush operation.
* Return true from EnqueueExport when stopped
* Update sdk/log/batch.go
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Remove TODO
* Comment re-trigger in poll
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
The BatchingProcessor is not expected to ultimately contain
configuration fields for queue size or export parameters (see #5093).
This will break TestNewBatchingProcessorConfiguration which tests the
configuration by evaluating the BatchingProcessor directly.
Instead, test the batchingConfig and rename the test to
TestNewBatchingConfig to match what is being tested.
