From
https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/sdk.md#export:
> `Export` will never be called concurrently for the same exporter
instance.
From our SDK perspective this change will make
https://pkg.go.dev/go.opentelemetry.io/otel/exporters/stdout/stdoutlog
concurrent-safe when used with the simple processor. Before the change,
there can be multiple goroutines calling `Write` in parallel (via
`json.Encoder.Encode`).
### Remarks
[Maybe we should simply state that "whole" exporter implementation does
not need to be
concurrent-safe?](https://github.com/open-telemetry/opentelemetry-specification/pull/4173#discussion_r1699958752)
However:
1. we would need to make complex changes in `BatchExporter` to
synchronize the `Export`, `Shutdown`, `ForceFlush` calls
2. we would need to update all exporters (remove synchronization) and
simple exporter (add locks to `Shutdown`, `ForceFlush`)
3. I am 100% not sure if this would be compliant with the specification
- I think it would be complaint because we would simply give stronger
safety-measures
We should probably discuss it separately, but I wanted to highlight my
though process.
Even if we decide that simple and batch processors to synchronize all
calls then I would prefer to address it in a separate PR.
Related spec clarification PR:
-
https://github.com/open-telemetry/opentelemetry-specification/pull/4173
* [WIP] Implement the BatchingProcessor
* Add TestExportSync
* Add TestChunker
* Test export error default to ErrorHandler
* Fix lint
* Fix chunk smaller than size error
* Add batch tests
* Fix lint
* Update OnEmit test
Check the len of records in eventually assertion given that is what we
are going to measure.
* Revert unneeded change to BatchingProcessor doc
* Add batch type
* Refactor testing of batching config
The BatchingProcessor is not expected to ultimately contain
configuration fields for queue size or export parameters (see #5093).
This will break TestNewBatchingProcessorConfiguration which tests the
configuration by evaluating the BatchingProcessor directly.
Instead, test the batchingConfig and rename the test to
TestNewBatchingConfig to match what is being tested.
* Implement the BatchingProcessor without polling
* Add TestBatchingProcessor
* Add ConcurrentSafe test
* Expand Shutdown tests
* Test context canceled for ForceFlush
* Refactor batch to queue
* Use exportSync
* Update docs and naming
* Split buffered export to its own type
* Update comments and naming
* Fix lint
* Remove redundant triggered type
* Add interval polling
* Refactor test structure
* Add custom ring implimementation
* Add BenchmarkBatchingProcessor
* Fix merge
* Remove custom ring impl
* Remove BenchmarkBatchingProcessor
* Update dev docs
* Test nil exporter
* Update OnEmit test
Ensure the poll goroutine will completely flush the queue of batches.
* Test RetriggerFlushNonBlocking
* Update ascii diagram
* Fix flaky OnEmit
* Revert unnecessary change to test pkg name
* Use batching term in docs
* Document EnqueueExport
* Return from EnqueueExport if blocked
Do not wait for the enqueue to succeed.
* Do not drop failed flush log records
* Use cancelable ctx in concurrency test
* Fix comments
* Apply feedback
Do not spawn a goroutine for the flush operation.
* Return true from EnqueueExport when stopped
* Update sdk/log/batch.go
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Remove TODO
* Comment re-trigger in poll
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Add chunker exporter
The batching log processor needs to be able to export payloads in
chuncks. This adds a chunker type that will forward all Shutdown and
ForceFlush calls to the embedded exporter and chunk data passed to
Export.
* Concurrent safe testExporter
* Add test for zero size
* Fix lint
* Refactor chunker into chunkExporter
* Remove ExportTrigger
* Add export sync
The batching log processor will generate records from 4 different
locations (polling, OnEmit, ForceFlush, Shutdown). In order to ensure an
Exporter is called serially, as is required by the interface, this
function will be used in the processor to centralize the interaction
with its Exporter.
Part of #5063.
See #5093 for the implementation use.
* Concurrent safe testExporter
---------
Co-authored-by: Sam Xie <sam@samxie.me>
