Fix#5342
Run of `go test -v -timeout 10s -short -count=10000 ./...
-run='TestBatchProcessor/ForceFlush/ErrorPartialFlush'`
**Before**:
Failed with `panic: test timed out after 10s`
**After**:
Passed
Problem was with `bufferExporter.input` chan.
In test expected:
> 1 export being performed, 1 export in buffer chan, >1 batch
But buffer chan wasn't checked. Now we ensure, that record is in buffer
chan
Co-authored-by: Tyler Yahn <MrAlias@users.noreply.github.com>
Fix#5384
Run of `go test -count=1000000 -run="TestBatchProcessor/DroppedLogs"`
**Before**:
Failed with either `Condition never satisfied` or panic
**After**:
Passed
First, bytes.Buffer is not thread-safe, so writing log and reading
(`bytes.String()`) caused panic. Added `concurrentBuffer`
Second, fixed flaky test with 4 records:
1. Record goes to `testExporter.Export` function and blocks in this
function because of `ExportTrigger`
(19ee6d4775/sdk/log/exporter_test.go (L87))
2. Record goes to `bufferExporter.input`
(7c5e64cccc/sdk/log/exporter.go (L129))
3. Record goes to `BatchProcessor.q` queue and it could not be enqueued
to export, because `bufferExporter.input` is full
4. Record goes to `BatchProcessor.q` and because of overfill, drops
third record
---------
Co-authored-by: Sam Xie <sam@samxie.me>
Fix#5384
On single-threaded, or in general slow, systems where the log message
may not be emitted, use assert.Everything to allow multiple lookups for
the expected log message.
Co-authored-by: Sam Xie <sam@samxie.me>
* [WIP] Implement the BatchingProcessor
* Add TestExportSync
* Add TestChunker
* Test export error default to ErrorHandler
* Fix lint
* Fix chunk smaller than size error
* Add batch tests
* Fix lint
* Update OnEmit test
Check the len of records in eventually assertion given that is what we
are going to measure.
* Revert unneeded change to BatchingProcessor doc
* Add batch type
* Refactor testing of batching config
The BatchingProcessor is not expected to ultimately contain
configuration fields for queue size or export parameters (see #5093).
This will break TestNewBatchingProcessorConfiguration which tests the
configuration by evaluating the BatchingProcessor directly.
Instead, test the batchingConfig and rename the test to
TestNewBatchingConfig to match what is being tested.
* Implement the BatchingProcessor without polling
* Add TestBatchingProcessor
* Add ConcurrentSafe test
* Expand Shutdown tests
* Test context canceled for ForceFlush
* Refactor batch to queue
* Use exportSync
* Update docs and naming
* Split buffered export to its own type
* Update comments and naming
* Fix lint
* Remove redundant triggered type
* Add interval polling
* Refactor test structure
* Add custom ring implimementation
* Add BenchmarkBatchingProcessor
* Fix merge
* Remove custom ring impl
* Remove BenchmarkBatchingProcessor
* Update dev docs
* Test nil exporter
* Update OnEmit test
Ensure the poll goroutine will completely flush the queue of batches.
* Test RetriggerFlushNonBlocking
* Update ascii diagram
* Fix flaky OnEmit
* Revert unnecessary change to test pkg name
* Use batching term in docs
* Document EnqueueExport
* Return from EnqueueExport if blocked
Do not wait for the enqueue to succeed.
* Do not drop failed flush log records
* Use cancelable ctx in concurrency test
* Fix comments
* Apply feedback
Do not spawn a goroutine for the flush operation.
* Return true from EnqueueExport when stopped
* Update sdk/log/batch.go
Co-authored-by: Robert Pająk <pellared@hotmail.com>
* Remove TODO
* Comment re-trigger in poll
---------
Co-authored-by: Robert Pająk <pellared@hotmail.com>
The BatchingProcessor is not expected to ultimately contain
configuration fields for queue size or export parameters (see #5093).
This will break TestNewBatchingProcessorConfiguration which tests the
configuration by evaluating the BatchingProcessor directly.
Instead, test the batchingConfig and rename the test to
TestNewBatchingConfig to match what is being tested.
