8.2 KiB
Logs SDK
Abstract
go.opentelemetry.io/otel/sdk/log provides Logs SDK compliant with the
specification.
The prototype was created in #4955.
Background
The goal is to design the exported API of the SDK would have low performance overhead. Most importantly, have a design that reduces the amount of heap allocations and even make it possible to have a zero-allocation implementation. Eliminating the amount of heap allocations reduces the GC pressure which can produce some of the largest improvements in performance.1
The main and recommended use case is to configure the SDK to use an OTLP exporter with a batch processor.2 Therefore, the implementation aims to be high-performant in this scenario. Some users that require high throughput may also want to use e.g. an user_events, LLTng or ETW exporter with a simple processor. Users may also want to use OTLP File or Standard Output exporter in order to emit logs to standard output/error or files.
Modules structure
The SDK is published as a single go.opentelemetry.io/otel/sdk/log Go module.
The exporters are going to be published as following Go modules:
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpcgo.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttpgo.opentelemetry.io/otel/exporters/stdout/stdoutlog
LoggerProvider
The LoggerProvider
is implemented as LoggerProvider struct in provider.go.
LogRecord limits
The LogRecord limits can be configured using following options:
func WithAttributeCountLimit(limit int) LoggerProviderOption
func WithAttributeValueLengthLimit(limit int) LoggerProviderOption
The limits can be also configured using the OTEL_LOGRECORD_* environment variables as
defined by the specification.
Processor
The LogRecordProcessor
is defined as Processor interface in processor.go.
The user set processors for the LoggerProvider using
func WithProcessor(processor Processor) LoggerProviderOption.
The user can configure custom processors and decorate built-in processors.
The specification may add new operations to the LogRecordProcessor. If it happens, CONTRIBUTING.md describes how the SDK can be extended in a backwards-compatible way.
SimpleProcessor
The Simple processor
is implemented as SimpleProcessor struct in simple.go.
BatchProcessor
The Batching processor
is implemented as BatchProcessor struct in batch.go.
The Batcher can be also configured using the OTEL_BLRP_* environment variables as
defined by the specification.
Exporter
The LogRecordExporter
is defined as Exporter interface in exporter.go.
The slice passed to Export must not be retained by the implementation
(like e.g. io.Writer)
so that the caller can reuse the passed slice
(e.g. using sync.Pool)
to avoid heap allocations on each call.
The specification may add new operations to the LogRecordExporter. If it happens, CONTRIBUTING.md describes how the SDK can be extended in a backwards-compatible way.
Record
The ReadWriteLogRecord
is defined as Record struct in record.go.
The Record is designed similarly to log.Record
in order to reduce the number of heap allocations when processing attributes.
The SDK does not have have an additional definition of ReadableLogRecord as the specification does not say that the exporters must not be able to modify the log records. It simply requires them to be able to read the log records. Having less abstractions reduces the API surface and makes the design simpler.
Benchmarking
The benchmarks are supposed to test end-to-end scenarios and avoid I/O that could affect the stability of the results.
The benchmark results can be found in the prototype.
Rejected alternatives
Represent both LogRecordProcessor and LogRecordExporter as Exporter
Because the LogRecordProcessor
and the LogRecordProcessor
abstractions are so similar, there was a proposal to unify them under
single Exporter interface.3
However, introducing a Processor interface makes it easier
to create custom processor decorators4
and makes the design more aligned with the specification.
Embed log.Record
Because Record and log.Record
are very similar, there was a proposal to embed log.Record in Record definition.
log.Record
supports only adding attributes.
In the SDK, we also need to be able to modify the attributes (e.g. removal)
provided via API.
Moreover it is safer to have these abstraction decoupled. E.g. there can be a need for some fields that can be set via API and cannot be modified by the processors.
Processor.OnEmit to accept Record values
There was a proposal to make the Processor's OnEmit
to accept a Record value instead of a pointer to reduce allocations
as well as to have design similar to slog.Handler.
There have been long discussions within the OpenTelemetry Specification SIG5 about whether such a design would comply with the specification. The summary was that the current processor design flaws are present in other languages as well. Therefore, it would be favorable to introduce new processing concepts (e.g. chaining processors) in the specification that would coexist with the current "mutable" processor design.
The performance disadvantages caused by using a pointer (which at the time of writing causes an additional heap allocation) may be mitigated by future versions of the Go compiler, thanks to improved escape analysis and profile-guided optimization (PGO)6.
On the other hand, Processor's Enabled is fine to accept
a Record value as the processors should not mutate the passed
parameters.