opentelemetry-go/sdk/log/DESIGN.md

Logs SDK

Abstract

go.opentelemetry.io/otel/sdk/log provides Logs SDK compliant with the specification.

The prototype was created in #4955.

Background

The goal is to design the exported API of the SDK would have low performance overhead. Most importantly, have a design that reduces the amount of heap allocations and even make it possible to have a zero-allocation implementation. Eliminating the amount of heap allocations reduces the GC pressure which can produce some of the largest improvements in performance.¹

The main and recommended use case is to configure the SDK to use an OTLP exporter with a batch processor.² Therefore, the implementation aims to be high-performant in this scenario. Some users that require high throughput may also want to use e.g. an user_events, LLTng or ETW exporter with a simple processor. Users may also want to use OTLP File or Standard Output exporter in order to emit logs to standard output/error or files.

Modules structure

The SDK is published as a single go.opentelemetry.io/otel/sdk/log Go module.

The exporters are going to be published as following Go modules:

• go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc
• go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
• go.opentelemetry.io/otel/exporters/stdout/stdoutlog

LoggerProvider

The LoggerProvider is implemented as LoggerProvider struct in provider.go.

LogRecord limits

The LogRecord limits can be configured using following options:

func WithAttributeCountLimit(limit int) LoggerProviderOption
func WithAttributeValueLengthLimit(limit int) LoggerProviderOption

The limits can be also configured using the OTEL_LOGRECORD_* environment variables as defined by the specification.

Processor

The LogRecordProcessor is defined as Processor interface in processor.go.

The user set processors for the LoggerProvider using func WithProcessor(processor Processor) LoggerProviderOption.

The user can configure custom processors and decorate built-in processors.

The specification may add new operations to the LogRecordProcessor. If it happens, CONTRIBUTING.md describes how the SDK can be extended in a backwards-compatible way.

SimpleProcessor

The Simple processor is implemented as SimpleProcessor struct in simple.go.

BatchProcessor

The Batching processor is implemented as BatchProcessor struct in batch.go.

The Batcher can be also configured using the OTEL_BLRP_* environment variables as defined by the specification.

Exporter

The LogRecordExporter is defined as Exporter interface in exporter.go.

The slice passed to Export must not be retained by the implementation (like e.g. io.Writer) so that the caller can reuse the passed slice (e.g. using sync.Pool) to avoid heap allocations on each call.

The specification may add new operations to the LogRecordExporter. If it happens, CONTRIBUTING.md describes how the SDK can be extended in a backwards-compatible way.

Record

The ReadWriteLogRecord is defined as Record struct in record.go.

The Record is designed similarly to log.Record in order to reduce the number of heap allocations when processing attributes.

The SDK does not have have an additional definition of ReadableLogRecord as the specification does not say that the exporters must not be able to modify the log records. It simply requires them to be able to read the log records. Having less abstractions reduces the API surface and makes the design simpler.

Benchmarking

The benchmarks are supposed to test end-to-end scenarios and avoid I/O that could affect the stability of the results.

The benchmark results can be found in the prototype.

Rejected alternatives

Represent both LogRecordProcessor and LogRecordExporter as Expoter

Because the LogRecordProcessor and the LogRecordProcessor abstractions are so similar, there was a proposal to unify them under single Expoter interface.³

However, introducing a Processor interface makes it easier to create custom processor decorators⁴ and makes the design more aligned with the specification.

Embed log.Record

Because Record and log.Record are very similar, there was a proposal to embed log.Record in Record definition.

log.Record supports only adding attributes. In the SDK, we also need to be able to modify the attributes (e.g. removal) provided via API.

Moreover it is safer to have these abstraction decoupled. E.g. there can be a need for some fields that can be set via API and cannot be modified by the processors.

Processor.OnEmit to accept Record values

There was a proposal to make the Processor's OnEmit to accept a Record value instead of a pointer to reduce allocations as well as to have design similar to slog.Handler.

There have been long discussions within the OpenTelemetry Specification SIG⁵ about whether such a design would comply with the specification. The summary was that the current processor design flaws are present in other languages as well. Therefore, it would be favorable to introduce new processing concepts (e.g. chaining processors) in the specification that would coexist with the current "mutable" processor design.

The performance disadvantages caused by using a pointer (which at the time of writing causes an additional heap allocation) may be mitigated by future versions of the Go compiler, thanks to improved escape analysis and profile-guided optimization (PGO)⁶.

On the other hand, Processor's Enabled is fine to accept a Record value as the processors should not mutate the passed parameters.

---

1. A Guide to the Go Garbage Collector ↩︎

2. OpenTelemetry Logging ↩︎

3. Conversation on representing LogRecordProcessor and LogRecordExporter via a single Expoter interface ↩︎

4. Introduce Processor ↩︎

5. Log record mutations do not have to be visible in next registered processors ↩︎

6. Profile-guided optimization ↩︎