2021-04-01 18:36:51 +02:00
|
|
|
---
|
2021-12-22 02:11:19 +02:00
|
|
|
title: Manual Instrumentation
|
|
|
|
linkTitle: Manual
|
2023-02-24 15:35:24 +02:00
|
|
|
aliases:
|
|
|
|
- /docs/instrumentation/go/instrumentation
|
|
|
|
- /docs/instrumentation/go/manual_instrumentation
|
|
|
|
weight: 3
|
2021-04-01 18:36:51 +02:00
|
|
|
---
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
Instrumentation is the process of adding observability code to your application.
|
|
|
|
There are two general types of instrumentation - automatic, and manual - and you
|
|
|
|
should be familiar with both in order to effectively instrument your software.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2021-12-23 19:18:08 +02:00
|
|
|
## Getting a Tracer
|
|
|
|
|
|
|
|
To create spans, you'll need to acquire or initialize a tracer first.
|
|
|
|
|
2022-06-01 16:59:27 +02:00
|
|
|
### Initializing a new tracer
|
2021-12-23 19:18:08 +02:00
|
|
|
|
|
|
|
Ensure you have the right packages installed:
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
```sh
|
2021-12-23 19:18:08 +02:00
|
|
|
go get go.opentelemetry.io/otel \
|
|
|
|
go.opentelemetry.io/otel/trace \
|
|
|
|
go.opentelemetry.io/otel/sdk \
|
|
|
|
```
|
|
|
|
|
|
|
|
Then initialize an exporter, resources, tracer provider, and finally a tracer.
|
|
|
|
|
|
|
|
```go
|
|
|
|
package app
|
|
|
|
|
|
|
|
import (
|
|
|
|
"context"
|
|
|
|
"fmt"
|
|
|
|
"log"
|
|
|
|
|
|
|
|
"go.opentelemetry.io/otel"
|
|
|
|
"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
|
|
|
|
"go.opentelemetry.io/otel/sdk/resource"
|
|
|
|
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
2023-01-24 18:10:41 +02:00
|
|
|
semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
|
2021-12-23 19:18:08 +02:00
|
|
|
"go.opentelemetry.io/otel/trace"
|
|
|
|
)
|
|
|
|
|
|
|
|
var tracer trace.Tracer
|
|
|
|
|
|
|
|
func newExporter(ctx context.Context) /* (someExporter.Exporter, error) */ {
|
|
|
|
// Your preferred exporter: console, jaeger, zipkin, OTLP, etc.
|
|
|
|
}
|
|
|
|
|
|
|
|
func newTraceProvider(exp sdktrace.SpanExporter) *sdktrace.TracerProvider {
|
2022-04-25 17:12:24 +02:00
|
|
|
// Ensure default SDK resources and the required service name are set.
|
|
|
|
r, err := resource.Merge(
|
|
|
|
resource.Default(),
|
|
|
|
resource.NewWithAttributes(
|
|
|
|
semconv.SchemaURL,
|
2023-02-07 23:42:47 +02:00
|
|
|
semconv.ServiceName("ExampleService"),
|
2023-01-11 01:22:56 +02:00
|
|
|
),
|
2021-12-23 19:18:08 +02:00
|
|
|
)
|
2023-02-24 15:35:24 +02:00
|
|
|
|
2022-04-25 17:12:24 +02:00
|
|
|
if err != nil {
|
|
|
|
panic(err)
|
|
|
|
}
|
2021-12-23 19:18:08 +02:00
|
|
|
|
|
|
|
return sdktrace.NewTracerProvider(
|
|
|
|
sdktrace.WithBatcher(exp),
|
2022-04-25 17:12:24 +02:00
|
|
|
sdktrace.WithResource(r),
|
2021-12-23 19:18:08 +02:00
|
|
|
)
|
|
|
|
}
|
|
|
|
|
|
|
|
func main() {
|
|
|
|
ctx := context.Background()
|
|
|
|
|
|
|
|
exp, err := newExporter(ctx)
|
|
|
|
if err != nil {
|
|
|
|
log.Fatalf("failed to initialize exporter: %v", err)
|
|
|
|
}
|
|
|
|
|
|
|
|
// Create a new tracer provider with a batch span processor and the given exporter.
|
|
|
|
tp := newTraceProvider(exp)
|
|
|
|
|
|
|
|
// Handle shutdown properly so nothing leaks.
|
|
|
|
defer func() { _ = tp.Shutdown(ctx) }()
|
|
|
|
|
|
|
|
otel.SetTracerProvider(tp)
|
|
|
|
|
|
|
|
// Finally, set the tracer that can be used for this package.
|
|
|
|
tracer = tp.Tracer("ExampleService")
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
You can now access `tracer` to manually instrument your code.
|
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
## Creating Spans
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
Spans are created by tracers. If you don't have one initialized, you'll need to
|
|
|
|
do that.
|
2021-12-21 00:00:14 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
To create a span with a tracer, you'll also need a handle on a `context.Context`
|
|
|
|
instance. These will typically come from things like a request object and may
|
|
|
|
already contain a parent span from an [instrumentation library][].
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
```go
|
2021-12-21 00:00:14 +02:00
|
|
|
func httpHandler(w http.ResponseWriter, r *http.Request) {
|
|
|
|
ctx, span := tracer.Start(r.Context(), "hello-span")
|
|
|
|
defer span.End()
|
|
|
|
|
|
|
|
// do some work to track with hello-span
|
|
|
|
}
|
2021-04-01 18:36:51 +02:00
|
|
|
```
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
In Go, the `context` package is used to store the active span. When you start a
|
|
|
|
span, you'll get a handle on not only the span that's created, but the modified
|
|
|
|
context that contains it.
|
2021-12-21 00:00:14 +02:00
|
|
|
|
|
|
|
Once a span has completed, it is immutable and can no longer be modified.
|
|
|
|
|
|
|
|
### Get the current span
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
To get the current span, you'll need to pull it out of a `context.Context` you
|
|
|
|
have a handle on:
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
```go
|
2021-12-21 00:00:14 +02:00
|
|
|
// This context needs contain the active span you plan to extract.
|
|
|
|
ctx := context.TODO()
|
|
|
|
span := trace.SpanFromContext(ctx)
|
|
|
|
|
|
|
|
// Do something with the current span, optionally calling `span.End()` if you want it to end
|
|
|
|
```
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
This can be helpful if you'd like to add information to the current span at a
|
|
|
|
point in time.
|
2021-12-21 00:00:14 +02:00
|
|
|
|
|
|
|
### Create nested spans
|
|
|
|
|
|
|
|
You can create a nested span to track work in a nested operation.
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
If the current `context.Context` you have a handle on already contains a span
|
|
|
|
inside of it, creating a new span makes it a nested span. For example:
|
2021-12-21 00:00:14 +02:00
|
|
|
|
|
|
|
```go
|
|
|
|
func parentFunction(ctx context.Context) {
|
|
|
|
ctx, parentSpan := tracer.Start(ctx, "parent")
|
2021-04-28 17:19:15 +02:00
|
|
|
defer parentSpan.End()
|
2021-12-21 00:00:14 +02:00
|
|
|
|
|
|
|
// call the child function and start a nested span in there
|
2021-04-28 17:19:15 +02:00
|
|
|
childFunction(ctx)
|
2021-12-21 00:00:14 +02:00
|
|
|
|
|
|
|
// do more work - when this function ends, parentSpan will complete.
|
2021-04-01 18:36:51 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
func childFunction(ctx context.Context) {
|
2021-12-21 00:00:14 +02:00
|
|
|
// Create a span to track `childFunction()` - this is a nested span whose parent is `parentSpan`
|
|
|
|
ctx, childSpan := tracer.Start(ctx, "child")
|
2021-04-28 17:19:15 +02:00
|
|
|
defer childSpan.End()
|
2021-12-21 00:00:14 +02:00
|
|
|
|
2021-04-28 17:19:15 +02:00
|
|
|
// do work here, when this function returns, childSpan will complete.
|
2021-04-01 18:36:51 +02:00
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
Once a span has completed, it is immutable and can no longer be modified.
|
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
### Span Attributes
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
Attributes are keys and values that are applied as metadata to your spans and
|
|
|
|
are useful for aggregating, filtering, and grouping traces. Attributes can be
|
|
|
|
added at span creation, or at any other time during the lifecycle of a span
|
|
|
|
before it has completed.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
```go
|
|
|
|
// setting attributes at creation...
|
2021-04-24 22:57:23 +02:00
|
|
|
ctx, span = tracer.Start(ctx, "attributesAtCreation", trace.WithAttributes(attribute.String("hello", "world")))
|
2021-04-01 18:36:51 +02:00
|
|
|
// ... and after creation
|
2021-04-24 22:57:23 +02:00
|
|
|
span.SetAttributes(attribute.Bool("isTrue", true), attribute.String("stringAttr", "hi!"))
|
2021-04-01 18:36:51 +02:00
|
|
|
```
|
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
Attribute keys can be precomputed, as well:
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
```go
|
2021-04-24 22:57:23 +02:00
|
|
|
var myKey = attribute.Key("myCoolAttribute")
|
2021-04-01 18:36:51 +02:00
|
|
|
span.SetAttributes(myKey.String("a value"))
|
|
|
|
```
|
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
#### Semantic Attributes
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
Semantic Attributes are attributes that are defined by the [OpenTelemetry
|
|
|
|
Specification][] in order to provide a shared set of attribute keys across
|
|
|
|
multiple languages, frameworks, and runtimes for common concepts like HTTP
|
|
|
|
methods, status codes, user agents, and more. These attributes are available in
|
|
|
|
the `go.opentelemetry.io/otel/semconv/v1.12.0` package.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2021-11-12 21:41:36 +02:00
|
|
|
For details, see [Trace semantic conventions][].
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
### Events
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
An event is a human-readable message on a span that represents "something
|
|
|
|
happening" during it's lifetime. For example, imagine a function that requires
|
|
|
|
exclusive access to a resource that is under a mutex. An event could be created
|
|
|
|
at two points - once, when we try to gain access to the resource, and another
|
|
|
|
when we acquire the mutex.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
```go
|
|
|
|
span.AddEvent("Acquiring lock")
|
|
|
|
mutex.Lock()
|
|
|
|
span.AddEvent("Got lock, doing work...")
|
|
|
|
// do stuff
|
|
|
|
span.AddEvent("Unlocking")
|
|
|
|
mutex.Unlock()
|
|
|
|
```
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
A useful characteristic of events is that their timestamps are displayed as
|
|
|
|
offsets from the beginning of the span, allowing you to easily see how much time
|
|
|
|
elapsed between them.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
Events can also have attributes of their own -
|
|
|
|
|
|
|
|
```go
|
2021-04-24 22:57:23 +02:00
|
|
|
span.AddEvent("Cancelled wait due to external signal", trace.WithAttributes(attribute.Int("pid", 4328), attribute.String("signal", "SIGHUP")))
|
2021-04-01 18:36:51 +02:00
|
|
|
```
|
|
|
|
|
2022-05-10 20:17:38 +02:00
|
|
|
### Set span status
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
A status can be set on a span, typically used to specify that there was an error
|
|
|
|
in the operation a span is tracking - .`Error`.
|
2022-05-10 20:17:38 +02:00
|
|
|
|
|
|
|
```go
|
|
|
|
import (
|
|
|
|
// ...
|
|
|
|
"go.opentelemetry.io/otel/codes"
|
|
|
|
// ...
|
|
|
|
)
|
|
|
|
|
|
|
|
// ...
|
|
|
|
|
|
|
|
result, err := operationThatCouldFail()
|
|
|
|
if err != nil {
|
|
|
|
span.SetStatus(codes.Error, "operationThatCouldFail failed")
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
By default, the status for all spans is `Unset`. In rare cases, you may also
|
|
|
|
wish to set the status to `Ok`. This should generally not be necessary, though.
|
2022-05-10 20:17:38 +02:00
|
|
|
|
|
|
|
### Record errors
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
If you have an operation that failed and you wish to capture the error it
|
|
|
|
produced, you can record that error.
|
2022-05-10 20:17:38 +02:00
|
|
|
|
|
|
|
```go
|
|
|
|
import (
|
|
|
|
// ...
|
|
|
|
"go.opentelemetry.io/otel/codes"
|
|
|
|
// ...
|
|
|
|
)
|
|
|
|
|
|
|
|
// ...
|
|
|
|
|
|
|
|
result, err := operationThatCouldFail()
|
|
|
|
if err != nil {
|
|
|
|
span.SetStatus(codes.Error, "operationThatCouldFail failed")
|
|
|
|
span.RecordError(err)
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
It is highly recommended that you also set a span's status to `Error` when using
|
|
|
|
`RecordError`, unless you do not wish to consider the span tracking a failed
|
|
|
|
operation as an error span. The `RecordError` function does **not**
|
|
|
|
automatically set a span status when called.
|
2022-05-10 20:17:38 +02:00
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
## Creating Metrics
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
The metrics API is currently unstable, documentation TBA.
|
|
|
|
|
2021-12-21 00:00:14 +02:00
|
|
|
## Propagators and Context
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
Traces can extend beyond a single process. This requires _context propagation_,
|
|
|
|
a mechanism where identifiers for a trace are sent to remote processes.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
In order to propagate trace context over the wire, a propagator must be
|
|
|
|
registered with the OpenTelemetry API.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
|
|
|
```go
|
|
|
|
import (
|
|
|
|
"go.opentelemetry.io/otel"
|
|
|
|
"go.opentelemetry.io/otel/propagation"
|
|
|
|
)
|
|
|
|
...
|
|
|
|
otel.SetTextMapPropagator(propagation.TraceContext{})
|
|
|
|
```
|
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
> OpenTelemetry also supports the B3 header format, for compatibility with
|
|
|
|
> existing tracing systems (`go.opentelemetry.io/contrib/propagators/b3`) that
|
|
|
|
> do not support the W3C TraceContext standard.
|
2021-04-01 18:36:51 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
After configuring context propagation, you'll most likely want to use automatic
|
|
|
|
instrumentation to handle the behind-the-scenes work of actually managing
|
|
|
|
serializing the context.
|
2021-12-21 00:00:14 +02:00
|
|
|
|
2023-02-24 15:35:24 +02:00
|
|
|
[opentelemetry specification]: /docs/reference/specification/
|
|
|
|
[trace semantic conventions]:
|
|
|
|
/docs/reference/specification/trace/semantic_conventions/
|
|
|
|
[instrumentation library]: ../libraries/
|