Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as manual instrumentation.

​
Use Cases

Examples of custom spans you might want to add to your code include:

  • Spans for the execution of some potentially heavy or slow computation in you service.
  • Tracing for internal or third party libraries for which you don’t have automatic or integrated instrumentation.
  • Spans to describe some logical operations in your business logic that are meaningful in your domain.

​
Required Dependencies

Add the following dependencies to your project by running:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace

​
Spans

Spans are the OpenTelemetry representation of operations in the execution of your code. You should record a span if you think the operation is meaningful and that observing it will help you understand the behavior of your system and operate it.

A span includes a name, start and end time, and key-value attributes for detailed information about the execution.

​
Tracer

In OpenTelemetry, spans are created using the Tracer object which you can acquire from the OpenTelemetry API.

Each tracer is associated with a name and optional version. These name and version are recorded as “instrumentation scope” into all spans created by this tracer. Humans viewing the trace and data processors downstream can use this information to understand the origin of each span.

It is best practice to use the go module and package name for the tracer name.

package main

import (
   "context"
   "go.opentelemetry.io/otel"
)

var tracer = otel.Tracer("github.com/<yourusername>/<reponame>/<packagename>")

​
Creating Spans

func my_function(ctx context.Context) {
   // Span start time is set here
   ctx, span := tracer.Start(ctx, "my_function")
   // Span end time is set here
   defer span.End()
   // ...Do some work, use ctx if needed when calling next functions
}

Important Notes:

  1. Always End a span: Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function.
  2. Propagate and use a valid context object: When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace.
  3. Pass the context object downstream: When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow.
  4. Assign meaningful names to spans: Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span’s purpose and context.
  5. Avoid dynamic, high cardinality data in span names: Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

​
Recording Span Attributes

Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting

Examples:

  • User ID, organization ID, Account ID or other identifiers.
  • Inputs - the relevant parameters or configuration that influenced the operation.
  • Outputs - the result of the operation.
  • Entities - the entities or resources that the operation interacted with.

Attribute names are lowercased strings in the form my_application.my_attribute, example: my_service.user_id. Read more here

To record attributes, use the SetAttributes method on the span object.

import (
   "context"
   "go.opentelemetry.io/otel/attribute"
)

func my_function(ctx context.Context, userId string) {
   _, span := tracer.Start(ctx, "my_function")
   defer span.End()
   span.SetAttributes(attribute.String("my_service.user_id", userId))
   // ...Do some work
}

Important Notes:

  1. Be cautious when recording data: Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
  2. Attribute cost considerations: Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
  3. Use static names for attributes: Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute_name) to provide clear context for downstream consumers.
  4. Adhere to OpenTelemetry semantic conventions: Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.

​
Recording Errors

To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts.

If an operation you are tracing fails, you can mark the span’s status as an error and record the error details within the span. Here’s how you can do it:

import (
   "context"
   "go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

func my_function(ctx context.Context, userId string) {
   _, span := tracer.Start(ctx, "my_function")
   defer span.End()
   err := doSomeWork() // ...Do some work
   if err != nil {
      span.SetStatus(codes.Error, "Operation failed")
      span.RecordError(err)
   }
}

​
Additional Information

For more use cases, see the OpenTelemetry Go API documentation.