manual instrumentation
.
Use Cases
Examples of custom spans you might want to add to your code include:- Spans for the execution of some potentially heavy or slow computation in you service.
- Tracing for internal or third party libraries for which you don’t have automatic or integrated instrumentation.
- Spans to describe some logical operations in your business logic that are meaningful in your domain.
Required Dependencies
Add the following dependencies to your project by running:Spans
Spans are the OpenTelemetry representation of operations in the execution of your code. You should record a span if you think the operation is meaningful and that observing it will help you understand the behavior of your system and operate it. A span includes a name, start and end time, and key-value attributes for detailed information about the execution.Tracer
In OpenTelemetry, spans are created using theTracer
object which you can acquire from the OpenTelemetry API.
Each tracer is associated with a name and optional version.
These name and version are recorded as “instrumentation scope” into all spans created by this tracer.
Humans viewing the trace and data processors downstream can use this information to understand the origin of each span.
It is best practice to use the go module and package name for the tracer name.
Creating Spans
- Always End a span: Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function.
- Propagate and use a valid context object: When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace.
- Pass the context object downstream: When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow.
- Assign meaningful names to spans: Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span’s purpose and context.
- Avoid dynamic, high cardinality data in span names: Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.
Recording Span Attributes
Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting Examples:- User ID, organization ID, Account ID or other identifiers.
- Inputs - the relevant parameters or configuration that influenced the operation.
- Outputs - the result of the operation.
- Entities - the entities or resources that the operation interacted with.
my_application.my_attribute
, example: my_service.user_id
.
Read more here
To record attributes, use the SetAttributes
method on the span object.
- Be cautious when recording data: Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
- Attribute cost considerations: Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
- Use static names for attributes: Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute_name) to provide clear context for downstream consumers.
- Adhere to OpenTelemetry semantic conventions: Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.