Enrich with OpenTelemetry API

Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as manual instrumentation.

Use Cases

Examples of custom spans you might want to add to your code include:

Spans for the execution of some potentially heavy or slow computation in you service.
Tracing for internal or third party libraries for which you don’t have automatic or integrated instrumentation.
Spans to describe some logical operations in your business logic that are meaningful in your domain.

Required Dependencies

Install the API from PyPI using pip:

pip install opentelemetry-api

Creating Spans

To create a span, use the tracer object from the opentelemetry.trace module. The tracer object is a factory for creating spans.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def my_function():
   with tracer.start_as_current_span("my_function") as span:
      print("Hello world!")

Important Notes:

Assign meaningful names to spans: Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span’s purpose and context.
Avoid dynamic, high cardinality data in span names: Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

Recording Span Attributes

Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting Examples:

User ID, organization ID, Account ID or other identifiers.
Inputs - the relevant parameters or configuration that influenced the operation.
Outputs - the result of the operation.
Entities - the entities or resources that the operation interacted with.

Attribute names are lowercased strings in the form my_application.my_attribute, example: my_service.user_id. Read more here To record attributes, use the set_attribute method on the span object.

def my_function(arg: str):
   with tracer.start_as_current_span("my_function") as span:
      span.set_attribute("argument_name", arg)

Important Notes:

Be cautious when recording data: Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
Attribute cost considerations: Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
Use static names for attributes: Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute_name) to provide clear context for downstream consumers.
Adhere to OpenTelemetry semantic conventions: Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.

Recording Errors

To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts. If an operation you are tracing fails, you can mark the span’s status as an error and record the error details within the span. Here’s how you can do it:

An exception has been raised to demonstrate an error that occurred in your code.

def my_function():
   with tracer.start_as_current_span("my_function") as span:
      try:
         print("Hello world!")
         raise Exception("Some Exception")
      except Exception as e:
         span.record_exception(e)
         span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))

Advanced Capabilities

The Odigos Python agent extends the standard OpenTelemetry SDK with capabilities that apply to all spans — including custom spans you create with the API above. These features are configured remotely through OpAMP and take effect without restarting your application.

Head Sampling

The Python agent supports head sampling at the root span. Odigos pushes Noisy Operation rules to the agent, which evaluates them through a custom OdigosSampler wired into the TracerProvider. Custom spans participate in the same sampling pipeline as auto-instrumented spans:

Root spans — If your custom span is the entry point of a trace (no active parent), head-sampling rules can match it and decide whether the entire trace is kept or dropped.
Child spans — Follow parent-based sampling: if the parent trace is sampled, custom child spans are recorded; if not, they are dropped (or recorded for metrics only — see Span metrics mode below).
Context propagation — Use tracer.start_as_current_span(...) (as shown above) so custom spans attach to the active trace and inherit the sampling decision.

Head-sampling rules match HTTP server spans (SpanKind.SERVER) and HTTP client spans (SpanKind.CLIENT) using OpenTelemetry semantic conventions. The sampler accepts both stable and legacy attribute names — for example http.request.method and http.method, server.address and http.host. Server rules match on route (with {param} wildcard support), route prefix, and method. Client rules match on server address, templated path, path prefix, and method. When multiple rules match, the lowest keep percentage wins. Even when a trace is dropped, the agent attaches an odigos tracestate entry so downstream services and the Odigos tail sampler can coordinate on the same decision. For rule configuration and evaluation order, see Sampling rules overview.

Dry Run Mode

When cluster-wide dry run (samplingDryRun in OdigosConfiguration) is enabled, the Python agent still exports every span but annotates the sampling decision in tracestate (dry:t if the trace would have been kept, dry:f if it would have been dropped). Use dry run to validate Noisy rules before committing to real drops.

Span Metrics Mode

The global spanMetricsMode setting controls what happens to spans that head sampling marks as not sampled:

Mode	Not-sampled spans
`sampled-spans-only` (default)	Dropped entirely — no trace export, no span metrics
`all-spans`	Recorded with `RECORD_ONLY` — excluded from trace export but forwarded for span metrics computation

This applies to custom spans the same way it applies to auto-instrumented spans. If you rely on span metrics for RED dashboards, set spanMetricsMode: all-spans in your Odigos configuration so unsampled traffic still contributes to rate and error metrics.

Agent Diagnostics

Debug logging for the Odigos Python agent and its OpenTelemetry components can be toggled at runtime through an InstrumentationRule with agentDiagnostics:

spec:
  agentDiagnostics:
    odigosLogLevel: debug
    openTelemetryComponentsLogLevel: debug

Supported levels are error, warn, info, and debug. Remote config from OpAMP fully overrides the startup environment variables — if a level is not set in the remote config, that logger is silenced. For the window before the first OpAMP message arrives, you can set startup log levels with environment variables:

Variable	Logger	Values
`ODIGOS_LOG_LEVEL`	Odigos agent (`odigos`)	`ALL`, `VERBOSE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `NONE`
`OTEL_LOG_LEVEL`	OpenTelemetry SDK components	Same as above

At debug level, the sampler logs each rule evaluation — useful when troubleshooting why a custom or auto-instrumented span was kept or dropped.

Additional Information

For more use cases, see the OpenTelemetry Python API documentation.

​Use Cases

​Required Dependencies

​Creating Spans

​Recording Span Attributes

​Recording Errors

​Advanced Capabilities

​Head Sampling

​Dry Run Mode

​Span Metrics Mode

​Agent Diagnostics

​Additional Information