This feature is in beta. It may be subject to changes and improvements based on user feedback.

Considerations

Before enabling the Service Name Sampler, please consider the following:
  • Only supports traces.
  • Sampling is trace-wide: either all spans in a trace are kept, or all are dropped.
  • The sampler introduces a delay of up to 30 seconds before traces are exported.
  • Traces with durations longer than 30 seconds might not be sampled correctly.
  • Using the json_condition will cause a higher resource utilization!

Overview

The Span Attribute Sampler is an action designed to sample traces from a specified service based on conditions applied to span attributes. It supports several types of attribute evaluations:
  • String Conditions: Check for existence, equality, inequality, substring containment, or regex matching.
  • Number Conditions: Compare numeric attribute values using equality, inequality, or threshold comparisons.
  • Boolean Conditions: Evaluate the presence of a boolean attribute or check for a specific boolean value.
  • JSON Conditions: Validate JSON structure or evaluate nested JSON content using JSONPath expressions.
If any span from the specified service meets the condition, the trace is sampled. Otherwise, the trace is sampled based on the configured fallback sampling ratio.
Using JSON Conditions will cause spikes in performance and resources utilization.

Use Cases

Targeted Debugging

Sample only traces where a specific error flag is set. For example, sample traces from a checkout service where error = true. See example →

Feature Rollout Monitoring

Retain traces for users or sessions flagged via JSON content. For instance, sample traces for an API service where the JSON payload indicates "user.role": "admin". See example →

Route-Specific Analytics

Track performance or usage of specific endpoints by matching against HTTP target attributes. See example →

Production Environment Sampling

Collect traces only from the production environment. See example →

Latest Version Monitoring

Sample only traces from the latest relesed version, while still capturing a representative subset of older versions. See example →

Configuration Options

The SpanAttributeSampler action is configured using the odigos.io/v1alpha1.Action CRD with the samplers configuration section.
If any span from the specified service satisfies the attribute condition, the trace is sampled. If no match is found, the trace is sampled based on the fallback ratio.

Examples

Below are several YAML examples that demonstrate different use cases.

Match Error in Checkout Service

Sample traces where the error attribute (a boolean) equals true for the checkout-service using the new Action CRD.
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: checkout-errors
  namespace: odigos-system
spec:
  signals:
    - TRACES
  samplers:
    spanAttributeSampler:
      attribute_filters:
        - service_name: checkout-service
          attribute_key: error
          condition:
            boolean_condition:
              operation: equals
              expected_value: "true"
          fallback_sampling_ratio: 10

JSON Nested Attribute Match

For JSON evaluation, the JSON attribute is provided as a string. In this example the rule checks that the JSONPath expression returns a value and that the nested key equals the expected value using the new Action CRD.
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: admin-json-check
  namespace: odigos-system
spec:
  signals:
    - TRACES
  samplers:
    spanAttributeSampler:
      attribute_filters:
        - service_name: api-service
          attribute_key: payload
          condition:
            json_condition:
              operation: key_equals
              json_path: "$.user.role"
              expected_value: admin
          fallback_sampling_ratio: 5

Match HTTP Target Endpoint

Sample traces where the HTTP target endpoint (a string) contains /api/products using the new Action CRD.
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: match-api-products
  namespace: odigos-system
spec:
  signals:
    - TRACES
  samplers:
    spanAttributeSampler:
      attribute_filters:
        - service_name: frontend
          attribute_key: http.target
          condition:
            string_condition:
              operation: contains
              expected_value: "/api/products"
          fallback_sampling_ratio: 5

Sample Prod Traces

For example, sample all traces from the staging environment by matching on the env attribute using the new Action CRD.
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: sample-staging
  namespace: odigos-system
spec:
  signals:
    - TRACES
  samplers:
    spanAttributeSampler:
      attribute_filters:
        - service_name: orders-service
          attribute_key: env
          condition:
            string_condition:
              operation: equals
              expected_value: prod
          fallback_sampling_ratio: 1

Regex Condition (No Match)

The following example shows a regex condition for a version string. If the version does not match the expected pattern, the rule does not match the span. In this case, if the attribute exists but fails the regex test, the trace will not be sampled by this rule (and the fallback ratio applies) using the new Action CRD.
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: version-regex-test
  namespace: odigos-system
spec:
  signals:
    - TRACES
  samplers:
    spanAttributeSampler:
      attribute_filters:
        - service_name: test-service
          attribute_key: version
          condition:
            string_condition:
              operation: regex
              expected_value: "^v1\.2\.\d+$"
          fallback_sampling_ratio: 12

Final Notes

  • Multiple Filters: You can define multiple attribute filters across different services and attribute types. These filters are OR-combined; if any filter matches, the trace is sampled.
  • Fallback Sampling: When no span satisfies the condition, the trace is sampled based on the fallback_sampling_ratio. This allows you to collect a percentage of non-matching traces for broad observability without overwhelming storage costs.
  • JSON Evaluation: For JSON conditions, the sampler always uses the provided json_path to navigate the JSON structure. Ensure that the attribute value is a JSON-encoded string.
By following these guidelines and examples, you can configure the Span Attribute Sampler for production environments with clarity and precision.