Datastreams
Datastreams let you split your observability data into multiple, independent pipelines - each routed to its own destination and optionally enhanced with custom actions like sampling, enrichment, or masking.
This feature is in beta. It may be subject to changes and improvements based on user feedback.
Introduction
Odigos supports Datastreams, a powerful feature that allows you to split telemetry sources into multiple logical streams. Each stream can be independently routed to different destinations and have specific actions applied—such as sampling, attribute enrichment, masking rules, and more.
Motivation
There are many use cases for adopting Datastreams, including:
-
Cost optimization - Route lower-priority services (like internal or backoffice apps) to more cost-effective destinations, such as Parquet files stored in S3.
-
Targeted processing - Apply custom processing (e.g., sampling, enrichment) to only a subset of services.
-
Gradual migration - Test a new destination on specific services before committing to a full migration.
-
Organizational segmentation - Split data based on development teams, product areas, or environments.
Creating Datastreams
After installing Odigos CLI, run the following command to start the Odigos UI:
In a new Odigos install
You’ll be taken to the Setup page. This is where your journey with Odigos begins, and you’ll be asked to provide a few initial details:
- Provide a name for the Datastream
- Select the telemetry Sources to include
- Choose the destination(s) to send the data to
- Click Done
In an existing Odigos install
You can create a new Datastream from the Overview page. Click the + NEW button at the top-left (under the current Datastream views) and follow the same steps as in the setup phase.
After installing Odigos CLI, run the following command to start the Odigos UI:
In a new Odigos install
You’ll be taken to the Setup page. This is where your journey with Odigos begins, and you’ll be asked to provide a few initial details:
- Provide a name for the Datastream
- Select the telemetry Sources to include
- Choose the destination(s) to send the data to
- Click Done
In an existing Odigos install
You can create a new Datastream from the Overview page. Click the + NEW button at the top-left (under the current Datastream views) and follow the same steps as in the setup phase.
This example demonstrates how to configure example-source to send telemetry to jaeger-example through the example Datastream.
- Create or update a Source custom resource with the Datastream label, as described in the linked documentation.
For example:
- Create or Update a destination custom resource spec with the Datastream source selector.
For example:
Now let’s take Jaeger as complete example here:
💡 Tip: Even if you configure datastreams via manifests, you can still view and manage them through the Odigos UI for better visibility.
Caveats and Considerations
-
Trace integrity - Distributed traces often span multiple services that participate in the same operation. If you assign these services to different datastreams, the trace will be broken across destinations. Each destination will only receive part of the trace, making it incomplete and harder to troubleshoot. To preserve trace integrity, keep all services involved in a single transaction within the same datastream.
-
Action support - At present, actions defined in Odigos are applied globally to all streams. Stream-specific actions (e.g., custom sampling rules per datastream) are not yet supported but will be in a future release.