My Destination

If the trace matches a Noisy rule, it is dropped or sampled down as configured. The trace exits the pipeline at this stage; Highly Relevant and Cost Reduction are not evaluated for that trace.

↓

▶

Only traces that did not match any Noisy rule reach this stage. Rules here classify important traffic (for example errors and duration); important matches can still be sampled per each rule's Keep Percentage. Traces continue to Cost Reduction after this stage.

↓

▶

For traces that match rules in this category after the earlier stages, this stage applies caps to control volume and cost.

The pipeline runs the categories in this order. At each stage, every rule in that category is considered; how multiple matches resolve is covered in [Rule evaluation](#rule-evaluation). ## Rule evaluation ### Evaluation inside a category Within a single category, rule order does not matter. When multiple rules match the same trace, the engine combines them so all of them are satisfied at once. For Highly Relevant, percentages are floors — the highest keep percentage wins, so any retention guarantee is honored. For Noisy Operation and Cost Reduction, percentages are caps — the lowest keep percentage wins, so the strictest cap is honored. This makes it safe to add rules without fear of one silently overriding another. ### Evaluation between Highly Relevant and Cost Reduction categories When a trace matches rules in **both** the **Highly Relevant** and **Cost Reduction** categories, the **Highly Relevant** keep percentage **always wins** — regardless of whether it is higher or lower than the **Cost Reduction** cap. **Trace:** `POST /api/orders` that returned a `500` error. **Matching rules:** * **Highly Relevant** — keep **100%** of error traces. * **Cost Reduction** — keep at most **10%** of `/api/orders` traffic. **Outcome:** the trace is **kept at 100%**. Highly Relevant overrides the Cost Reduction cap. **Trace:** `GET /api/products` that took **750 ms**. **Matching rules:** * **Highly Relevant** — keep **25%** of traces longer than **500 ms**. * **Cost Reduction** — keep at most **50%** of `/api/products` traffic. **Outcome:** the trace is **kept at 25%**. Highly Relevant still wins, even though its keep rate is lower than the Cost Reduction cap. ## Why full traces still matter For traces that are evaluated on the **tail** path, the evaluator needs the assembled trace (within the aggregation window) so it can apply **Highly Relevant** conditions that depend on any span. That is why tail sampling and aggregation are part of the pipeline for those policies, even when the category order is Noisy → Highly Relevant → Cost Reduction. # Service Graph Source: https://docs.odigos.io/enterprise/pipeline/service-graph Visualize how your services connect to each other and to uninstrumented dependencies like Redis or Kafka. ## Overview The Service Graph is a live topology map that shows which services call each other and how those calls are performing. It is built entirely from your distributed traces — no additional instrumentation or agents are required. Odigos generates the service graph automatically when traces are enabled and at least one tracing destination is configured. ## How It Works Odigos uses the OpenTelemetry [Service Graph Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector) inside the Cluster Gateway Collector. The pipeline looks like this: ``` Instrumented services │ (OTLP traces) ▼ Node Collector (DaemonSet) │ (OTLP traces) ▼ Cluster Gateway Collector ├──► servicegraph connector ──► Odigos UI (topology map) ``` 1. The **servicegraph connector** receives every trace that passes through the gateway. It matches CLIENT spans with their corresponding SERVER spans to detect service-to-service calls. 2. For each matched pair it emits a metric labelled with the `client` and `server` service names plus any configured extra dimensions, which the Odigos UI uses to render the interactive topology map. ### Virtual Nodes (Uninstrumented Dependencies) When a service calls an uninstrumented dependency (Redis, Kafka, a third-party API, …) there is no SERVER span. The connector handles this with **virtual nodes**: it reads certain attributes from the CLIENT span to infer the name of the remote endpoint and creates a synthetic node in the graph. The attributes checked — in priority order — are configured via `serviceGraphVirtualNodePeerAttributes`. The connector picks the **first attribute that is present** on the CLIENT span. Only **span-level attributes** are read when resolving virtual node identities (e.g. `server.address`, `db.system`, `peer.service`). ## Enabling and Disabling The service graph is **enabled by default** when traces are active. Disabling it removes the `servicegraph` connector from the gateway pipeline and stops emitting the related metrics. Disabling the service graph also removes the load balancing behavior between the node collectors and the gateway, because the gateway no longer needs to aggregate spans from the same trace. ```yaml theme={null} # values.yaml collectorGateway: serviceGraphDisabled: true # set to true to disable ``` ## Configuration Options All options live under `collectorGateway` in your `values.yaml` (Helm) or by manually editing the `odigos-configuration` ConfigMap in your cluster. ### Extra Dimensions By default the graph distinguishes services by `service.name` only. `serviceGraphExtraDimensions` adds more labels to the edge metrics, creating a separate node for each unique combination of values. **When to use it:** multiple deployments share the same `service.name` (e.g. the same service deployed in `staging` and `production` namespaces). ```yaml theme={null} collectorGateway: serviceGraphExtraDimensions: - k8s.namespace.name # separate nodes per namespace ``` Both span-level and resource-level attributes are supported for extra dimensions — the connector reads from the full span resource context. Avoid high-cardinality attributes such as `k8s.pod.name`; each unique value creates a separate time series in Prometheus. ### Virtual Node Peer Attributes `serviceGraphVirtualNodePeerAttributes` controls which span attribute is used to name uninstrumented dependencies. The connector evaluates the list in order and uses the first attribute that has a non-empty value. **When to use it:** multiple instances of the same technology (e.g. a Redis Cluster with several shards, or two separate Redis caches) are collapsing into a single node because they all resolve to the same value (e.g. `db.system=redis`). ```yaml theme={null} collectorGateway: serviceGraphVirtualNodePeerAttributes: - peer.service # explicit logical name — highest priority - server.address # hostname of the specific instance (e.g. redis-cache.default.svc.cluster.local) - net.peer.name # older convention, wide SDK support - db.name # database index (rarely useful for Redis — most apps use index 0) - db.system # technology name (e.g. "redis") — last resort, not instance-specific ``` ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Add Sources Source: https://docs.odigos.io/enterprise/pipeline/sources/create Add a Source to enable telemetry collection for specific Kubernetes workloads, or a whole namespace. ## With Odigos UI The easiest way to add a source to Odigos is to use the Odigos UI.
After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI: ```shell theme={null} odigos ui ``` And then, in the setup page or the sources page, select the workloads / namespaces for which you want telemetry collection to be enabled. To include all the workloads in a namespace as sources, and automatically add future-new workloads, enable the `Future Apps` toggle in the UI. Otherwise, you can enable the `Select All` toggle to include all the workloads in the namespace as sources, excluding future-new workloads. ## With Odigos CLI See the [CLI documentation on Sources](../../cli/odigos_sources) for detailed command documentation for manipulating Sources. ## Kubernetes Manifests If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can add a source to Odigos by creating a Source object. Only one Source object may exist for a given workload, except in namespace instrumentation. ### Workload Source The following Source object enables instrumentation for a Deployment named `frontend` in the `default` namespace and belong to Datastream `example`. > **Note:** Sources that do not have a Datastream label will be automatically attached to the default Datastream. ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default labels: odigos.io/data-stream-example: "true" spec: workload: name: frontend namespace: default kind: Deployment ``` ### Namespace Source The following Source object enables instrumentation for all the workloads in the `default` namespace: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: namespace-source namespace: default spec: workload: name: default namespace: default kind: Namespace ``` Namespace Sources must have the following fields:
* `spec.workload.name == spec.workload.namespace` * `spec.workload.kind: Namespace` ### Disable Workload Source in Namespace You can have a namespace Source and a workload Source that both enable instrumentation for a workload. In this case, if one is deleted, the other will persist instrumentation for the workload. To exclude specific workloads from namespace instrumentation, set `disableInstrumentation: true` on the workload Source. For example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: namespace-source namespace: default spec: workload: name: default namespace: default kind: Namespace --- apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default spec: disableInstrumentation: true workload: name: frontend namespace: default kind: Deployment ``` In this example, all workloads in the `default` namespace will be instrumented except for the `frontend` Deployment. If `example-source` is deleted (or `disableInstrumentation` is set to `false`), the `frontend` Deployment will be instrumented. # Remove Sources Source: https://docs.odigos.io/enterprise/pipeline/sources/delete Remove a Source to disable telemetry collection for specific Kubernetes workloads, or a whole namespace. Sources are persistent and decoupled from the workloads they instrument.
If a workload is deleted without deleting the Source, the Source will resume its effects if the workload is re-created later. ## With Odigos CLI The Odigos CLI supports instrumenting and uninstrumenting workloads with the [`odigos sources enable`](../../cli/odigos_sources_enable) and [`odigos sources disable`](../../cli/odigos_sources_disable) commands. Each of these commands accepts a source type, name, and namespace. For example, to remove a source from odigos, run: ```shell theme={null} odigos sources disable deployment -n ``` This command will rollout existing pods to restart without any odigos additions. It can be undone (re-instrumented) by running: " ```shell theme={null} odigos sources enable deployment -n ``` ## With Odigos UI The easiest way to remove a source from Odigos is to use the Odigos UI.
After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI: ```shell theme={null} odigos ui ``` And then, in the sources page, unselect the workloads / namespace for which you want telemetry collection to be disabled. ## Kubernetes Manifests If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can remove a source from Odigos by deleting the Source object, for example: Run: ```shell theme={null} kubectl get sources -n default ``` Output: ```shell theme={null} NAME WORKLOAD KIND NAMESPACE source-7sxsb frontend Deployment default source-b42rn inventory Deployment default source-b7zn9 coupon Deployment default source-v6sxs membership Deployment default source-v9jjw pricing Deployment default ``` Run: ```shell theme={null} kubectl delete source source-7sxsb -n default ``` Output: ```shell theme={null} source.odigos.io "source-7sxsb" deleted ``` # Introduction Source: https://docs.odigos.io/enterprise/pipeline/sources/introduction Sources are a Custom Resource that define which Kubernetes workloads should be auto-instrumented by Odigos. ## How Sources work Source custom resource objects are created as a reference to a workload or namespace ("entities"). Odigos reacts to changes in these custom resource objects to configure instrumentation for the referenced entity. On this page, "Sources" refers to the custom resource object, while "workload", "namespace", or "entity" refers to the application being instrumented. See the [API Reference](../../api-reference/odigos.io.v1alpha1#source) for the full structure of `Source` objects and their fields. ## Supported Entities Odigos supports the following Kubernetes entities: * Kubernetes **Deployment** * Kubernetes **DaemonSet** * Kubernetes **StatefulSet** * Kubernetes **Namespace** ## Types of Sources A Source object can be either: * A **Workload Source**, which controls instrumentation for a single workload (Deployment, DaemonSet, or StatefulSet) * A **Namespace Source**, which controls instrumentation for every workload in a namespace. ### Workload Sources Workload Sources specify a single workload, and the Source represents the instrumentation configuration for that workload. Use Workload Sources to control instrumentation and configuration for individual Deployments, StatefulSets, or DaemonSets with unique requirements. Example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-workload namespace: default spec: workload: kind: DaemonSet name: sample-app namespace: default ``` ### Namespace Sources Namespace Sources represent instrumentation settings for an entire namespace. Namespace Sources are created by setting `spec.workload.kind: Namespace` along with `spec.workload.name` and `spec.workload.namespace` equal to the name of the namespace to control. Use Namespace Sources to control instrumentation for multiple Deployments, StatefulSets, and DaemonSets from a single spot. Example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-namespace namespace: default spec: workload: kind: Namespace name: default namespace: default ``` ## Using multiple Sources Workload Sources are intended to be combined with Namespace Sources for flexible, fine-grained control. For example, you may choose to create a Namespace Source to enable instrumentation for namespace `foo`, but want to exclude deployment `bar`. In this case, you can create a Workload Source for deployment `bar` with `disableInstrumentation: true`: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-namespace namespace: foo spec: workload: kind: Namespace name: foo namespace: foo --- apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: exclude-bar namespace: foo spec: disableInstrumentation: true workload: kind: Deployment name: bar namespace: foo ``` The opposite is also possible: you could disable instrumentation for namespace `foo` with a Namespace Source but choose to explicitly enable deployment `bar` via a Workload Source that sets `disableInstrumentation: false` (the default). This is because Workload Sources take priority over Namespace Sources, and any settings in a Workload Source will override those in a parent Namespace Source. ## Source Workflows Sources are intended to control and configure instrumentation as objects that are completely decoupled from their target workloads. When working with Sources, they should be treated as such. The only requirements for a Source to identify a workload are: * The Source must set `spec.workload.name`, `spec.workload.namespace`, and `spec.workload.kind` to match the workload (or namespace). * The Source must exist in the same namespace as the target workload (or namespace). Besides these requirements, you are encouraged to manage your Sources as their own objects. For example, when creating Sources from YAML manifests, consider storing them together in their own file such as `sources.yaml`, separate from your workload manifests. This separation of concerns allows you to declaratively control your application's observability without tying it to your application manifests. You should also define RBAC policies that restrict access to only what is needed to work with Source objects. In other words, your observability team may not require the ability to view deployments in a namespace to manage the instrumentation for those deployments. For example, the following Role grants full editor permissions for Sources: ```yaml theme={null} apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: default name: source-editor rules: - apiGroups: ["odigos.io"] resources: ["sources"] verbs: ["*"] ``` ## Source Labels Odigos uses mutating webhooks to automatically copy the values of `spec.workload` to matching labels. It is encouraged to use these labels to manage Sources in a cluster: * `odigos.io/workload-name` * `odigos.io/workload-namespace` * `odigos.io/workload-kind` The value of these labels will be validated to always match the corresponding fields in `spec.workload`. They can be used to group and list Sources, for example: ```shell theme={null} kubectl get sources -l odigos.io/workload-kind=Deployment ``` This will list all `Sources` for `Deployments`. ## Automatically created Sources In some scenarios, Odigos may create Sources for you automatically. * When instrumenting workloads in the UI, Odigos will create a Workload Source for each workload. * When choosing **Include future workloads** in the UI, Odigos will create a Namespace Source for the namespace. * When creating a Source grouping in the UI, Odigos will create a Workload Source for each Workload in the group. In all of these cases, if a Source already exists, Odigos will either patch the existing Source or return an error if it can't. # Troubleshooting Source: https://docs.odigos.io/enterprise/pipeline/sources/troubleshooting If you selected a workload to auto-instrument and you don't see any data, use the checklist below to troubleshoot the issue. It is recommended to use the [`odigos describe`](../../cli/odigos_describe) command to get detailed information in one command. ## Check is Source Created? A `Source` object is required for Odigos to instrument the workload or namespace, such as: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example namespace: foo spec: workload: name: myapp namespace: foo kind: StatefulSet ``` If you expect to see data from a workload but don't, check if the `Source` object exists: ```shell get all in namespace theme={null} kubectl get sources -n ``` ```shell get by deployment name theme={null} kubectl get sources -n -l odigos.io/workload-name= ``` ```shell get by deployment kind theme={null} kubectl get sources -n -l odigos.io/workload-kind= ``` ## Check is Language Detected? After a `Source` object is created, Odigos will attempt to detect the programming language of the containers in the workload pods, and write the detected language to an `InstrumentationConfig` object in the same namespace. The resulting language is used to determine which instrumentation to apply. If a `Source` object exists, verify that the `InstrumentationConfig` also exists: ```shell get all in namespace theme={null} kubectl get instrumentationconfigs -n ``` ```shell get by crd name theme={null} kubectl get instrumentationconfigs -n

-o yaml ```

If an `InstrumentationConfig` object exists, verify that the `status.runtimeDetailsByContainer` array contains the expected languages, for example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: InstrumentationConfig metadata: name: example namespace: foo spec: serviceName: example status: runtimeDetailsByContainer: - containerName: example language: go runtimeVersion: 1.21.2 ``` ## Check is Device Instrumented? Once an `InstrumentationConfig` object is created, and the supported language is detected, Odigos will wait until a [destination](../../backends-overview) is added, and OpenTelemetry Collectors are deployed to the cluster (which will receive the telemetry data). To inject the instrumentation agents into the pods, Odigos will patch the workload template containers with a resource based on the detected language. If an `InstrumentationConfig` object exists, and the languages are detected, verify that the `Deployment` object contains the expected instrumentation devices: ```shell get all in namespace theme={null} kubectl get deployments -n ``` ```shell get by deployment name theme={null} kubectl get deployments -n

-o yaml ```

It should look something like this: ```yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: example namespace: foo spec: template: spec: containers: - resources: limits: instrumentation.odigos.io/go-ebpf-community: "1" ``` The `instrumentation.odigos.io/go-ebpf-community` resource-limit instructs Odigos on how to mount the relevant OpenTelemetry agents and environment variables into the container, and which processes should be instrumented with eBPF. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Configuration Source: https://docs.odigos.io/enterprise/pipeline/sources/update Beyond just enabling or disabling instrumentation, Source objects can be used to configure instrumentation settings as well. All Source settings can be configured with the Odigos CLI (see [CLI documentation on updating Sources](../../cli/odigos_sources_update)) or Kubernetes-native tools like `kubectl`. ## service.name The `service.name` attribute is a fundamental resource attribute in [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/resource/#service).
It is required, and is reported in all telemetry data.
It is used to identify the service that emitted a span, metric or log record. Odigos will automatically populate this attribute based on the name of the workload. If there are multiple containers in the workload, the `service.name` will be set to the container name. If you want to override the default value, you can also set the `service.name` attribute by adding the `otelServiceName` attribute to an exiting Source, or by creating a `Source` with that attribute: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default spec: workload: name: frontend namespace: default kind: Deployment otelServiceName: my-custom-service-name ``` Namespace Sources do not support the `otelServiceName` attribute. If a namespace is instrumented, and a Source for a specific workload is not present, a specific source for that workload will need to be created including the desired `otelServiceName`. ### From Odigos UI The easiest way to update a source from Odigos is to use the Odigos UI.
After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI: ```shell theme={null} odigos ui ``` And then, in the sources page, select the source you want to edit and fill in the value for the `service.name` field. To undo the override, simply erase the value from the `service.name` field. ### Odigos CLI To set the `service.name` using the Odigos CLI, use the `odigos sources update` command ([see documentation](../../cli/odigos_sources_update)). If you know the name of the Source object to update, pass that to `odigos sources update` along with the `--otel-service` flag: ```shell theme={null} odigos sources update --otel-service=my-service-name -n ``` If you don't know the name of the Source object to update, you can provide flags to `odigos sources update` to identify the Source by workload: ```shell theme={null} odigos sources update --workload-kind= \ --workload-name= \ --workload-namespace= \ --otel-service=my-service-name ``` ### Kubernetes Manifest If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can can set the `service.name` attribute by updating the corresponding Source. For example, to set the `service.name` attribute in telemetry for an existing source: To get the relevant Source for your workload you can either use: ```shell theme={null} kubectl get sources -n \ -l odigos.io/workload-kind= \ -l odigos.io/workload-name= \ -l odigos.io/workload-namespace= ``` or list the sources in the relevant namespace: ```shell theme={null} kubectl get sources -n ``` After we have the desired Source we can patch it: ```shell theme={null} kubectl patch source/ -n --type=merge -p '{"spec":{"otelServiceName":""}}' ``` If you don't know the name of a Source object (such as when a Source is automatically created by instrumentation done via the UI), you can combine the use of Source labels with the `kubectl patch` command. For example: ```shell theme={null} kubectl patch source \ $(kubectl get source -l 'odigos.io/workload-name=,odigos.io/workload-kind=' --no-headers -o custom-columns=":metadata.name") \ --type=merge \ -p '{"spec":{"otelServiceName":""}}' ``` # Building a pipeline Source: https://docs.odigos.io/enterprise/quickstart/building-a-pipeline ## Choosing target applications You should now see the following page: Select target applications

Select all the applications in the `default` namespace and click `Next`. Selecting **Apply for future apps** will automatically generate distributed traces for any application that will be deployed on that namespace in the future. ## Adding Jaeger as a destination In the next step, you will be asked to select a destination for your traces. Scroll down to the self hosted destinations and click on **Jaeger**. Select Jaeger

#### Connection details You will now be asked to provide the connection details for your Jaeger instance: * **Destination name**: A name for your destination * **Host**: The hostname of your Jaeger instance. Enter any name you want for the destination, and enter `jaeger.tracing:4317` as the host. Jaeger connection details

**That's it!** You can now click `Done` and finish the wizard. Odigos will now instrument your selected applications and deploy the nessesary OpenTelemetry collectors. The following page will show an overview of your observability pipeline:

# Generating Data Source: https://docs.odigos.io/enterprise/quickstart/generation-data ## Wait for collectors to be ready Before generating data, make sure that the collectors and applications are ready. To check this run the following command: ```bash theme={null} kubectl get pods -n default ``` ```bash theme={null} kubectl get pods -n odigos-system ```

## Access frontend application and generate data In order to access the frontend application, run the following command: ```bash theme={null} kubectl port-forward svc/frontend 8080:8080 ``` Then, open the following URL in your browser: [http://localhost:8080](http://localhost:8080) and purchase some items.

# Installation Source: https://docs.odigos.io/enterprise/quickstart/installation ## Installing Odigos CLI Odigos CLI is the easiest way to install Odigos on your Kubernetes cluster. It is a single binary that can be installed on any Linux, MacOS or Windows machine. If you are on MacOS, you can install Odigos CLI using Homebrew: ```bash theme={null} brew install odigos-io/odigos-cli/odigos ``` 1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux). 2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system. 3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH". * For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt. * For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`. 4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH. 1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows). 2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files. 3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how: * **Option 1**: Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder. * **Option 2**: Add the folder containing the extracted binary to your PATH manually: 1. Right-click the **Start** button and select **System**. 2. Click **Advanced system settings** on the left. 3. In the **System Properties** window, click **Environment Variables** at the bottom. 4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**. 5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file. 6. Click **OK** to close all windows. 4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located. ## Installing Odigos on Kubernetes To install Odigos on your Kubernetes cluster, run the following command: ```bash theme={null} odigos install ``` ## Accessing Odigos UI After Odigos is installed, open the Odigos UI by running the following command: ```bash theme={null} odigos ui ``` By default, Odigos UI is available at [http://localhost:3000](http://localhost:3000). # Introduction Source: https://docs.odigos.io/enterprise/quickstart/introduction In this tutorial we are going to use Odigos for generating distributed traces for a microservices-based application written in Go, Java, Python, .NET and Node.js. We are going to deploy the application in a Kubernetes cluster and use [Jaeger](https://www.jaegertracing.io/) as the backend for storing and visualizing the traces. This guide is designed for: * IT professionals with experience in system administration or DevOps * Software engineers familiar with containerization and Kubernetes concepts * Anyone comfortable with command-line interfaces and basic cloud infrastructure If you're new to these concepts or prefer a more detailed, step-by-step approach, we recommend starting with our [Beginner's Guide to Odigos Installation](../quickstart/installation-guide-for-beginners). ## Preparing the environment ### Creating a Kubernetes cluster This first step is to create a Kubernetes cluster. We recommend using [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/docs/) for trying Odigos out in a local environment. **Mac users**: please avoid using Docker Desktop built-in Kubernetes cluster as it [does not](https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation) support bind propagation. Create a new local Kubernetes cluster, by running the following command: ```bash theme={null} kind create cluster ``` ```bash theme={null} minikube start ``` ### Deploying the target application We are going to install a [demo application](https://github.com/odigos-io/simple-demo) that consists of 6 microservices written in Java, Go, Python, Node.js, .NET and PHP. Architecture of demo application

Deploy the application using the following command: ```bash theme={null} kubectl apply -f https://raw.githubusercontent.com/odigos-io/simple-demo/main/kubernetes/deployment.yaml ``` ### Deploying Jaeger Jaeger is a popular open source distributed tracing backend. We are going to use it for storing and visualizing the traces generated by Odigos. Install Jaeger using the following command: ```bash theme={null} kubectl apply -f https://raw.githubusercontent.com/odigos-io/simple-demo/main/kubernetes/jaeger.yaml ``` ### Waiting for the applications to start Before moving to the next step, make sure that the application pods and Jaeger pods are running, this may take a few minutes. ```bash theme={null} kubectl wait --for=condition=available --timeout=300s deployment --all --all-namespaces ``` Deployments Ready

# Next Steps Source: https://docs.odigos.io/enterprise/quickstart/next-steps You just finished the Odigos quickstart guide! 🎉 ## Not just distributed tracing **Odigos supports metrics and logs too!** You can send metrics and logs from your application to Odigos, and then visualize them in your favorite backend. In order to send metrics and logs, just select a destination that supports them. Odigos supports all the major observability backends, both open-source and commercial. Select the relevant backend for your use case below to connect it to Odigos. # Viewing traces Source: https://docs.odigos.io/enterprise/quickstart/viewing-traces Now that we have some traces, let's view them! Access the Jaeger UI by running the following command: ```bash theme={null} kubectl port-forward -n tracing svc/jaeger 16686:16686 ``` Then, open [http://localhost:16686](http://localhost:16686) in your browser and explore the traces.

# Install Odigos Images from a Custom Docker Registry Source: https://docs.odigos.io/enterprise/setup/docker-registry ## Tagging and Pushing Images ### Step 1: Set Environment Variables First, export the `VERSION`, `CUSTOM_DOCKER_REGISTRY`, `PLATFORM`, and `ODIGOS_TOKEN` environment variables in your shell: ```bash theme={null} export VERSION= export CUSTOM_DOCKER_REGISTRY= export PLATFORM= export ODIGOS_TOKEN= ``` Replace `` with the version of the Odigos images you want to use, `` with the URL of your custom Docker registry, `` with the platform architecture of the environment where you are deploying Odigos (linux/amd64, linux/arm64), and `` with your Odigos Enterprise on-prem token. *** ### Step 2: Authenticate to the Enterprise Registry Enterprise images are hosted on the private registry `registry.odigos.io`. Authenticate with your on-prem token before pulling. The username is always `odigos`; the password is your on-prem token. Choose the method that matches how you will pull images: Log in with Docker so you can `docker pull` images locally or in CI: ```bash theme={null} echo "$ODIGOS_TOKEN" | docker login registry.odigos.io -u odigos --password-stdin ``` Log in with Podman so you can `podman pull` images locally or in CI: ```bash theme={null} echo "$ODIGOS_TOKEN" | podman login registry.odigos.io -u odigos --password-stdin ``` Manually generate a Docker config JSON pull secret for local Docker or Podman. Use this when `login` is not available, or you need an auth file for CI. ```bash theme={null} export ODIGOS_DOCKER_AUTH=$(echo -n "odigos:$ODIGOS_TOKEN" | base64) cat < odigos-dockerconfig.json { "auths": { "registry.odigos.io": { "username": "odigos", "password": "$ODIGOS_TOKEN", "auth": "$ODIGOS_DOCKER_AUTH" } } } EOF ``` For Docker, write the file to the default config path (this overwrites `~/.docker/config.json` if it already exists): ```bash theme={null} mkdir -p ~/.docker cp odigos-dockerconfig.json ~/.docker/config.json ``` To keep your existing Docker config, use a dedicated directory instead: ```bash theme={null} mkdir -p /tmp/odigos-docker cp odigos-dockerconfig.json /tmp/odigos-docker/config.json export DOCKER_CONFIG=/tmp/odigos-docker ``` For Podman, write the file to the default auth path: ```bash theme={null} mkdir -p ~/.config/containers cp odigos-dockerconfig.json ~/.config/containers/auth.json ``` Or pass the file per command: ```bash theme={null} ``` Create a Docker registry pull secret when an in-cluster Job, GitOps tool, or other Kubernetes process needs to pull from `registry.odigos.io`. ```bash theme={null} kubectl create secret docker-registry odigos-enterprise-registry \ --namespace odigos-system \ --docker-server=registry.odigos.io \ --docker-username=odigos \ --docker-password="$ODIGOS_TOKEN" kubectl label secret odigos-enterprise-registry -n odigos-system odigos.io/system-object=true ``` You can use a different secret name if this credential is only for your mirroring pipeline. Use the name `odigos-enterprise-registry` if Odigos itself will pull from `registry.odigos.io`. Or apply the secret declaratively. To do so, first generate the `auth` value with: ```bash theme={null} echo -n "odigos:$ODIGOS_TOKEN" | base64 ``` Then apply: ```yaml theme={null} apiVersion: v1 kind: Secret metadata: name: odigos-enterprise-registry namespace: odigos-system labels: odigos.io/system-object: "true" type: kubernetes.io/dockerconfigjson stringData: .dockerconfigjson: | { "auths": { "registry.odigos.io": { "username": "odigos", "password": "", "auth": "" } } } ``` The pull, tag, and push examples below use Docker. If you authenticated with Podman, replace `docker` with `podman`. ### Step 3: Pull the Images Start by pulling the required images from the enterprise registry (`registry.odigos.io`) onto your local machine or CI environment: ```bash theme={null} docker pull registry.odigos.io/odigos-enterprise-instrumentor:$VERSION docker pull registry.odigos.io/odigos-ui:$VERSION docker pull registry.odigos.io/odigos-enterprise-odiglet:$VERSION docker pull registry.odigos.io/odigos-enterprise-collector:$VERSION docker pull registry.odigos.io/odigos-victoria-metrics:$VERSION docker pull registry.odigos.io/odigos-cache:$VERSION ``` If you are using the [k8s-init-container](/enterprise/instrumentations/configuration/mount-method#3-InitContainer) mount method make sure the odigos-agents image is also pulled, as it's required by the injected InitContainer. On Kubernetes 1.35+, that init container also needs the registry pull secret in each instrumented namespace. See [Init container image pull secrets](/enterprise/instrumentations/configuration/mount-method#init-container-image-pull-secrets). **This is not the default configuration**, if you're unsure what this means, you can safely ignore this step and continue with the standard setup. ```bash theme={null} docker pull registry.odigos.io/odigos-enterprise-agents:$VERSION ``` Odigos component images are published in multi-arch for ARM64 and AMD64 architectures. When pulling these images, be sure to specify the correct architecture *for the environment they will be deployed*. ```bash theme={null} docker pull --platform $PLATFORM registry.odigos.io/odigos-enterprise-instrumentor:$VERSION docker pull --platform $PLATFORM registry.odigos.io/odigos-ui:$VERSION docker pull --platform $PLATFORM registry.odigos.io/odigos-enterprise-odiglet:$VERSION docker pull --platform $PLATFORM registry.odigos.io/odigos-enterprise-collector:$VERSION docker pull --platform $PLATFORM registry.odigos.io/odigos-victoria-metrics:$VERSION docker pull --platform $PLATFORM registry.odigos.io/odigos-cache:$VERSION ``` ### Step 4: Tag the Images Next, Tag each image with your custom Docker registry prefix: ```bash theme={null} docker tag registry.odigos.io/odigos-enterprise-instrumentor:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-instrumentor:$VERSION docker tag registry.odigos.io/odigos-ui:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION docker tag registry.odigos.io/odigos-enterprise-odiglet:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-odiglet:$VERSION docker tag registry.odigos.io/odigos-enterprise-collector:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-collector:$VERSION docker tag registry.odigos.io/odigos-victoria-metrics:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION docker tag registry.odigos.io/odigos-cache:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION ``` > **Note:** Enterprise images are hosted at `registry.odigos.io`. Image names are `odigos-` or `odigos-enterprise-` without an embedded `keyval/` path segment. If you previously mirrored images under an older layout, re-tag them to match this naming when pushing to your custom registry. > **Note:** Odigos Enterprise now runs a dedicated collector image, `odigos-enterprise-collector`, in place of `odigos-collector`. If you were already mirroring `odigos-collector`, mirror `odigos-enterprise-collector` as well before upgrading, otherwise the gateway and the node collectors will fail to pull. ### Step 5: Push the Images Now, push the tagged images to your custom Docker registry: ```bash theme={null} docker push $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-instrumentor:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-odiglet:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-collector:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION ``` ### Step 6: Configure Access for Private Registries If your Docker registry is private, configure your Kubernetes cluster to pull images from it. Refer to the official Kubernetes documentation for guidance: [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/). Pass the secret name as `imagePullSecrets` at install time so Odigos components can pull mirrored images. For `k8s-init-container` on Kubernetes 1.35+, also copy that secret into each instrumented namespace. See [Init container image pull secrets](/enterprise/instrumentations/configuration/mount-method#init-container-image-pull-secrets). > **Note: To ensure your Odigos deployments and daemonsets can pull images from a private registry, patch them to add the image pull secret.** > > **Use the following commands:** > > ``` > export ODIGOS_NAMESPACE= > export IMAGE_PULL_SECRET= > kubectl patch deployment odigos-ui -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-instrumentor -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-gateway -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-victoriametrics -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-cache -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch daemonset odiglet -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch daemonset odigos-data-collection -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > ``` ### Step 7: Install Odigos Using Custom Images Finally, install Odigos using the images from your custom Docker registry: ```bash theme={null} odigos install --image-prefix=$CUSTOM_DOCKER_REGISTRY --onprem-token $ODIGOS_TOKEN ``` ```bash theme={null} helm repo update helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace --set imagePrefix=$CUSTOM_DOCKER_REGISTRY --set onPremToken=$ODIGOS_TOKEN kubectl label namespace odigos-system odigos.io/system-object="true" ``` These steps cover the **Odigos K8s agent** chart (`odigos`). For Odigos Central (`odigos-central`), including the bundled Keycloak image, see [Custom Docker Registry](/central/installation#custom-docker-registry). # Installation Source: https://docs.odigos.io/enterprise/setup/installation Installation issues? Need help? Message us on [Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) Checkout the [installation options](/enterprise/setup/installation-options) to customize your installation. There are 3 ways to install odigos in your kubernetes cluster: * [Odigos CLI](#odigos-cli) * [Helm Chart](#helm-chart) * [OpenShift Operator](#openshift-operator) All install methods will deploy and configure resources in the active Kubernetes cluster as per the current context. Before proceeding with the installation, ensure that you are targeting the correct cluster with `kubectl config current-context`. Odigos enterprise data collection is done using eBPF, so Odigos requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0. ## Odigos CLI ### Install the CLI If you are on MacOS, you can install Odigos CLI using Homebrew: ```bash theme={null} brew install odigos-io/odigos-cli/odigos ``` 1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You'll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux). 2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you'll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system. 3. Move the File to a Folder in Your `PATH`. You'll need to move this file to a folder where your computer looks for programs, known as your "PATH". * For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system's PATH. You can find out where by typing echo `%PATH%` in Command Prompt. * For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`. 4. Verify It's Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you've successfully installed it! If not, double-check that you moved the file to a folder included in your PATH. 1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows). 2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files. 3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you'll need to move the program (binary) into one of these folders. Here's how: * **Option 1**: Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder. * **Option 2**: Add the folder containing the extracted binary to your PATH manually: 1. Right-click the **Start** button and select **System**. 2. Click **Advanced system settings** on the left. 3. In the **System Properties** window, click **Environment Variables** at the bottom. 4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**. 5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file. 6. Click **OK** to close all windows. 4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you've successfully installed it. If it doesn't, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located. ### Install Odigos The enterprise version of Odigos requires a license token. Please contact the Odigos team to inquire about access to the Enterprise version. ```shell theme={null} odigos install --onprem-token $ODIGOS_TOKEN ``` This will install odigos in the `odigos-system` namespace in your kubernetes cluster. At this point, odigos is installed in your cluster. You can now add sources and destinations to instrument your cluster with OpenTelemetry. ## Helm Chart ### Add Helm Repository ```shell theme={null} helm repo add odigos https://odigos-io.github.io/odigos/ ``` ### Install Odigos The enterprise version of Odigos requires a license token. Please contact the Odigos team to inquire about access to the Enterprise version. ```shell theme={null} helm repo update helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace --set onPremToken=$ODIGOS_TOKEN ``` The license token can also be provided by creating a secret in kubernetes before installing Odigos. ```yaml theme={null} apiVersion: v1 kind: Secret metadata: name: odigos-pro namespace: odigos-system labels: odigos.io/system-object: "true" stringData: odigos-onprem-token: ``` When pulling enterprise images from `registry.odigos.io`, Odigos also needs a Docker registry pull secret named `odigos-enterprise-registry`. The Helm chart can create this automatically when `onPremToken` is set, or when it can read the token from an existing `odigos-pro` secret. For GitOps tools such as Argo CD that cannot read cluster secrets during `helm template`, manage both secrets externally and set: ```yaml theme={null} externalOnpremTokenSecret: true externalOnpremPullSecret: true ``` Those flags only provide the pull secret in the Odigos namespace. With `k8s-init-container` on Kubernetes 1.35+, copy that secret into each instrumented namespace as well. See [Init container image pull secrets](/enterprise/instrumentations/configuration/mount-method#init-container-image-pull-secrets). Create the registry pull secret before syncing Odigos. The username is always `odigos` and the password is your enterprise token: ```shell theme={null} kubectl create secret docker-registry odigos-enterprise-registry \ --namespace odigos-system \ --docker-server=registry.odigos.io \ --docker-username=odigos \ --docker-password="$ODIGOS_TOKEN" kubectl label secret odigos-enterprise-registry -n odigos-system odigos.io/system-object=true ``` Or apply it declaratively: ```yaml theme={null} apiVersion: v1 kind: Secret metadata: name: odigos-enterprise-registry namespace: odigos-system labels: odigos.io/system-object: "true" type: kubernetes.io/dockerconfigjson stringData: .dockerconfigjson: | { "auths": { "registry.odigos.io": { "username": "odigos", "password": "", "auth": "" } } } ``` Generate the `auth` value with: ```shell theme={null} echo -n "odigos:$ODIGOS_TOKEN" | base64 ``` Once the Odigos pods are online, simply run the following command to launch the Odigos UI. ```shell theme={null} odigos ui ``` ## OpenShift Operator For Operator installation steps, see the docs on: [OpenShift Installation](/enterprise/setup/installation-options#openshift-installation) # Installation Options Source: https://docs.odigos.io/enterprise/setup/installation-options You can configure odigos installation using the following options: ## Ignore Namespaces Don't show specific uninteresting namespaces in the Odigos UI. Default values: `kube-system`, `local-path-storage`, `istio-system`, `linkerd`, `kube-node-lease` Add additional namespaces to ignore with **cli**: ```shell theme={null} odigos install --ignore-namespace --ignore-namespace ``` List all namespaces to be ignored with **helm chart**: ```shell theme={null} helm install odigos odigos/odigos --set ignoreNamespace=, ``` Any Source objects for the namespace and workloads in the namespace will still be honored. On GKE, Google may warn that a custom admission controller can affect `kube-system`. The Odigos **pod mutating webhook** is cluster-wide so it can inject agents into application namespaces. It uses `failurePolicy: Ignore`, so webhook errors do not block pod creates in `kube-system`. Ignored namespaces (including `kube-system` by default) are not shown for instrumentation and are not instrumented unless you create a Source for them. ## Ignore Containers This option is useful for excluding specific sidecar containers from being instrumentated by odigos. Default values: `istio-proxy` ```shell theme={null} odigos install --ignore-container --ignore-container ``` ```shell theme={null} helm install odigos odigos/odigos --set ignoreContainer=, ``` ## Custom Container Runtime Socket Path Odigos uses the container runtime socket to query for application environment variables info, and distinguish between those injected by the runtime (via Dockerfile for example) and those defined in Kubernetes manifests. To perform this analysis, Odiglet requires access to the container runtime’s Unix socket (e.g., `containerd.sock`). However, in certain environments—such as Rancher—the socket may not be located in a default path. To address this, Odigos provides a configuration option to mount a custom container runtime socket path into the Odiglet. This path can be specified either during installation or afterwards. ### Usage 1. During and Post installation using set flag for the install command: ```shell theme={null} odigos install --set odiglet.customContainerRuntimeSocketPath= ``` 1. During installation: ```shell theme={null} helm install odigos odigos/odigos --set odiglet.customContainerRuntimeSocketPath= ``` ## OpenShift Installation Installing on OpenShift requires additional settings that can be configured based on your installation mode: Run the Odigos CLI with the `--openshift` flag: ```shell theme={null} odigos install --openshift ``` Set `openshift.enabled=true` in `Values.yaml`: ```yaml theme={null} openshift enabled: true ``` In the OpenShift console, navigate to **Operators** > **OperatorHub** and search for **Odigos**: Click **Install**: Choose a namespace to install the Operator in (this will be the same namespace that Odigos is installed in) and click **Install**: When installation is complete, click **View Operator**: Click **Create Instance** to create an `Odigos` custom resource: If you have an Enterprise Token, fill in the **On Prem Token** field with your Enterprise Token. Otherwise leave it blank to use Odigos OpenSource. Scroll down and click **Create**: Confirm that the Custom Resource has reached state `OdigosInstalled`: OpenShift installation does the following: * Uses [Red Hat certified component container images](https://odigos.io/blog/openshift-certified) * Configures required SELinux permissions for pods to access auto-instrumentation agents on RHEL nodes * Configures RBAC permissions for Odigos components required by default in OpenShift clusters By default, the `--openshift` flag will pull Odigos component images from `registry.connect.redhat.com/odigos`. These images are based on [Red Hat's Universal Base Image 9](https://catalog.redhat.com/software/base-images) and denoted with the `-rhel-certified` suffix. These images are also hosted on the Odigos DockerHub registry, which you can deploy by setting `--openshift` with `--image-prefix docker.io/keyval`. ### Exposing the Odigos UI via OpenShift Routes To access the Odigos UI in OpenShift, use either the standard `odigos ui` command or create a `Route` that exposes the `ui` service on your cluster's external address. For example: ``` oc expose service ui -n odigos-system ``` Then get the external address for the Route with `oc get routes ui -n odigos-system`. **Note:** If you're deploying Odigos on OpenShift, you may need to set the route's TLS termination to Edge to load it in your browser. you will need to modify the YAML of the oc route with `spec.tls.termination: edge` Check the OpenShift documentation or run `oc expose -h` for more information on configuring OpenShift routes according to your cluster's requirements. ## GKE Installation Installing on Google Kubernetes Engine (GKE) requires additional settings that can be configured based on your cluster type: For GKE Standard clusters: ```shell theme={null} odigos install --set gke.enabled=true ``` For GKE Autopilot clusters: ```shell theme={null} odigos install --set gke.enabled=true --set gke.autopilot=true ``` For GKE Standard clusters, set `gke.enabled=true` in `values.yaml`: ```yaml theme={null} gke: enabled: true ``` Or pass it as a flag during installation: ```shell theme={null} helm install odigos odigos/odigos --set gke.enabled=true ``` For GKE Autopilot clusters, also set `gke.autopilot=true`: ```yaml theme={null} gke: enabled: true autopilot: true ``` Or pass both flags during installation: ```shell theme={null} helm install odigos odigos/odigos --set gke.enabled=true --set gke.autopilot=true ``` GKE installation with `gke.enabled=true` does the following: * Configures a `ResourceQuota` for `system-node-critical` pods, which is required for odiglet on GKE ### GKE Autopilot GKE Autopilot restricts privileged workloads by default. Odiglet requires elevated permissions for eBPF-based instrumentation, so it must be allowlisted before it can run on Autopilot clusters. Allowlist synchronization requires GKE version `1.32.2-gke.1652000` or later. When `gke.autopilot=true`, the Helm chart: * Creates an `AllowlistSynchronizer` that syncs the Odigos odiglet allowlist from the [Google-managed partner repository](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads) * Skips node label setup in the odiglet init phase, where Autopilot nodes cannot be modified. Autopilot is auto-detected by the `gk3-` node name prefix; set `gke.autopilot=true` to force this behavior #### Node label limitations On standard clusters, odiglet sets `odigos.io/odiglet-oss-installed=true` (or `odigos.io/odiglet-enterprise-installed=true` for enterprise) on each node after preparing it for instrumentation. Autopilot does not allow modifying node labels, so odiglet skips this step. Without the label: * **Instrumented pod scheduling with hostPath or CSI mount methods** — For `k8s-host-path` and `k8s-csi-driver` mount methods, the instrumentor webhook adds required node affinity matching the odiglet installed label so pods only schedule on nodes where odiglet has prepared `/var/odigos`. Without the label on Autopilot nodes, newly instrumented pods may remain Pending and fail to schedule. The default `k8s-virtual-device` mount method does not rely on this label and is recommended on Autopilot. * **Karpenter integration** — Odiglet normally removes the `odigos.io/needs-init` startup taint when it sets the node label. This step is also skipped on Autopilot, so the Karpenter integration workflow is not supported. #### Manually create the AllowlistSynchronizer If you do not set `gke.autopilot=true`, create the `AllowlistSynchronizer` yourself before odiglet pods are scheduled. See the [GKE documentation for running Autopilot partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads) for more details. Apply the following manifest to install all Odigos allowlists from the partner repository: ```yaml theme={null} apiVersion: auto.gke.io/v1 kind: AllowlistSynchronizer metadata: name: odigos-allowlist-synchronizer spec: allowlistPaths: - "Odigos/Odigos/*" ``` ```shell theme={null} kubectl apply -f allowlist-synchronizer.yaml ``` After the allowlist is installed, verify that it is ready before odiglet pods start: ```shell theme={null} kubectl wait --for=condition=Ready allowlistsynchronizer/odigos-allowlist-synchronizer --timeout=60s kubectl get workloadallowlist ``` For more information on privileged workload allowlisting in Autopilot, see the [GKE documentation on running Autopilot partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads). # Configure Odigos with Argo CD Source: https://docs.odigos.io/enterprise/setup/odigos-with-argocd This guide shows you how to deploy Odigos in your Kubernetes cluster using Argo CD with the official Odigos Helm chart. This allows for GitOps-based management and observability instrumentation across your applications. ## Prerequisites Before you begin, make sure you have: * A running Kubernetes cluster * Argo CD installed and configured in the cluster * Helm CLI installed (optional but useful for inspection and verification) ## Step 1: Add the Odigos Helm Repository Add the Odigos Helm repository to your Helm configuration: ```bash theme={null} helm repo add odigos https://odigos-io.github.io/odigos --force-update helm repo update ``` ## Step 2: Identify the Helm Chart Version Check available versions of the Odigos chart to determine the desired `targetRevision`: ```bash theme={null} helm search repo odigos --versions ``` ## Step 3: Define the Argo CD Application Manifest Create an Argo CD `Application` custom resource to deploy Odigos using Helm. Save the following YAML as `odigos-application.yaml`: ```yaml theme={null} apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: odigos namespace: # <- Replace with your Argo CD namespace spec: project: default source: repoURL: https://odigos-io.github.io/odigos chart: odigos targetRevision: # REQUIRED: set the version from 'helm search repo odigos --versions' helm: releaseName: odigos values: | # OPTIONAL: add your custom Helm values here if needed. # Argo CD expects this field to be a YAML-formatted string. # To use chart defaults, delete this field entirely or set it to an empty string (""). destination: server: https://kubernetes.default.svc namespace: # <- Replace with the desired namespace where you want Odigos to be installed syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` ## Step 4: Apply the Application Use `kubectl` to create the application in Argo CD: ```bash theme={null} kubectl apply -f odigos-application.yaml ``` This will instruct Argo CD to deploy Odigos using the Helm chart. ## Step 5: Monitor the Deployment You can monitor the status of the deployment through the Argo CD UI: * Navigate to your Argo CD web interface. * Find the application named `odigos`. * Verify that it reaches a **Healthy** and **Synced** state. Once complete, Odigos will be installed in your cluster and ready to instrument workloads. # Configure Odigos with Karpenter Source: https://docs.odigos.io/enterprise/setup/odigos-with-karpenter ## What is Karpenter? [Karpenter](https://karpenter.sh/) is an open-source cluster autoscaler built for Kubernetes. It dynamically launches just the right compute resources to handle your cluster’s workloads and efficiently scales them based on real-time demand. Unlike traditional autoscalers, Karpenter can provision nodes in seconds and makes scheduling decisions at the pod level, enabling highly dynamic infrastructure scaling. *** ## Mount Method Compatibility Odigos support for Karpenter depends on the configured [mount method](../instrumentations/configuration/mount-method): | Mount method | Karpenter support | What you need | | - | - | - | | [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer) | Works out of the box | No additional Karpenter configuration | | [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) | Supported with Odigos Karpenter integration | Enable `karpenter.enabled` and configure NodePool `startupTaints` (see [HostPath and CSI](#hostpath-and-csi-startup-taints)) | | [`k8s-virtual-device`](../instrumentations/configuration/mount-method#1-VirtualDevice) (default) | Supported only when your Karpenter installation can apply [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) capacity | Enable the NodeOverlay feature (when required) and apply a NodeOverlay CR (see [Virtual Device](#virtual-device-nodeoverlay)) | Follow the section for your mount method below. *** ## Init Container If you use `k8s-init-container`, no Karpenter-specific configuration is required. You can stop here. *** ## Virtual Device (NodeOverlay) With `k8s-virtual-device`, instrumented pods request the extended resource `instrumentation.odigos.io/generic`. That capacity is advertised by the Odigos device plugin on nodes where `odiglet` is already running. Karpenter simulates new nodes from instance-type information. Because that simulation does not include the Odigos device resource by default, Karpenter can fail provisioning with errors such as: ```text theme={null} no instance type has enough resources ... instrumentation.odigos.io/generic: "1" ``` ### Provider and version caveats Support depends on your **Karpenter version, cloud provider, and how the controller is packaged**: * [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is an alpha Karpenter feature (available since Karpenter v1.7.x). Exact enablement steps and API readiness vary by release. * On many installs (including official AWS Karpenter Helm charts), NodeOverlay is **disabled by default** and must be turned on with a feature gate, for example: ```bash theme={null} helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \ --namespace \ --reuse-values \ --set settings.featureGates.nodeOverlay=true ``` Equivalently, set `--feature-gates NodeOverlay=true` / `FEATURE_GATES=...,NodeOverlay=true` on the Karpenter controller. * Not every Karpenter distribution applies NodeOverlays during provisioning, even if the CRD and feature gate exist. Validate in your environment that creating the overlay below allows Karpenter to launch nodes for pods that request `instrumentation.odigos.io/generic`. If NodeOverlay is unavailable or ineffective in your Karpenter install, use [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer), or [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) with the [startup taint integration](#hostpath-and-csi-startup-taints). ### Apply a NodeOverlay After NodeOverlay support is enabled in your Karpenter controller, apply a [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) that adds the Odigos device capacity for the NodePool that should run instrumented workloads. Replace `default` with your NodePool name if needed: ```bash theme={null} cat <<'EOF' | kubectl apply -f - apiVersion: karpenter.sh/v1alpha1 kind: NodeOverlay metadata: name: odigos-instrumentation spec: requirements: - key: karpenter.sh/nodepool operator: In values: ["default"] capacity: instrumentation.odigos.io/generic: "1024" EOF ``` Confirm the overlay is ready: ```bash theme={null} kubectl get nodeoverlay odigos-instrumentation ``` Karpenter should then be able to provision nodes for pods that request `instrumentation.odigos.io/generic`. After the node registers, `odiglet` advertises the real device capacity so pods can schedule. *** ## HostPath and CSI (Startup Taints) For the [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) and [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) mount methods, Odigos adds a **node affinity** rule to instrumented pods so they only run on nodes where `odiglet` is installed. This works fine in static clusters — but in Karpenter-managed clusters, this causes a problem: * `odiglet` is installed by a DaemonSet **after** the node is created. * The label `odigos.io/odiglet-installed=true` is only added to the node at runtime. * **Karpenter does not know in advance** that this label will appear. * So when the scheduler tries to place a pod with this affinity, Karpenter concludes: "no nodes exist (or can be created) that match this label" — and **no node is provisioned**. Once you enable the **Karpenter integration option** (explained below), Odigos will no longer apply node affinity to new instrumented pods. Instead, it relies on a **startup taint mechanism**: * Nodes launched by Karpenter should be initialized with a special taint (`odigos.io/needs-init=NoSchedule`). The exact method for configuring this taint in Karpenter will be explained in the following section. * The `odiglet` removes this taint after it prepares the node for instrumentation. * This allows instrumented pods to schedule only **after** the node is ready — achieving the same safety as affinity, without blocking provisioning. ### 1. Configure the Karpenter NodePool CRD You must configure your Karpenter [NodePool](https://karpenter.sh/docs/concepts/nodepools/) to include a startup taint. This ensures Odigos can prepare each node before instrumented pods are scheduled onto it. Add the following under `spec.template.spec.startupTaints`: ```yaml theme={null} spec: template: spec: startupTaints: - key: odigos.io/needs-init effect: NoSchedule ``` To safely append this taint without overwriting any existing `startupTaints`, use the following command: ```bash theme={null} export NODEPOOL_NAME= kubectl patch nodepool $NODEPOOL_NAME --type merge --patch "$(kubectl get nodepool $NODEPOOL_NAME -o json | jq -c '.spec.template.spec.startupTaints = (.spec.template.spec.startupTaints // []) + [{"key":"odigos.io/needs-init","effect":"NoSchedule"}] | {spec: {template: {spec: .spec.template.spec}}}')" ``` Replace `` with the actual name of your Karpenter NodePool. ### 2. Add the Startup Taint to Existing Nodes To manually add the startup taint to all current nodes: ```bash theme={null} kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} kubectl taint node {} odigos.io/needs-init=:NoSchedule --overwrite ``` This ensures existing nodes behave consistently until they're replaced by Karpenter-managed ones. ### 3. Enable the Karpenter Integration in Odigos Once the tainting setup is complete, inform Odigos to disable node affinity and rely on the taint mechanism. Run the following using the Odigos CLI: ```bash theme={null} odigos set config karpenter-enabled=true ``` If installing or upgrading using helm: ```bash theme={null} helm upgrade --install odigos odigos/odigos \ --namespace odigos-system \ --set karpenter.enabled=true ``` ### 4. Restart the Odiglet DaemonSet To ensure that taint-removal logic is triggered on all nodes, perform a rolling restart of the `odiglet` DaemonSet: ```bash theme={null} kubectl rollout restart daemonset/odiglet -n odigos-system ``` # System Requirements Source: https://docs.odigos.io/enterprise/setup/system-requirements To ensure optimal performance and compatibility when deploying Odigos, your environment must meet the following system requirements: ## Kubernetes Version Odigos requires Kubernetes version 1.21.12 or higher. Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0. The required Linux capabilities depend on your kernel version: * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`. * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`. We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`. **What to expect if some nodes don't meet the Kernel requirement:** * eBPF Agents: If a node's kernel version is below 5.4.0, eBPF-based data collection will not be applied to workloads on that node. An error will appear in the UI for any source running on unsupported nodes, indicating that instrumentation was not possible. * Non-eBPF Agents: Non-eBPF-based agents are not affected by kernel version requirements and will run and instrument workloads on any Linux version. This allows users to maintain observability on nodes that meet the requirements without needing to upgrade all cluster nodes immediately. ## Kubernetes Cluster Types * [Kind](https://kind.sigs.k8s.io) * [Minikube](https://minikube.sigs.k8s.io/docs/start) * [K3s](https://k3s.io) * [Red Hat OpenShift](https://catalog.redhat.com/software/container-stacks/detail/675201b3ade18c062e2efc0b) (see [how to install on OpenShift](../setup/installation-options#openshift-installation)) * [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/) * [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) (see [how to install on GKE](/enterprise/setup/installation-options#gke-installation)) * [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service) ## Resource Requirements To achieve its functionality Odigos uses a set of Deployments and DaemonSets. ### Cluster The following components run as Deployment and request the following resources **(per pod)**: | Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit | | - | - | - | - | - | | odigos-autoscaler | 64Mi | 512Mi | 10m | 500m | | odigos-instrumentor (2 replicas) | 64Mi | 512Mi | 10m | 500m | | odigos-ui | 64Mi | 512Mi | 10m | 500m | In addition, Odigos deploys OpenTelemetry Collector Deployment, with auto scaling (multiple replicas) and configurable resources requests and limits. The default values (unless overridden) are: | Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit | | - | - | - | - | - | | odigos-gateway | 600Mi | 600Mi | 1000m | 1000m | Read more [here](/enterprise/pipeline/configuration). ### Per Node A DaemonSet running on every node in the cluster, consuming resources as outlined below: | Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit | | - | - | - | - | - | | odiglet | 500Mi | 500Mi | 500m | 500m | | data-collection | 250Mi | 500Mi | 250m | 500m | | deviceplugin | 200Mi | 300Mi | 40m | 100m | You can override these values by providing a custom configuration (read more [here](/enterprise/pipeline/configuration)). ## Network Requirements Ensure that any network-layer components are properly configured to allow the required connectivity between Odigos components. Improper configuration may impact the correct operation of Odigos. To ensure proper operation of Odigos components and instrumented applications, the following network connectivity must be allowed inside the Kubernetes cluster: ### Core Network Connectivity | Source | Destination Service | Ports | Purpose | | - | - | - | - | | Instrumented Application Pods | `odiglet-local` | `4320` | Retrieve instrumentation configuration from Odiglet | | Instrumented Application Pods | `odigos-data-collection-local-traffic` | `4317` (gRPC), `4318` (HTTP) | Export application telemetry data (OTLP protocol) | | Odiglet Pods | `odigos-data-collection-local-traffic` | `4318` | Export eBPF-collected telemetry data | ### Network Layer Considerations In some environments, the following network components may affect traffic between these services: * Service meshes (e.g. Istio, Linkerd, Cilium, Consul) * HTTP proxies or egress gateways * Transparent proxies (corporate or cluster-level) * Network security appliances (WAFs, L7 firewalls, DPI systems) * Kubernetes NetworkPolicies # Uninstall Source: https://docs.odigos.io/enterprise/setup/uninstall ## With CLI ```shell theme={null} odigos uninstall ``` ## Uninstall Existing Odigos Installation with Helm If you installed Odigos using Helm, you can uninstall it by running the following commands: ```shell theme={null} helm delete odigos -n odigos-system kubectl delete ns odigos-system ``` # Upgrade Source: https://docs.odigos.io/enterprise/setup/upgrade ## Finding the Current Version ```shell theme={null} odigos version ``` Will print the version of the CLI tool and the version of odigos that is currently installed in the active cluster. ## Upgrading Existing Odigos Installation with CLI First, upgrade the CLI tool to latest version: If you are on MacOS, you can install Odigos CLI using Homebrew: ```bash theme={null} brew install odigos-io/odigos-cli/odigos ``` 1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux). 2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system. 3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH". * For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt. * For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`. 4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH. 1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows). 2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files. 3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how: * **Option 1**: Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder. * **Option 2**: Add the folder containing the extracted binary to your PATH manually: 1. Right-click the **Start** button and select **System**. 2. Click **Advanced system settings** on the left. 3. In the **System Properties** window, click **Environment Variables** at the bottom. 4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**. 5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file. 6. Click **OK** to close all windows. 4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located. And upgrade Odigos installation in the active kubernetes cluster with CLI: ```shell theme={null} odigos upgrade ``` ## Upgrade Existing Odigos Installation with Helm ```shell theme={null} helm repo update helm upgrade odigos odigos/odigos --namespace odigos-system ``` ```shell theme={null} helm repo update helm search repo odigos --versions ## list all available versions helm upgrade --install odigos odigos/odigos \ --namespace odigos-test \ --create-namespace \ --version ``` # Verifying Odigos Images Source: https://docs.odigos.io/enterprise/setup/verifying-images All Odigos container images are signed at build time with [Sigstore cosign](https://docs.sigstore.dev/) (keyless), and each carries a CycloneDX SBOM attestation. Signatures are stored in the registry next to the image and recorded in the public [Rekor transparency log](https://search.sigstore.dev/), so you can verify that any image you pull was published by the Odigos release pipeline — no keys involved. ## Verify an image Verification needs [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) v3+. Always verify by digest: Verify against the canonical registry, `us-central1-docker.pkg.dev/odigos-cloud/components`. `registry.odigos.io` is a pull alias: images resolve through it, but signature discovery for retroactively signed versions does not currently surface there. ```bash theme={null} IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-odiglet DIGEST=$(docker buildx imagetools inspect "${IMG}:" --format '{{.Manifest.Digest}}') cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \ "${IMG}@${DIGEST}" ``` The identity accepts two workflows on purpose: new releases are signed by the release pipeline, while versions published before signing went live were signed retroactively by the audited backfill workflow. A single-identity pattern would fail on one era or the other. To verify and inspect the attached SBOM (same identity rules): ```bash theme={null} cosign verify-attestation --type cyclonedx \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \ "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components' ``` The SBOM attestation is attached to the **multi-arch index digest** — always pass the digest from `imagetools inspect` as above, not a per-platform child digest (children carry signatures but no attestation). Run as-is — no placeholders to fill in: ```bash theme={null} IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor TAG=v1.35.0 DIGEST=$(docker buildx imagetools inspect "${IMG}:${TAG}" --format '{{.Manifest.Digest}}') IDENTITY='^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp "$IDENTITY" \ "${IMG}@${DIGEST}" cosign verify-attestation --type cyclonedx \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp "$IDENTITY" \ "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components | length' ``` Expected output (trimmed): ```text theme={null} Verification for us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor@sha256:f46fa614… -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates [{"critical":{"identity":{"docker-reference":"…/odigos-instrumentor"}, "image":{"docker-manifest-digest":"sha256:f46fa614…"},"type":"cosign container image signature"}, …}] 1022 ``` The 1022 components are the SBOM contents — inspect them with `jq '.predicate.components'` instead of `length`, e.g. the first entries here are `base-files 13.8+deb13u6` and `github.com/argoproj/argo-rollouts v1.9.1`. Because v1.35.0 predates release-time signing, the certificate for this version names the audited backfill workflow (`ci-core/.github/workflows/backfill-sign.yml`) — that is the expected identity for retroactively signed history, and the reason the identity regexp accepts both workflows. ## Signing identities Each artifact is signed by its own release workflow — never verify with a broad pattern like `.*odigos-io.*`. The canonical list of identities and the version each artifact is signed from lives in [`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md). | Artifact | Release identity (regexp) | | - | - | | Component images (`odigos-*`) | `^https://github\.com/odigos-io/odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*$` | | CLI image + `odigos-victoria-metrics` + `odigos-cache` | `^https://github\.com/odigos-io/odigos/\.github/workflows/release\.yml@refs/heads/main$` | | Enterprise images (`odigos-enterprise-*`) | `^https://github\.com/odigos-io/odigos-enterprise/\.github/workflows/release-images\.yml@refs/(tags/.*\|heads/.*)$` | | `odigos-cli-offsets` | `^https://github\.com/odigos-io/enterprise-go-instrumentation/\.github/workflows/offsets-push\.yml@refs/heads/.*$` | | `odigos-vmagent-instrumentations` | `^https://github\.com/odigos-io/vm-agent/\.github/workflows/publish-agents\.yml@refs/(heads/.*\|tags/agents/.*)$` | To also cover retroactively signed history, combine the release identity with the backfill identity in one pattern, as the commands above do: `^https://github\.com/odigos-io/(@|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$` Two more certificate details are expected and legitimate: * **Backfill**: historical releases were signed retroactively by `odigos-io/ci-core/.github/workflows/backfill-sign.yml` — the certificate honestly records the backfill identity and date, and it is auditable in Rekor. * **Stable `x.y.0` tags** verify against the release-candidate tag identity: the stable tag is a retag of the exact RC digest, so the certificate names the RC that built it. ## Attestable images Every image below is signed and carries a CycloneDX SBOM attestation, from the version listed in [`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md): * **Odigos platform**: `odigos-instrumentor`, `odigos-collector`, `odigos-odiglet`, `odigos-ui`, `odigos-operator`, `odigos-agents`, `odigos-cli`, `odigos-victoria-metrics`, `odigos-cache` * **Odigos Enterprise**: `odigos-enterprise-odiglet`, `odigos-enterprise-instrumentor`, `odigos-enterprise-ui`, `odigos-enterprise-collector`, `odigos-enterprise-central-proxy`, `odigos-enterprise-central-backend`, `odigos-enterprise-central-ui`, `odigos-enterprise-agents`, `odigos-enterprise-connector-runtime`, `odigos-enterprise-connector-aws`, `odigos-enterprise-connector-azure`, `odigos-enterprise-connector-gcp` — `odiglet` and `agents` also publish `-extended` variants (separate digests, same identity) * **Supporting images**: `odigos-cli-offsets`, and `odigos-vmagent-instrumentations` (signed for publishes after 2026-08-12) ## Air-gapped verification The Rekor proof travels inside the signature bundle, so verification works offline: supply `--trusted-root` with a Sigstore trusted-root file instead of relying on `--offline` (removed in cosign 3.x). # Telemetry Types Source: https://docs.odigos.io/enterprise/telemetry-types Odigos supports the producing and correlation of the following telemetry types: * Traces * Metrics * Logs For a telemetry type to be collected, a [destination](./backends-overview) that accepts it must be configured. In addition, users can enable or disable the collection of a telemetry type on a per-application basis. If none of the applications are configured to collect a telemetry type, the telemetry type will not be collected. ## Traces Distributed traces collected by Odigos will automatically include spans for popular open source projects such as HTTP clients and servers, gRPC clients and servers, Database clients, and many more. In addition, users can enrich their distributed traces with manually created spans by using the relevant [OpenTelemetry APIs](https://opentelemetry.io/docs/instrumentation/). The collection of traces is achieved by combining two open source technologies: * OpenTelemetry for languages with JIT compilation such as Python, Java,.NET and Javascript. * eBPF for compiled languages such as Go. Being based on popular open-source standards allows Odigos to automatically support a huge number of libraries and frameworks. Below is a list of supported libraries for every language: ## Metrics There are three kinds of metrics that Odigos supports: * Metrics related to the running of the application (number of HTTP requests, latency, DB connections, etc.) * Metrics related to the language runtime (GC, threads, heap, etc.) * Metrics related to the host environment (CPU, memory, disk, etc.) ### Applicative Metrics Application-related metrics are derived from the distributed traces described above. Many observability vendors automatically compute application metrics according to the distributed traces. For example, the number of requests served by the application is computed by counting the number of spans with the `http.server` label. In case a destination that does not automatically compute application metrics (like Prometheus + Tempo) is configured, Odigos will compute them on its own by invoking the [spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md) connector. ### Runtime Metrics Those metrics describe the behavior of the language runtime, such as garbage collection, threads, heap, etc. Runtime metrics are collected by the same mechanism that collects distributed traces. ### Host Metrics Host-related metrics are collected via the [host metrics receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) ## Logs Currently, Odigos will ship logs written by the application to the stdout or stderr of the process. Correlation to other telemetry types will be done according to timestamp and resource. We are in the process of shipping logs using the same mechanism that collects distributed traces, this would allow users to correlate logs to distributed traces and metrics in a more accurate way. # Usage report Source: https://docs.odigos.io/enterprise/usage-reports Odigos sends non-sensitive, non-personally identifiable information to our system. This information helps Odigos maintainers to understand how the open-source community runs Odigos and improve the project. ### Disabling the report #### Using helm chart When installing Odigos using helm chart, you can disable the report by setting `telemetry.enabled` to `false`. #### Using CLI When installing Odigos using CLI, you can disable the report by setting `--telemetry` to `false`. ### What is being reported Odigos collects the following anonymous information: * Are traces/metrics/logs enabled * Selected destinations type (e.g. Jaeger, Prometheus, etc.) * Number of instrumented applications by programming language * Number of unrecognized applications * Number of nodes # Adding New Observability Destination Source: https://docs.odigos.io/oss/adding-new-dest There are tens if not hundreds of different observability destinations. Odigos's goal is to provide a seamless and easy way to ship observability data to any one of them. In this guide, you will learn how to contribute a new destination to Odigos. We will create a new dummy destination called `mydest`. Creating a new destination involves these steps: 1. Extending the UI for the new destination 2. Adding the collector configuration for the new destination 3. Generating the documentation for the new destination ### User Interface For our new destination to be visible in the UI, we need to define the destination metadata and fields in YAML format: Go to [`destinations/logos/`](https://github.com/odigos-io/odigos/tree/main/destinations/logos) directory and add your logo. * Must be in SVG format * Example: `mydest.svg` ```svg mydest.svg theme={null} ``` Go to [`destinations/data/`](https://github.com/odigos-io/odigos/tree/main/destinations/data) directory and create a new YAML file. * Example: `mydest.yaml` | Key | Example | Required | Description | Docs | | - | - | :-: | - | :-: | | metadata.type | `mydest` | ✅ | unique identifier for the destination | ✅ | | metadata.displayName | `My Destination` | ✅ | display name of the destination | ✅ | | metadata.category | `managed` | ✅ | one-of: `managed` `self hosted` | ✅ | | spec.image | `mydest.svg` | ✅ | logo of the destination | ✅ | | spec.signals.traces.supported | `true` | ✅ | is tracing supported? | ✅ | | spec.signals.metrics.supported | `true` | ✅ | is metrics supported? | ✅ | | spec.signals.logs.supported | `false` | ✅ | is logging supported? | ✅ | | spec.note.type | `Note` | ❌ | type of [callout box](https://mintlify.com/docs/content/components/callouts) to display

one-of: `Note` `Info` `Warning` `Tip` `Check` | ✅ | | spec.note.content | `We handle the endpoint internally, so you don't need to provide it.` | ❌ | content of the callout box | ✅ | | spec.fields | - | ✅ | list of fields to be configured, [see specification here](#allowed-properties-for-spec-fields) | ✅ | | Key | Example | Required | Description | Docs | | - | - | :-: | - | :-: | | name | `MYDEST_API_KEY` | ✅ | key name of the field | ✅ | | displayName | `API Key` | ✅ | display name of the field | ✅ | | secret | `true` | ❌ | is the field a secret? (e.g. password, token, etc.) | ❌ | | initialValue | - | ❌ | default value for the field | ✅ | | componentType | `input` | ✅ | one-of: `input` `textarea` `dropdown` `checkbox` `multiInput` `keyValuePairs` | ✅ | | componentProps.type | `password` | ❌ | the type of value (e.g. string, number, password, etc.) | ✅ | | componentProps.required | `true` | ❌ | is the field required? | ✅ | | componentProps.values | - | ❌ | list of options for dropdown | ❌ | | componentProps.placeholder | `glc_eyJvIj...` | ❌ | placeholder text for input | ✅ | | componentProps.tooltip | `Obtained from your "My Dest" account` | ❌ | tooltip text for input | ✅ | | renderCondition | `["MYDEST_URL", "!=", ""]` | ❌ | hide/show the field in the UI (form only)

one-of: `[boolean]`
`[string, operator, any]`

where `string` is the key name of the dependency field, and `operator` is one-of: `!=` `==` `>=` `<=` `>` `<` | ❌ | | hideFromReadData | `["MYDEST_URL", "==", ""]` | ❌ | hide/show the field in the UI (read only)

one-of: `[boolean]`
`[string, operator, any]`

where `string` is the key name of the dependency field, and `operator` is one-of: `!=` `==` `>=` `<=` `>` `<` | ❌ | | customReadDataLabels | - | ❌ | custom labels for the UI (read only) | ❌ | ```yaml mydest.yaml theme={null} apiVersion: internal.odigos.io/v1beta1 kind: Destination metadata: type: mydest displayName: My Destination category: managed spec: image: mydest.svg signals: traces: supported: true metrics: supported: true logs: supported: false fields: - name: MYDEST_URL displayName: URL componentType: input componentProps: required: true - name: MYDEST_REGION displayName: Region componentType: input componentProps: required: true - name: MYDEST_API_KEY displayName: API Key componentType: input componentProps: type: password required: true secret: true ``` ### Collector Configuration Now that our new vendor can be persisted/loaded in the Kubernetes data store, we need to implement the collector configuration. Go to [`common/dests.go`](https://github.com/odigos-io/odigos/tree/main/common/dests.go) and add your new destination to the `DestinationType` enum. * Make sure the value is the same as the `type` property in `mydest.yaml` Go to [`common/config`](https://github.com/odigos-io/odigos/tree/main/common/config) directory and create a new GO file. * Example: `mydest.go` - **DestType** - Returns the enum value of the destination added to [`common/dests.go`](https://github.com/odigos-io/odigos/tree/main/common/dests.go) in step 1. - **ModifyConfig** - Is called with the `dest` object which holds the data received from the UI and the `currentConfig` object. The `currentConfig` object contains the current configuration of the gateway collector. Modify this object to include the [OpenTelemetry configuration](https://opentelemetry.io/docs/collector/configuration/) needed by your destination. You can assume a basic configuration is already provided in the `currentConfig` object, for details see `getBasicConfig` method in [`common/config/root.go`](https://github.com/odigos-io/odigos/tree/main/common/config/root.go) file. * Make sure to give any exporter or pipeline a unique name in order to avoid conflicts, use the conventions: * `otlp/-` for OTel gRPC exporters * `otlphttp/-` for OTel HTTP exporters * `traces/-` for traces pipelines * `metrics/-` for metrics pipelines * `logs/-` for logs pipelines * You can use the utility methods `isTracingEnabled`, `isMetricsEnabled` and `isLoggingEnabled` to determine which signals are selected by the user for the destination, and configure the collector accordingly. ```go mydest.go theme={null} package config import ( "fmt" "github.com/odigos-io/odigos/common" ) const ( regionKey = "MYDEST_REGION" urlKey = "MYDEST_URL" apiKeyKey = "MYDEST_API_KEY" ) type MyDest struct{} func (m *MyDest) DestType() common.DestinationType { // DestinationType defined in common/dests.go return common.MyDestDestinationType } func (m *MyDest) ModifyConfig(dest ExporterConfigurer, currentConfig *Config) ([]string, error) { config := dest.GetConfig() // To make sure that the exporter and pipeline names are unique, we'll need to define a unique ID uniqueUri := "mydest-" + dest.GetID() region, exists := config[regionKey] if !exists { region = "us" return errorMissingKey(regionKey) } endpoint, exists := config[urlKey] if !exists { endpoint = fmt.Sprintf("https://%s.mydest.com:4317", region) } // Modify the exporter here exporterName := "otlp/" + uniqueUri exporterConfig := GenericMap{ "endpoint": endpoint, "headers": GenericMap{ // The API Key is a secret, so we can't expose it in the config "x-mydest-header-apikey": "${MYDEST_API_KEY}", }, } currentConfig.Exporters[exporterName] = exporterConfig // Modify the pipelines here var pipelineNames []string if isTracingEnabled(dest) { pipeName := "traces/" + uniqueUri currentConfig.Service.Pipelines[pipeName] = Pipeline{ Exporters: []string{exporterName}, } pipelineNames = append(pipelineNames, pipeName) } if isMetricsEnabled(dest) { pipeName := "metrics/" + uniqueUri currentConfig.Service.Pipelines[pipeName] = Pipeline{ Exporters: []string{exporterName}, } pipelineNames = append(pipelineNames, pipeName) } if isLoggingEnabled(dest) { pipeName := "logs/" + uniqueUri currentConfig.Service.Pipelines[pipeName] = Pipeline{ Exporters: []string{exporterName}, } pipelineNames = append(pipelineNames, pipeName) } return pipelineNames, nil } ``` Go to [`common/config/root.go`](https://github.com/odigos-io/odigos/tree/main/common/config/root.go) and add the new destination to the `availableConfigers` list. * Example: `&MyDest{}` ```go common/config/root.go theme={null} var availableConfigers = []Configer{ &MyDest{}, /* List of existing destinations... */ } ``` ### Kubernetes-Specific Collector Configuration If your destination configures Kubernetes settings, for example you have a config field that accepts a value that should be mounted in the Collector Pod, create a `K8sConfiger` object in [`instrumentor/k8sconfig`](https://github.com/odigos-io/odigos/tree/main/instrumentor/k8sconfig). This interface provides an additional function, `ModifyGatewayCollectorDeployment`, which accepts a Destination and Collector Deployment. Your implementation can use this function to modify the Deployment as necessary based on the Destination config. Note: It is the responsibility of your implementation to check for the validity of this config, such as checking for conflicts or values that already exist. The `K8sConfiger` interface also requires a `DestType()` function, similar to common Destinations. This function returns the type of common Destination that this Kubernetes implementation relies on. This is used by the instrumentor to conditionally apply the `K8sConfiger` only for the matching Destination. When it is implemented, add your `K8sConfiger` implementation to the `availableConfigers` list in [`instrumentor/k8sconfig/root.go`](https://github.com/odigos-io/odigos/tree/main/instrumentor/k8sconfig/root.go) ### Generating Documentation Make sure you have [Python installed](https://www.python.org/downloads/) on your system. Go to [`docs`](https://github.com/odigos-io/odigos/tree/main/docs) directory, and run: ```bash theme={null} python sync-dest-doc.py # OR python3 sync-dest-doc.py ``` The documentation has been generated in the [`docs/backends`](https://github.com/odigos-io/odigos/tree/main/docs/backends) directory. Feel free to add any additional content to the generated file. * Edit only between the `!! START CUSTOM EDIT !!` and `!! END CUSTOM EDIT !!` comments * Example: `mydest.mdx` ```mdx mydest.mdx theme={null} {/* !! Do not remove this comment, this acts as a key indicator in `docs/sync-dest-doc.py` !! !! START CUSTOM EDIT !! */} **Creating Account**
Go to the **[🔗 website](https://odigos.io) > Account** and click **Sign Up** **Obtaining Access Token**
Go to **⚙️ > Access Tokens** and click **Create New** {/* !! Do not remove this comment, this acts as a key indicator in `docs/sync-dest-doc.py` !! !! END CUSTOM EDIT !! */} ``` Go to [`README.md`](https://github.com/odigos-io/odigos/tree/main/README.md) and add your new destination to the list of destinations. #### That's it! Now you can use your new destination in the UI and send data to it. **Please submit a PR to the [odigos git repository](https://github.com/odigos-io/odigos).** # actions.odigos.io/v1alpha1 Source: https://docs.odigos.io/oss/api-reference/actions.odigos.io.v1alpha1 ## Resource Types * [AddClusterInfo](#AddClusterInfo) * [DeleteAttribute](#DeleteAttribute) * [K8sAttributesResolver](#K8sAttributesResolver) * [RenameAttribute](#RenameAttribute) ## `AddClusterInfo`

AddClusterInfo is the Schema for the addclusterinfo odigos action API DEPRECATED: Use odigosv1.Action instead

Field	Type	Description
`apiVersion`	string	`/v1alpha1`
`kind`	string	`AddClusterInfo`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`AddClusterInfoSpec`	No description provided.
`status` \[Required]	`AddClusterInfoStatus`	No description provided.

## `DeleteAttribute`

DeleteAttribute is the Schema for the DeleteAttribute odigos action API DEPRECATED: Use odigosv1.Action instead

Field	Type	Description
`apiVersion`	string	`/v1alpha1`
`kind`	string	`DeleteAttribute`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`DeleteAttributeSpec`	No description provided.
`status` \[Required]	`DeleteAttributeStatus`	No description provided.

## `K8sAttributesResolver`

K8sAttributesResolver allows adding an action to collect k8s attributes. DEPRECATED: Use odigosv1.Action instead

Field	Type	Description
`apiVersion`	string	`/v1alpha1`
`kind`	string	`K8sAttributesResolver`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`K8sAttributesSpec`	No description provided.
`status` \[Required]	`K8sAttributesStatus`	No description provided.

## `RenameAttribute`

RenameAttribute is the Schema for the RenameAttribute odigos action API DEPRECATED: Use odigosv1.Action instead

Field	Type	Description
`apiVersion`	string	`/v1alpha1`
`kind`	string	`RenameAttribute`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`RenameAttributeSpec`	No description provided.
`status` \[Required]	`RenameAttributeStatus`	No description provided.

## `AddClusterInfoConfig` **Appears in:** * [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

Field	Type	Description
`clusterAttributes` \[Required]	`\[]OtelAttributeWithValue`	No description provided.
`overwriteExistingValues` \[Required]	`bool`	No description provided.

## `AddClusterInfoSpec` **Appears in:** * [AddClusterInfo](#AddClusterInfo)

AddClusterInfoSpec defines the desired state of AddClusterInfo action

Field	Type	Description
`actionName` \[Required]	`string`	No description provided.
`notes` \[Required]	`string`	No description provided.
`disabled` \[Required]	`bool`	No description provided.
`signals` \[Required]	`\[]common.ObservabilitySignal`	No description provided.
`clusterAttributes` \[Required]	`\[]OtelAttributeWithValue`	No description provided.
`overwriteExistingValues` \[Required]	`bool`	No description provided.

## `AddClusterInfoStatus` **Appears in:** * [AddClusterInfo](#AddClusterInfo)

AddClusterInfoStatus defines the observed state of AddClusterInfo action

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a addclusterinfo's current state. Known .status.conditions.type are: "Available", "Progressing"

## `DeleteAttributeConfig` **Appears in:** * [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

Field	Type	Description
`attributeNamesToDelete` \[Required]	`\[]string`	No description provided.

## `DeleteAttributeSpec` **Appears in:** * [DeleteAttribute](#DeleteAttribute)

DeleteAttributeSpec defines the desired state of DeleteAttribute action

Field	Type	Description
`actionName` \[Required]	`string`	No description provided.
`notes` \[Required]	`string`	No description provided.
`disabled` \[Required]	`bool`	No description provided.
`signals` \[Required]	`\[]common.ObservabilitySignal`	No description provided.
`attributeNamesToDelete` \[Required]	`\[]string`	No description provided.

## `DeleteAttributeStatus` **Appears in:** * [DeleteAttribute](#DeleteAttribute)

DeleteAttributeStatus defines the observed state of DeleteAttribute action

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a DeleteAttribute's current state. Known .status.conditions.type are: "Available", "Progressing"

## `K8sAnnotationAttribute` **Appears in:** * [K8sAttributesConfig](#K8sAttributesConfig) * [K8sAttributesSpec](#K8sAttributesSpec)

Field	Type	Description
`annotationKey` \[Required]	`string`	The annotation name to be extracted. e.g. "kubectl.kubernetes.io/restartedAt"
`attributeKey` \[Required]	`string`	The attribute key to be used for the resource attribute created from the annotation. e.g. "kubectl.kubernetes.restartedAt"
`from` \[Required]	`string`	The source of the annotation. e.g. "pod" or "namespace" Deprecated: Use FromSources instead for specifying multiple sources with precedence.
`fromSources` \[Required]	`\[]K8sAttributeSource`	The sources from which to extract the annotation, in order of precedence (most specific first). When multiple sources are specified, the most specific source (e.g., pod) takes precedence over less specific sources (e.g., namespace). If an annotation exists in multiple sources, the value from the most specific source will be used. Supported sources: "pod", "namespace", "node"

## `K8sAttributeSource` (Alias of `string`) **Appears in:** * [K8sAnnotationAttribute](#K8sAnnotationAttribute) * [K8sLabelAttribute](#K8sLabelAttribute) ## `K8sAttributesConfig` **Appears in:** * [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

Field	Type	Description
`collectContainerAttributes` \[Required]	`bool`	Collect the following container related attributes: k8s.container.name container.id container.image.name container.image.tag
`collectReplicaSetAttributes` \[Required]	`bool`	collect replicaset related attributes (when relevant, e.g. for deployments): k8s.replicaset.name if CollectWorkloadUID is set, also collect: k8s.replicaset.uid DEPRECATED: ReplicaSet attributes are now collected by default during instrumentation.
`collectWorkloadUID` \[Required]	`bool`	Collect the following workload UID attributes: k8s.deployment.uid k8s.daemonset.uid k8s.statefulset.uid DEPRECATED: Workload UID attributes are now collected by default during instrumentation.
`collectClusterUID` \[Required]	`bool`	Collect the k8s.cluster.uid attribute, which is set to the uid of the namespace "kube-system"
`labelsAttributes`	`\[]K8sLabelAttribute`	list of labels to be extracted from the pod, and the attribute key to be used for the resource attribute created from each label.
`annotationsAttributes`	`\[]K8sAnnotationAttribute`	list of annotations to be extracted from the pod, and the attribute key to be used for the resource attribute created from each annotation.

## `K8sAttributesSpec` **Appears in:** * [K8sAttributesResolver](#K8sAttributesResolver)

Field	Type	Description
`actionName` \[Required]	`string`	No description provided.
`notes` \[Required]	`string`	No description provided.
`disabled` \[Required]	`bool`	No description provided.
`signals` \[Required]	`\[]common.ObservabilitySignal`	No description provided.
`collectContainerAttributes` \[Required]	`bool`	Collect the following container related attributes: k8s.container.name container.id container.image.name container.image.tag
`collectReplicaSetAttributes` \[Required]	`bool`	collect replicaset related attributes (when relevant, e.g. for deployments): k8s.replicaset.name if CollectWorkloadUID is set, also collect: k8s.replicaset.uid DEPRECATED: ReplicaSet attributes are now collected by default during instrumentation.
`collectWorkloadUID` \[Required]	`bool`	Collect the following workload UID attributes: k8s.deployment.uid k8s.daemonset.uid k8s.statefulset.uid DEPRECATED: Workload UID attributes are now collected by default during instrumentation.
`collectClusterUID` \[Required]	`bool`	Collect the k8s.cluster.uid attribute, which is set to the uid of the namespace "kube-system"
`labelsAttributes`	`\[]K8sLabelAttribute`	list of labels to be extracted from the pod, and the attribute key to be used for the resource attribute created from each label.
`annotationsAttributes`	`\[]K8sAnnotationAttribute`	list of annotations to be extracted from the pod, and the attribute key to be used for the resource attribute created from each annotation.

## `K8sAttributesStatus` **Appears in:** * [K8sAttributesResolver](#K8sAttributesResolver)

K8sAttributesStatus defines the observed state of K8sAttributes action

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a k8sattributes' current state. Known .status.conditions.type are: "Available", "Progressing"

## `K8sLabelAttribute` **Appears in:** * [K8sAttributesConfig](#K8sAttributesConfig) * [K8sAttributesSpec](#K8sAttributesSpec)

Field	Type	Description
`labelKey` \[Required]	`string`	The label name to be extracted. e.g. "app.kubernetes.io/name"
`attributeKey` \[Required]	`string`	The attribute key to be used for the resource attribute created from the label. e.g. "app.kubernetes.name"
`from` \[Required]	`K8sAttributeSource`	The source of the label. e.g. "pod" or "namespace" Deprecated: Use FromSources instead for specifying multiple sources with precedence.
`fromSources` \[Required]	`\[]K8sAttributeSource`	The sources from which to extract the label, in order of precedence (most specific first). When multiple sources are specified, the most specific source (e.g., pod) takes precedence over less specific sources (e.g., namespace). If a label exists in multiple sources, the value from the most specific source will be used. Supported sources: "pod", "namespace", "node"

## `OtelAttributeWithValue` **Appears in:** * [AddClusterInfoConfig](#AddClusterInfoConfig) * [AddClusterInfoSpec](#AddClusterInfoSpec)

Field	Type	Description
`attributeName` \[Required]	`string`	the name of the attribute to insert
`attributeStringValue` \[Required]	`string`	if the value is a string, this field should be used. empty string is a valid value

## `RenameAttributeConfig` **Appears in:** * [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

Field	Type	Description
`renames` \[Required]	`map\[string]string`	No description provided.

## `RenameAttributeSpec` **Appears in:** * [RenameAttribute](#RenameAttribute)

RenameAttributeSpec defines the desired state of RenameAttribute action

Field	Type	Description
`actionName` \[Required]	`string`	No description provided.
`notes` \[Required]	`string`	No description provided.
`disabled` \[Required]	`bool`	No description provided.
`signals` \[Required]	`\[]common.ObservabilitySignal`	No description provided.
`renames` \[Required]	`map\[string]string`	No description provided.

## `RenameAttributeStatus` **Appears in:** * [RenameAttribute](#RenameAttribute)

RenameAttributeStatus defines the observed state of RenameAttribute action

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a RenameAttribute's current state. Known .status.conditions.type are: "Available", "Progressing"

# odigos.io/v1alpha1 Source: https://docs.odigos.io/oss/api-reference/odigos.io.v1alpha1

Package v1alpha1 is the v1alpha1 version of the API.

## Resource Types * [Action](#odigos-io-v1alpha1-Action) * [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup) * [Destination](#odigos-io-v1alpha1-Destination) * [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig) * [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance) * [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule) * [Processor](#odigos-io-v1alpha1-Processor) * [Recommendation](#odigos-io-v1alpha1-Recommendation) * [Sampling](#odigos-io-v1alpha1-Sampling) * [Source](#odigos-io-v1alpha1-Source) ## `Action`

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`Action`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`ActionSpec`	No description provided.
`status` \[Required]	`ActionStatus`	No description provided.

## `CollectorsGroup`

CollectorsGroup is the Schema for the collectors API

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`CollectorsGroup`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`CollectorsGroupSpec`	No description provided.
`status` \[Required]	`CollectorsGroupStatus`	No description provided.

## `Destination`

Destination is the Schema for the destinations API

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`Destination`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`DestinationSpec`	No description provided.
`status` \[Required]	`DestinationStatus`	No description provided.

## `InstrumentationConfig`

InstrumentationConfig is the Schema for the instrumentationconfig API

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`InstrumentationConfig`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`InstrumentationConfigSpec`	No description provided.
`status` \[Required]	`InstrumentationConfigStatus`	No description provided.

## `InstrumentationInstance`

InstrumentationInstance is the Schema for the InstrumentationInstances API

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`InstrumentationInstance`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`InstrumentationInstanceSpec`	No description provided.
`status` \[Required]	`InstrumentationInstanceStatus`	No description provided.

## `InstrumentationRule`

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`InstrumentationRule`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`InstrumentationRuleSpec`	No description provided.
`status` \[Required]	`InstrumentationRuleStatus`	No description provided.

## `Processor`

Processor is the Schema for an Opentelemetry Collector Processor that is added to Odigos pipeline

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`Processor`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`ProcessorSpec`	No description provided.
`status` \[Required]	`ProcessorStatus`	No description provided.

## `Recommendation`

Recommendation is the Schema for enabling a catalog recommendation in the cluster.

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`Recommendation`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`RecommendationSpec`	No description provided.
`status` \[Required]	`RecommendationStatus`	No description provided.

## `Sampling`

Sampling is the Schema for the sampling rules API.

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`Sampling`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`SamplingSpec`	No description provided.
`status` \[Required]	`SamplingStatus`	No description provided.

## `Source`

Source configures an application for auto-instrumentation.

Field	Type	Description
`apiVersion`	string	`odigos.io/v1alpha1`
`kind`	string	`Source`
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`SourceSpec`	No description provided.
`status` \[Required]	`SourceStatus`	No description provided.

## `ActionSpec` **Appears in:** * [Action](#odigos-io-v1alpha1-Action)

Field	Type	Description
`actionName` \[Required]	`string`	Allows you to attach a meaningful name to the action for convenience. Odigos does not use or assume any meaning from this field.
`notes` \[Required]	`string`	A free-form text field that allows you to attach notes regarding the action for convenience. For example: why it was added. Odigos does not use or assume any meaning from this field.
`disabled` \[Required]	`bool`	A boolean field allowing to temporarily disable the action, but keep it around for future use
`signals` \[Required]	`\[]common.ObservabilitySignal`	Which signals should this action operate on.
`addClusterInfo` \[Required]	`actions/v1alpha1.AddClusterInfoConfig`	AddClusterInfo is the config for the AddClusterInfo Action.
`deleteAttribute` \[Required]	`actions/v1alpha1.DeleteAttributeConfig`	DeleteAttribute is the config for the DeleteAttribute Action.
`renameAttribute` \[Required]	`actions/v1alpha1.RenameAttributeConfig`	RenameAttribute is the config for the RenameAttribute Action.
`piiMasking` \[Required]	`actions.PiiMaskingConfig`	PiiMasking is the config for the PiiMasking Action.
`k8sAttributes` \[Required]	`actions/v1alpha1.K8sAttributesConfig`	K8sAttributes is the config for the K8sAttributes Action.
`urlTemplatization` \[Required]	`actions.URLTemplatizationConfig`	URLTemplatization is the config for the URLTemplatization Action.
`spanRenamer` \[Required]	`actions.SpanRenamerConfig`	SpanRenamer is the config for the SpanRenamer Action.
`extractAttribute` \[Required]	`actions.ExtractAttributeConfig`	ExtractAttribute is the config for the ExtractAttribute Action.
`dbQueryTemplatization` \[Required]	`actions.DbQueryTemplatizationConfig`	DbQueryTemplatization is the config for the DbQueryTemplatization Action.
`inferDbAttributes` \[Required]	`actions.InferDbAttributesConfig`	InferDbAttributes is the config for the InferDbAttributes Action.

## `ActionStatus` **Appears in:** * [Action](#odigos-io-v1alpha1-Action)

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a action's current state. Known .status.conditions.type are: "Available", "Progressing"

## `AgentEnabledReason` (Alias of `string`) **Appears in:** * [ContainerAgentConfig](#odigos-io-v1alpha1-ContainerAgentConfig) ## `AgentsTelemetrySettings` **Appears in:** * [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings) ## `Attribute` **Appears in:** * [InstrumentationInstanceStatus](#odigos-io-v1alpha1-InstrumentationInstanceStatus) * [InstrumentationLibraryStatus](#odigos-io-v1alpha1-InstrumentationLibraryStatus)

Attribute is a key-value pair that describes a component or instrumentation

Field	Type	Description
`key` \[Required]	`string`	No description provided.
`value` \[Required]	`string`	No description provided.

## `CollectorsGroupMetricsCollectionSettings` **Appears in:** * [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

Field	Type	Description
`spanMetrics` \[Required]	`common.MetricsSourceSpanMetricsConfiguration`	if not nil for node collector, it means span to metrics is enabled, and the node collector should set it up in the pipeline. span to metrics is the ability to calculate metrics like http requests/errors/duration etc from the individual spans recorded for relevant operation.
`hostMetrics` \[Required]	`common.MetricsSourceHostMetricsConfiguration`	if not nil for node collector, it means host metrics is enabled, and the opentelemetry collector "hostmetrics" receiver should be included in the pipeline. host metrics are metrics that are collected from the host node, such as cpu, memory, disk, network, etc.
`kubeletStats` \[Required]	`common.MetricsSourceKubeletStatsConfiguration`	if not nil for node collector, it means kubelet stats is enabled, and the opentelemetry collector "kubeletstats" receiver should be included in the pipeline. kubelet stats are metrics that are collected from the kubelet point of view, such as cpu, memory, disk, network, per pod, node and more.
`serviceGraph` \[Required]	`ServiceGraphSettings`	if not nil for cluster collector, it means service graph is enabled, and metrics for the "connectivity" between services should be calculated to be exported to metrics destinations.
`odigosOwnMetrics` \[Required]	`OdigosOwnMetricsSettings`	if not nil for node collector, it means that some metric destinations are intresseted in collecting metrics about: odigos, the collected data, and the pipeline itself. this allows for users to monitor and operate odigos within their existing system, create dashboards, alerting, and more.
`agentsTelemetry` \[Required]	`AgentsTelemetrySettings`	this part controls the metrics which are received from agents in the otlp receiver. it is generally enabled when we want to record metrics, and listed here for completeness. any "otlp receiver" specific settings can go here

## `CollectorsGroupResourcesSettings` **Appears in:** * [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

The raw values to control the collectors group resources and behavior. any defaulting, validations and calculations should be done in the controllers that create this CR. Values will be used as is without any further processing.

Field	Type	Description
`minReplicas` \[Required]	`int`	Minumum + Maximum number of replicas for the collector - these relevant only for gateway.
`maxReplicas` \[Required]	`int`	No description provided.
`memoryRequestMiB` \[Required]	`int`	MemoryRequestMiB is the memory resource request to be used on the pod template. it will be embedded in the as a resource request of the form `memory: \Mi`
`memoryLimitMiB` \[Required]	`int`	This option sets the limit on the memory usage of the collector. since the memory limiter mechanism is heuristic, and operates on fixed intervals, while it cannot fully prevent OOMs, it can help in reducing the chances of OOMs in edge cases. the settings should prevent the collector from exceeding the memory request, so one can set this to the same value as the memory request or higher to allow for some buffer for bursts.
`cpuRequestMillicores` \[Required]	`int`	CPU resource request to be used on the pod template. it will be embedded in the as a resource request of the form `cpu: \m`
`cpuLimitMillicores` \[Required]	`int`	CPU resource limit to be used on the pod template. it will be embedded in the as a resource limit of the form `cpu: \m`
`memoryLimiterLimitMiB` \[Required]	`int`	this parameter sets the "limit\_mib" parameter in the memory limiter configuration for the collector. it is the hard limit after which a force garbage collection will be performed. this value will end up comparing against the go runtime reported heap Alloc value. According to the memory limiter docs: Note that typically the total memory usage of process will be about 50MiB higher than this value a test from nov 2024 showed that fresh odigos collector with no traffic takes 38MiB, thus the 50MiB is a good value to start with.
`memoryLimiterSpikeLimitMiB` \[Required]	`int`	this parameter sets the "spike\_limit\_mib" parameter in the memory limiter configuration for the collector memory limiter. note that this is not the processor soft limit itself, but the diff in Mib between the hard limit and the soft limit. according to the memory limiter docs, it is recommended to set this to 20% of the hard limit. changing this value allows trade-offs between memory usage and resiliency to bursts.
`gomemlimitMiB` \[Required]	`int`	the GOMEMLIMIT environment variable value for the collector pod. this is when go runtime will start garbage collection. it is recommended to be set to 80% of the hard limit of the memory limiter.

## `CollectorsGroupRole` (Alias of `string`) **Appears in:** * [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec) * [ProcessorSpec](#odigos-io-v1alpha1-ProcessorSpec) ## `CollectorsGroupSpec` **Appears in:** * [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)

CollectorsGroupSpec defines the desired state of Collector

Field	Type	Description
`role` \[Required]	`CollectorsGroupRole`	No description provided.
`collectorOwnMetricsPort` \[Required]	`int32`	The port to use for exposing the collector's own metrics as a prometheus endpoint. This can be used to resolve conflicting ports when a collector is using the host network.
`resourcesSettings` \[Required]	`CollectorsGroupResourcesSettings`	Resources \[memory/cpu] settings for the collectors group. these settings are used to protect the collectors instances from: running out of memory and being killed by the k8s OOM killer consuming all available memory on the node which can lead to node instability pushing back pressure to the instrumented applications
`serviceGraphDisabled` \[Required]	`bool`	ServiceGraphEnabled is a feature that allows you to visualize the service graph of your application. It is enabled by default and can be disabled by setting the enabled flag to false.
`serviceGraphExtraDimensions` \[Required]	`\[]string`	ServiceGraphExtraDimensions are additional span attribute names to include as dimensions in the service graph metrics, on top of the default service.name dimension.
`serviceGraphVirtualNodePeerAttributes` \[Required]	`\[]string`	ServiceGraphVirtualNodePeerAttributes is an ordered list of span attributes used to identify uninstrumented (virtual) nodes in the service graph. The connector picks the first attribute that has a value on the span. When nil/empty, the connector uses its built-in defaults: \[peer.service, db.name, db.system].
`enableDataCompression` \[Required]	`bool`	Deprecated - use OtlpExporterConfiguration instead. EnableDataCompression is a feature that allows you to enable data compression before sending data to the Gateway collector. It is disabled by default and can be enabled by setting the enabled flag to true.
`otlpExporterConfiguration` \[Required]	`common.OtlpExporterConfiguration`	OtlpExporterConfiguration is the configuration for the OTLP exporter from node collector to cluster gateway collector.
`resourceDetectors` \[Required]	`common.ResourceDetectorsConfiguration`	ResourceDetectors controls which OpenTelemetry resource detectors are enabled on the node collector's resourcedetection processor.
`clusterMetricsEnabled` \[Required]	`bool`	ClusterMetricsEnabled is a feature that allows you to enable the cluster metrics. It is disabled by default and can be enabled by setting the enabled flag to true.
`httpsProxyAddress` \[Required]	`string`	for destinations that uses https for exporting data, this value can be used to set the address for an https proxy. when unset or empty, no proxy will be used.
`metrics` \[Required]	`CollectorsGroupMetricsCollectionSettings`	configuration for metrics handling in this collectors group if metric collection is disabled, this will be nil the content is populated only if relevant to this collectors group it's being calculated based on active destinations and their settings, and global settings provided in the odigos configuration or instrumentation rules. it allows for the collector group reconciler to be simplified, and for visibility into the aggregated settings being used to derive configurations deployments and rollouts.
`traceCorrelations` \[Required]	`CollectorsGroupTraceCorrelationsSettings`	TraceCorrelations holds trace correlations configuration derived from OdigosConfiguration. Currently only relevant for the cluster gateway collector.
`nodeSelector` \[Required]	`map\[string]string`	Node selector for the collectors group deployment. Use this to force the gateway to run only on nodes with specific labels. This is a hard requirement: the pod will be scheduled ONLY on nodes that match all labels. If no matching nodes exist, the pod will remain Pending.
`deploymentName` \[Required]	`string`	Deployment name for the collectors group deployment. Only relevant for cluster gateway collector.
`tailSampling` \[Required]	`common/api/sampling.TailSamplingConfiguration`	Sampling holds the sampling configuration derived from the OdigosConfiguration. Currently this is only relevant for the cluster gateway collector.
`samplingDryRun` \[Required]	`bool`	Set to true to enable dry run mode for sampling. When enabled, odigos will invoke the sampling logic, but will not drop any traces. This is useful while evaluating sampling rules to check for effectiveness and correctness before committing to any changes that might lose data.
`spanSamplingAttributes` \[Required]	`common/api/sampling.SpanSamplingAttributesConfiguration`	Controls whether spans are enhanced with sampling attributes (e.g. category and decisions). Capturing these attributes gives visibility into sampling decision-making and effective sampling percentages when viewing traces or querying the database with tools.

## `CollectorsGroupStatus` **Appears in:** * [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)

CollectorsGroupStatus defines the observed state of Collector

Field	Type	Description
`ready` \[Required]	`bool`	No description provided.
`receiverSignals` \[Required]	`\[]common.ObservabilitySignal`	Receiver Signals are the signals (trace, metrics, logs) that the collector has setup an otlp receiver for, thus it can accept data from an upstream component. this is used to determine if a workload should export each signal or not. this list is calculated based on the odigos destinations that were configured
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a collectorsroup's current state. Known .status.conditions.type are: "Available", "Progressing"

## `CollectorsGroupTraceCorrelationsSettings` **Appears in:** * [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

CollectorsGroupTraceCorrelationsSettings holds trace correlations configuration derived from OdigosConfiguration. Currently only relevant for the cluster gateway collector.

Field	Type	Description
`serviceIO` \[Required]	`common.TraceCorrelationsServiceIOConfiguration`	No description provided.

## `ContainerAgentConfig` **Appears in:** * [InstrumentationConfigSpec](#odigos-io-v1alpha1-InstrumentationConfigSpec)

ContainerAgentConfig is a configuration for a specific container in a workload.

Field	Type	Description
`containerName` \[Required]	`string`	The name of the container to which this configuration applies.
`agentEnabled` \[Required]	`bool`	boolean flag to indicate if the agent should be enabled for this container.
`agentEnabledReason` \[Required]	`AgentEnabledReason`	An enum reason for the agent injection decision.
`agentEnabledMessage` \[Required]	`string`	free text message to provide more information about the instrumentation decision. can be left empty if reason is self-explanatory.
`podManifestInjectionOptional` \[Required]	`bool`	set to true if the agent used in this container can be injected without pod manifest changes.
`otelDistroName` \[Required]	`string`	The name of the otel distribution to use for this container. if the name is empty, this container should not be instrumented.
`distroParams` \[Required]	`map\[string]string`	Additional parameters to the distro that controls how it's being applied. Keys are parameter names (like "libc") and values are the value to use for that parameter (glibc / musl)
`envInjectionMethod` \[Required]	`common.EnvInjectionDecision`	What method to use for injecting the agent environment variables (just those covered by the loader (PYTHONPATH, JAVA\_TOOLS\_OPTIONS, NODE\_OPTIONS)) Can be either "loader" or "pod-manifest". The injection should still check the actual values in the container manifest before injecting. Nil means that this container should not have env injection (agent should not be injected, or distro does not specify "loader" injection envs).
`traces` \[Required]	`common/api/agentsignalconfig.AgentTracesConfig`	Each enabled signal must be set with a non-nil value (even if the config content is empty). nil means that the signal is disabled and should not be instrumented/collected by the agent.
`metrics` \[Required]	`common/api/agentsignalconfig.AgentMetricsConfig`	No description provided.
`logs` \[Required]	`common/api/agentsignalconfig.AgentLogsConfig`	No description provided.
`agentDiagnostics` \[Required]	`common/api/instrumentationrules.AgentDiagnostics`	Configure the log level for the agent itselg

## `ContainerOverride` **Appears in:** * [InstrumentationConfigSpec](#odigos-io-v1alpha1-InstrumentationConfigSpec) * [SourceSpec](#odigos-io-v1alpha1-SourceSpec)

Field	Type	Description
`containerName` \[Required]	`string`	The name of the container to override.
`runtimeInfo` \[Required]	`RuntimeDetailsByContainer`	RuntimeInfo to use for agent enabling. If set for a container, the automatic detection will not be used for this container, and the distro to use will be calculated based on this value.
`otelDistroName` \[Required]	`string`	select one specific otel distro to use for this container. the selected distro must match the programming language and any constraints of the distro. if the programmin language is incorrect, the runtime info can be overridden to match
`allowConcurrentAgents` \[Required]	`bool`	Odigos by default will attempt to detect if another agent is already running in the container, and if so, will avoid adding odigos agent to the container. This option allows to override this setting for a specific container in a source. Can be used for selective setting of which containers should run odigos concurrently with other agents. empty means take odigos global config for this setting. when not empty, the value will override the global config for this container

## `CostReductionRule` **Appears in:** * [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

Field	Type	Description
`name` \[Required]	`string`	user provided name, for easier identification and reference. use short and descriptive name, like "health check", "always keep /transaction in payment service", etc. odigos does not use or assume any meaning from this field, but it is written as metric attribute, and stored as span attribute on participating spans.
`disabled` \[Required]	`bool`	if set to true, the rule will be disabled, e.g. will not be taken into account for any sampling decisions. disabled rules still participate in metrics calculations, allowing enhanced tools and data for troubleshooting and sampling maintenance.
`sourceScopes` \[Required]	`k8sconsts.SourcesScopes`	limit this rule to specific sources (by name, namespace, language, etc.) an empty list will match any source. if multiple items are set, the operation match if any one matches this relates to the "ResourceAttributes" part of a span.
`operation` \[Required]	`common/api/sampling.TailSamplingOperationMatcher`	limit this rule to specific operations. for example: specific endpoint or kafka topic. this field is optional, and if not set, the rule will be applied to all operations.
`percentageAtMost` \[Required]	`float64`	sampling percentage for cost reduction. this field is required. the final sampling percentage for traces that match this rule will be the highest possible, but at most this value.
`notes` \[Required]	`string`	optional free-form text field that allows you to attach notes for future context and maintenance. users can write why this rule was added, observations, document considerations, etc.

## `DestinationMetricsSettings` **Appears in:** * [DestinationSpec](#odigos-io-v1alpha1-DestinationSpec)

Field	Type	Description
`spanMetricsEnabled` \[Required]	`bool`	following settings enables/disables specific metric sources true - collect and send this metric source false - do not collect and send this metric source nil - use the default setting (from destination manifest, or cluster global setting)
`hostMetricsEnabled` \[Required]	`bool`	No description provided.
`kubeletStatsEnabled` \[Required]	`bool`	No description provided.
`serviceGraphEnabled` \[Required]	`bool`	No description provided.
`odigosOwnMetricsEnabled` \[Required]	`bool`	No description provided.
`agentsTelemetryEnabled` \[Required]	`bool`	No description provided.

## `DestinationSpec` **Appears in:** * [Destination](#odigos-io-v1alpha1-Destination)

DestinationSpec defines the desired state of Destination

Field	Type	Description
`type` \[Required]	`common.DestinationType`	No description provided.
`destinationName` \[Required]	`string`	No description provided.
`data` \[Required]	`map\[string]string`	No description provided.
`secretRef` \[Required]	`core/v1.LocalObjectReference`	No description provided.
`signals` \[Required]	`\[]common.ObservabilitySignal`	No description provided.
`disabled` \[Required]	`bool`	No description provided.
`metricsSettings`	`DestinationMetricsSettings`	MetricsSettings defines the metrics settings for this destination. ignored if destination does not support metrics or metrics are not enabled in signals
`sourceSelector`	`SourceSelector`	SourceSelector defines which sources can send data to this destination. If not specified, defaults to "all".

## `DestinationStatus` **Appears in:** * [Destination](#odigos-io-v1alpha1-Destination)

DestinationStatus defines the observed state of Destination

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a destination's current state.

## `EnvVar` **Appears in:** * [RuntimeDetailsByContainer](#odigos-io-v1alpha1-RuntimeDetailsByContainer)

Field	Type	Description
`name` \[Required]	`string`	No description provided.
`value` \[Required]	`string`	No description provided.

## `HighlyRelevantOperation` **Appears in:** * [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

define operations (spans) with high observability value. if found anywhere in the trace, the entire trace will be kept regaradless of any cost reduction rules.

Field	Type	Description
`name` \[Required]	`string`	user provided name, for easier identification and reference. use short and descriptive name, like "health check", "always keep /transaction in payment service", etc. odigos does not use or assume any meaning from this field, but it is written as metric attribute, and stored as span attribute on participating spans.
`disabled` \[Required]	`bool`	if set to true, the rule will be disabled, e.g. will not be taken into account for any sampling decisions. disabled rules still participate in metrics calculations, allowing enhanced tools and data for troubleshooting and sampling maintenance.
`sourceScopes` \[Required]	`k8sconsts.SourcesScopes`	limit the operation to specific sources. an empty list will match any source. if multiple items are set, the operation match if any one matches this relates to the "ResourceAttributes" part of a span.
`error` \[Required]	`bool`	if "Error" is set to true, only spans with SpanStatus set to "Error" are considered
`durationAtLeastMs` \[Required]	`int`	if Duration is set, only operations with duration in milli seconds larger then this value are considered
`operation` \[Required]	`common/api/sampling.TailSamplingOperationMatcher`	optionally, limit this rule to specific operations. for example: specific endpoint or kafka topic. this field is optional, and if not set, the rule will be applied to all operations.
`percentageAtLeast` \[Required]	`float64`	traces that contains this operation will be sampled by at least this percentage. if unset, 100% of such the traces will be sampled.
`notes` \[Required]	`string`	optional free-form text field that allows you to attach notes for future context and maintenance. users can write why this rule was added, observations, document considerations, etc.

## `InstrumentationConfigSpec` **Appears in:** * [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)

Config for the OpenTelemeetry SDKs that should be applied to a workload. The workload is identified by the owner reference

Field	Type	Description
`serviceName` \[Required]	`string`	the service.name property is used to populate the `service.name` resource attribute in the telemetry generated by this workload
`agentInjectionEnabled` \[Required]	`bool`	determines if odigos should inject agents to pods of this workload.
`podManifestInjectionOptional` \[Required]	`bool`	true if all containers with agent injection enabled in this source can be enabled without pod manifest changes. if false, at least one container requires pod manifest changes to enable agent injection.
`containers` \[Required]	`\[]ContainerAgentConfig`	configuration for each instrumented container in the workload
`workloadCollectorConfig` \[Required]	`\[]common/api.ContainerCollectorConfig`	No description provided.
`containersOverrides` \[Required]	`\[]ContainerOverride`	will always list all containers of this workload by name, and override data in case it is configured on the source. this peoperty can be used to know all container names in the workload, even if other controllers did not yet run.
`containerOverridesHash` \[Required]	`string`	An hash of the containers overrides, used to determine if the overrides have changed for event filtering. this is updated only when the overrides are changed, and not when some other change in the source occurs.
`agentsMetaHash` \[Required]	`string`	this hash is used to determine the deployment of the agents. e.g. when the distro for container changes, or it's compatibility version, or something else that requires rollout, the hash change will indicate that. if the hash is empty, it means that no agent should be enabled in any pod container.
`agentsMetaHashChangedTime` \[Required]	`meta/v1.Time`	The last time at which the agents meta hash value was changed. Pods created before this time may not be in alignment with the AgentsMetaHash. e.g. can lack the odigos label, or have a different value.

## `InstrumentationConfigStatus` **Appears in:** * [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)

Field	Type	Description
`runtimeDetailsByContainer` \[Required]	`\[]RuntimeDetailsByContainer`	Capture Runtime Details for the workloads that this CR applies to.
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a InstrumentationConfig's current state.
`workloadRolloutHash` \[Required]	`string`	This hash is recorded only after the rollout took place. it allows us to determine if the workload needs to be rollout based on previous rollout and the current config. if this field is different than the spec.AgentsDeploymentHash it means rollout is needed or not yet updated.
`rollbackOccurred` \[Required]	`bool`	Check if rollback happened to an application
`instrumentationTime` \[Required]	`meta/v1.Time`	This time recorded only after the rollout took place. This allows us to determine whether a crashing application should be rolled back or not
`podsManifestInjectionStatus` \[Required]	`PodsManifestInjectionStatus`	Represents the status of odigos MANIFEST injection to existing pods template.

## `InstrumentationInstanceSpec` **Appears in:** * [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)

Field	Type	Description
`containerName` \[Required]	`string`	stores the name of the container in the pod where the SDK is running. The pod details can be found as the owner reference on the CR.

## `InstrumentationInstanceStatus` **Appears in:** * [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)

InstrumentationInstanceStatus defines the observed state of InstrumentationInstance If the instrumentation is not active, this CR should be deleted

Field	Type	Description
`identifyingAttributes` \[Required]	`\[]Attribute`	Attributes that identify the SDK and are reported as resource attributes in the generated telemetry. One can identify if an arbitrary telemetry is generated by this SDK by checking those resource attributes.
`nonIdentifyingAttributes` \[Required]	`\[]Attribute`	Attributes that are not reported as resource attributes but useful to describe characteristics of the SDK.
`healthy` \[Required]	`bool`	Healthy true means that the odigos agent has started the SDK, and there are no errors. User can expect telemetry to be generated. Healthy false means that the agent has stopped and telemetry data is not expected to be generated. Healthy nil means that the agent did not report any health status yet (prefer to always report health status).
`message` \[Required]	`string`	message is a human readable message indicating details about the SDK general health. can be omitted if healthy is true
`reason` \[Required]	`string`	reason contains a programmatic identifier indicating the reason for the component status. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API.
`lastStatusTime` \[Required]	`meta/v1.Time`	No description provided.
`components` \[Required]	`\[]InstrumentationLibraryStatus`	No description provided.

## `InstrumentationLibraryConfigTraces`

Field	Type	Description
`enabled` \[Required]	`bool`	Whether the instrumentation library is enabled to record traces. When false, it is expected that the instrumentation library does not produce any spans regardless of any other configuration. When true, the instrumentation library should produce spans according to the other configuration options. If not specified, the default value for this signal should be used (whether to enable libraries by default or not).

## `InstrumentationLibraryId`

Field Type Description

libraryName \[Required]

string

The name of the instrumentation library

Node.js: The name of the npm package: @opentelemetry/instrumentation-\

spanKind \[Required]

common.SpanKind

SpanKind is only supported by Golang and will be ignored for any other SDK language. In Go, SpanKind is used because the same instrumentation library can be utilized for different span kinds (e.g., client/server).

## `InstrumentationLibraryStatus` **Appears in:** * [InstrumentationInstanceStatus](#odigos-io-v1alpha1-InstrumentationInstanceStatus)

InstrumentationLibraryStatus defines the observed state of an InstrumentationLibrary. if a library is not active/disable, it should not be included in the status

Field	Type	Description
`name` \[Required]	`string`	for example ("net/http", "@opentelemetry/instrumentation-redis")
`type` \[Required]	`InstrumentationLibraryType`	No description provided.
`identifyingAttributes` \[Required]	`\[]Attribute`	Attributes that identify the component. The combination of (Name, Type, IdentifyingAttributes) must be unique.
`nonIdentifyingAttributes` \[Required]	`\[]Attribute`	Attributes that do not necessarily identify the component but help describe its characteristics.
`healthy` \[Required]	`bool`	No description provided.
`message` \[Required]	`string`	message is a human readable message indicating details about the component health. can be omitted if healthy is true
`reason` \[Required]	`string`	reason contains a programmatic identifier indicating the reason for the SDK status. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API.
`lastStatusTime` \[Required]	`meta/v1.Time`	No description provided.

## `InstrumentationLibraryType` (Alias of `string`) **Appears in:** * [InstrumentationLibraryStatus](#odigos-io-v1alpha1-InstrumentationLibraryStatus) ## `InstrumentationRuleSpec` **Appears in:** * [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)

Field	Type	Description
`ruleName` \[Required]	`string`	Allows you to attach a meaningful name to the rule for convenience. Odigos does not use or assume any meaning from this field.
`notes` \[Required]	`string`	A free-form text field that allows you to attach notes regarding the rule for convenience. For example: why it was added. Odigos does not use or assume any meaning from this field.
`disabled` \[Required]	`bool`	A boolean field allowing to temporarily disable the rule, but keep it around for future use
`scopes` \[Required]	`k8sconsts.SourcesScopes`	SourcesScopes lists SourcesScope entries to which the rule should be applied. If unset or empty, the rule applies to all workloads.
`instrumentationLibraries` \[Required]	`\[]github.com/odigos-io/odigos/api/odigos/v1alpha1.InstrumentationLibraryGlobalId`	For fine grained control, the user can specify the instrumentation library to use. One can specify same rule for multiple languages and libraries at the same time. If nil, all instrumentation libraries will be used. If empty, no instrumentation libraries will be used.
`payloadCollection` \[Required]	`common/api/instrumentationrules.PayloadCollection`	Allows to configure payload collection aspects for different types of payloads.
`otelDistros` \[Required]	`common/api/instrumentationrules.OtelDistros`	Set the otel distros to use instead of the defaults.
`codeAttributes` \[Required]	`common/api/instrumentationrules.CodeAttributes`	Configure which code attributes should be recorded as span attributes.
`headersCollection` \[Required]	`common/api/instrumentationrules.HttpHeadersCollection`	Allows to configure the collection of http headers for different types of payloads.
`traceConfig` \[Required]	`common/api/instrumentationrules.TraceConfig`	Configure the tracing configuration for the library.
`customInstrumentations` \[Required]	`common/api/instrumentationrules.CustomInstrumentations`	Add custom instrumentation probes
`headSamplingFallbackFraction` \[Required]	`common/api/instrumentationrules.HeadSamplingFallbackFraction`	Configure general fraction of traces to record if none of the rules evaluate to true. this can help in reducing very noisy and heavy traffic services. note that traces will be dropped regardless of thier attributes/errors/importance. @deprecated: use odigos config to set this value instead.
`ebpfLogCapture` \[Required]	`common/api/instrumentationrules.EbpfLogCapture`	Configure eBPF-based log capture. When enabled, the node collector logs pipeline will use only the eBPF receiver instead of the filelog receiver, and odiglet will register instrumented processes for eBPF log capture.
`networkMetrics` \[Required]	`common/api/instrumentationrules.NetworkMetricsConfig`	Configure network flow and TCP stats metrics for scoped workloads.
`traceVerbosity` \[Required]	`common/api/instrumentationrules.TraceVerbosity`	Configure the verbosity of the traces for the library.
`agentDiagnostics` \[Required]	`common/api/instrumentationrules.AgentDiagnostics`	Configure the agent own logging configuration.

## `InstrumentationRuleStatus` **Appears in:** * [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a instrumentationrule's current state. Known .status.conditions.type are: "Available", "Progressing"

## `NoisyOperation` **Appears in:** * [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

endpoints (or other operations) which are considered "noise", and provide no or very little observability value. these traces should not be collected at all, or dropped aggresevly. motivation is data sentization and performance improvment (even if cost is not a factor)

examples:

health-checks (readiness and liveness probes)
metrics scrape endpoints (prometheus /metrics endpoint)
other agents calling home (outgoing http requests to collector.my.vendor.com)

Field	Type	Description
`name` \[Required]	`string`	user provided name, for easier identification and reference. use short and descriptive name, like "health check", "always keep /transaction in payment service", etc. odigos does not use or assume any meaning from this field, but it is written as metric attribute, and stored as span attribute on participating spans.
`disabled` \[Required]	`bool`	if set to true, the rule will be disabled, e.g. will not be taken into account for any sampling decisions. disabled rules still participate in metrics calculations, allowing enhanced tools and data for troubleshooting and sampling maintenance.
`sourceScopes` \[Required]	`k8sconsts.SourcesScopes`	limit this rule to specific sources (by name, namespace, language, etc.) for example: if "other agent" rule for noisty operation is relevant only in java, limit this rule by setting source scope to java and prevent other languages from being affected if the list is empty - all sources are matched.
`operation` \[Required]	`common/api/sampling.HeadSamplingOperationMatcher`	limit this rule to specific operations. for example: specific http server endpoint (GET "/healthz" as an example). this field is optional, and if not set, the rule will be applied to all operations.
`percentageAtMost` \[Required]	`float64`	sampling percentage for noisy operations. if unset, 0% of such the traces will be collected. this percentage has "at most" semantics - the final sampling percentage for traces that match this rule will be the highest possible, but at most this value.
`notes` \[Required]	`string`	optional free-form text field that allows you to attach notes for future context and maintenance. users can write why this rule was added, observations, document considerations, etc.

## `OdigosOwnMetricsSettings` **Appears in:** * [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

configuration for collecting and exporting odigos own metrics. e.g. metrics about odigos components, and not about the user's application.

Field	Type	Description
`sendToMetricsDestinations` \[Required]	`bool`	if true, odigos will send all the metrics it collects about itself to the metrics pipeline, which will make them available to the metrics destinations. users can troubleshoot odigos itself by monitoring these metrics in their existing systems, and create their own dashboards, alerting, and more.
`sendToOdigosMetricsStore` \[Required]	`bool`	if true, odigos will send all the metrics it collects about itself to the odigos metrics store, which is available to odigos UI. it can help in presenting a consistent view of odigos itself, without relying on user system and integrations.
`interval` \[Required]	`string`	time interval for flusing metrics (format: 15s, 1m etc). defaults: 10s

## `OtherAgent` **Appears in:** * [RuntimeDetailsByContainer](#odigos-io-v1alpha1-RuntimeDetailsByContainer)

Field	Type	Description
`name` \[Required]	`string`	No description provided.

## `PodsManifestInjectionStatus` **Appears in:** * [InstrumentationConfigStatus](#odigos-io-v1alpha1-InstrumentationConfigStatus)

represents the status of odigos MANIFEST injection to existing pods template. each pod can be in one of 3 states: injected and up-to-date, injected and out-of-date, or not injected. actual agent in pod container can be injected or not regardless of the pod manifest injection status. status only shows if pods exists from each category and not count, to avoid frequent (noisy/expensive) updates in steady state

Field	Type	Description
`hasInjectedUpToDatePods` \[Required]	`bool`	if the source has any pods which are using the latest agent deployment setup.
`hasInjectedOutOfDatePods` \[Required]	`bool`	if the source has any pods which are using an outdated agent deployment setup.
`hasUninjectedPods` \[Required]	`bool`	if the source has any pods which do not have the odigos agent injected.

## `ProcessorSpec` **Appears in:** * [Processor](#odigos-io-v1alpha1-Processor)

ProcessorSpec defines the an OpenTelemetry Collector processor in odigos telemetry pipeline

Field	Type	Description
`type` \[Required]	`string`	type of the processor (batch, attributes, etc). this field is only the type, not it's instance name in the collector configuration yaml
`processorName` \[Required]	`string`	this name is solely for the user convenience, to attach a meaningful name to the processor. odigos must not assume any semantics from this name. odigos cannot assume this name is unique, not empty, exclude spaces or dots, limited in length, etc.
`notes` \[Required]	`string`	user can attach notes to the processor, to document its purpose, usage, etc.
`disabled` \[Required]	`bool`	disable is a flag to enable or disable the processor. if the processor is disabled, it will not be included in the collector configuration yaml. this allows the user to keep the processor configuration in the CR, but disable it temporarily.
`signals` \[Required]	`\[]common.ObservabilitySignal`	signals can be used to control which observability signals are processed by the processor.
`collectorRoles` \[Required]	`\[]CollectorsGroupRole`	control which collector roles in odigos pipeline this processor is attached to.
`orderHint` \[Required]	`int`	control the order of processors. a processor with lower order hint value will be placed before other processors with higher value. if 2 processors have the same value, the order is arbitrary. if the value is missing (or 0) the processor can be placed anywhere in the pipeline
`processorConfig` \[Required]	`k8s.io/apimachinery/pkg/runtime.RawExtension`	this it the configuration of the opentelemetry collector processor component with the type specified in 'type'.

## `ProcessorStatus` **Appears in:** * [Processor](#odigos-io-v1alpha1-Processor)

ProcessorStatus defines the observed state of the processor

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a processor's current state.

## `RecommendationSpec` **Appears in:** * [Recommendation](#odigos-io-v1alpha1-Recommendation)

RecommendationSpec defines the desired state of Recommendation

Field	Type	Description
`type` \[Required]	`common.RecommendationType`	Type identifies which recommendation from the catalog this resource enables.
`applied`	`bool`	Applied indicates whether this recommendation is currently applied (via the recommendation flow or configured organically).
`conditionsMet`	`bool`	ConditionsMet indicates whether the catalog prerequisites for this recommendation currently hold (for example GoEnterpriseSources for AutoGoOffsetUpdater).

## `RecommendationStatus` **Appears in:** * [Recommendation](#odigos-io-v1alpha1-Recommendation)

RecommendationStatus defines the observed state of Recommendation

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a recommendation's current state.

## `RuntimeDetailsByContainer` **Appears in:** * [ContainerOverride](#odigos-io-v1alpha1-ContainerOverride) * [InstrumentationConfigStatus](#odigos-io-v1alpha1-InstrumentationConfigStatus)

Field	Type	Description
`containerName` \[Required]	`string`	No description provided.
`language` \[Required]	`common.ProgrammingLanguage`	No description provided.
`runtimeVersion` \[Required]	`string`	No description provided.
`envVars` \[Required]	`\[]EnvVar`	No description provided.
`otherAgents` \[Required]	`\[]OtherAgent`	OtherAgents lists other instrumentation agents detected in the container (a process can carry more than one); empty when none.
`libCType` \[Required]	`common.LibCType`	No description provided.
`secureExecutionMode` \[Required]	`bool`	Indicates whether the target process is running is secure-execution mode. nil means we were unable to determine the secure-execution mode.
`criErrorMessage` \[Required]	`string`	CriErrorMessage is set if the value in EnvFromContainerRuntime was not computed correctly and cannot be used safely. Sometimes, even if CRI check failed, it is possible to tell that relevant env vars are not coming from container runtime. Thus, this field is set only when there is: Actual CRI check failed The observed environment variables might come from container runtime
`envFromContainerRuntime` \[Required]	`\[]EnvVar`	Holds the environment variables retrieved from the container runtime.

## `SamplingSpec` **Appears in:** * [Sampling](#odigos-io-v1alpha1-Sampling)

define sampling rules. the rules can be defined as one or multiple objects in kubernetes, and are all joined together to form the global sampling rules. odigos users can group rules based on whatever criteria that makes sense for them, for example - by team, by client, by usecase, admin-policy, etc.

Field	Type	Description
`name` \[Required]	`string`	give these sampling rules a name for display, easier identification and reference.
`notes` \[Required]	`string`	a free-form text field that allows you to attach notes regardinag the rule for convenience. Odigos does not use or assume any meaning from this field.
`noisyOperations` \[Required]	`\[]NoisyOperation`	No description provided.
`highlyRelevantOperations` \[Required]	`\[]HighlyRelevantOperation`	No description provided.
`costReductionRules` \[Required]	`\[]CostReductionRule`	No description provided.

## `SamplingStatus` **Appears in:** * [Sampling](#odigos-io-v1alpha1-Sampling)

SamplingStatus defines the observed state of Sampling.

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a Sampling's current state. Known .status.conditions.type are: "Available", "Progressing"

## `ServiceGraphSettings` **Appears in:** * [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings) ## `SourceSelector` **Appears in:** * [DestinationSpec](#odigos-io-v1alpha1-DestinationSpec)

Field	Type	Description
`namespaces`	`\[]string`	If a namespace is specified, all workloads (sources) within that namespace are allowed to send data. Example: namespaces: \["default", "production"] This means the destination will receive data from all sources in "default" and "production" namespaces.
`dataStreams`	`\[]string`	Workloads (sources) are assigned to Datastreams via labels (odigos.io/data-stream: true), allowing a more flexible selection mechanism. Example: dataStreams: \["backend", "monitoring"] This means the destination will receive data only from sources labeled with "backend" or "monitoring".

## `SourceSpec` **Appears in:** * [Source](#odigos-io-v1alpha1-Source)

Field	Type	Description
`workload` \[Required]	`k8sconsts.PodWorkload`	Workload represents the workload or namespace to be instrumented. This field is required upon creation and cannot be modified.
`disableInstrumentation` \[Required]	`bool`	DisableInstrumentation excludes this workload from auto-instrumentation.
`otelServiceName`	`string`	OtelServiceName determines the "service.name" resource attribute which will be reported by the instrumentations of this source. If not set, the workload name will be used. It is not valid for namespace sources.
`containerOverrides`	`\[]ContainerOverride`	Specify specific override values for containers in a workload source. Not valid for namespace sources. Can be used to set the runtime info in case the automatic detection fails or produce wrong results. Containers are identified by their names. All containers not listed will retain their default behavior.
`matchWorkloadNameAsRegex`	`bool`	MatchWorkloadNameAsRegex indicates that the workload name should be matched using regex. When true, spec.workload.name is treated as a regular expression pattern. This allows matching multiple workloads with a single Source CRD. Not valid for namespace sources.

## `SourceStatus` **Appears in:** * [Source](#odigos-io-v1alpha1-Source)

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Represents the observations of a source's current state. Known .status.conditions.type are: "Available", "Progressing"

# operator.odigos.io/v1alpha1 Source: https://docs.odigos.io/oss/api-reference/operator.odigos.io.v1alpha1 ## Resource Types ## `Odigos`

Odigos is the Schema for the odigos API

Field	Type	Description
`metadata` \[Required]	`meta/v1.ObjectMeta`	No description provided. Refer to the Kubernetes API documentation for the fields of the `metadata` field.
`spec` \[Required]	`OdigosSpec`	No description provided.
`status` \[Required]	`OdigosStatus`	No description provided.

## `OdigosSpec` **Appears in:** * [Odigos](#Odigos)

OdigosSpec defines the desired state of Odigos

Field	Type	Description
`onPremToken` \[Required]	`string`	(Optional) OnPremToken is an enterprise token for Odigos Enterprise.
`uiMode` \[Required]	`common.UiMode`	(Optional) UIMode sets the UI mode to either "normal" or "readonly". In "normal" mode the UI is fully interactive, allowing users to view and edit Odigos configuration and settings. In "readonly" mode, the UI can only be used to view current Odigos configuration and is not interactive. Default=default
`telemetryEnabled` \[Required]	`bool`	(Optional) TelemetryEnabled records general telemetry regarding Odigos usage. Default=false
`ignoredNamespaces` \[Required]	`\[]string`	(Optional) IgnoredNamespaces is a list of namespaces to not show in the Odigos UI.
`ignoredContainers` \[Required]	`\[]string`	(Optional) IgnoredContainers is a list of container names to exclude from instrumentation (useful for ignoring sidecars).
`profiles` \[Required]	`\[]common.ProfileName`	(Optional) Profiles is a list of preset profiles with a specific configuration.
`agentEnvVarsInjectionMethod` \[Required]	`common.EnvInjectionMethod`	(Optional) AgentEnvVarsInjectionMethod is the method to inject the Odigos agent env vars into the pod. Default=pod-manifest
`nodeSelector` \[Required]	`map\[string]string`	(Optional) NodeSelector is a map of key-value Kubernetes NodeSelector labels to apply to all Odigos components. Note that Odigos will only be able to instrument applications on the same node.
`skipWebhookIssuerCreation` \[Required]	`bool`	(Optional) SkipWebhookIssuerCreation skips creating the Issuer and Certificate for the Instrumentor pod webhook if cert-manager is installed. Default=false
`podSecurityPolicy` \[Required]	`bool`	(Optional) PodSecurityPolicy allows Odigos pods to use a privileged pod security policy. Default=false
`imagePrefix` \[Required]	`string`	(Optional) ImagePrefix is a prefix for all container images. This should only be used if you are pulling Odigos images from the non-default registry. Default (community): docker.io/keyval Default (enterprise): registry.odigos.io Default (OpenShift): registry.connect.redhat.com/odigos
`logLevel` \[Required]	`string`	(Optional) LogLevel sets the default log level for all Odigos components (e.g. instrumentor, odiglet, collector). Accepted values: debug, info, warn, error. Default=info when unset.
`mountMethod` \[Required]	`common.MountMethod`	(Optional) MountMethod optionally defines the mechanism for mounting Odigos files into instrumented pods. One of "k8s-virtual-device" (default) or "k8s-host-path".
`openshiftEnabled` \[Required]	`bool`	(Optional) OpenShiftEnabled configures selinux on OpenShift nodes. DEPRECATED: OpenShift clusters are auto-detected and this API field will be removed in a future release.

## `OdigosStatus` **Appears in:** * [Odigos](#Odigos)

OdigosStatus defines the observed state of Odigos

Field	Type	Description
`conditions` \[Required]	`\[]meta/v1.Condition`	Conditions store the status conditions of the Odigos instances

# Overview Source: https://docs.odigos.io/oss/backends-overview Odigos makes it simple to add and configure destinations, allowing you to select the specific signals (`traces`,`metrics`,`logs`) that you want to send to each destination. Can't find your backend in Odigos? Please tell us! We are constantly adding new integrations.
You can also follow our quick [add new destination](/adding-new-dest) guide and submit a PR. | | | | Traces | Metrics | Logs | Profiles | | - | - | - | :-: | :-: | :-: | :-: | |

| [AppDynamics](/backends/appdynamics) | Managed | ✅ | ✅ | ✅ | | |

| [AWS CloudWatch](/backends/awscloudwatch) | Managed | | ✅ | ✅ | | |

| [AWS S3](/backends/awss3) | Managed | ✅ | ✅ | ✅ | | | xray

| [Axiom](/backends/axiom) | Managed | ✅ | | ✅ | | |

| [Azure Blob Storage](/backends/azureblob) | Managed | ✅ | | ✅ | | |

| [Azure Monitor](/backends/azuremonitor) | Managed | ✅ | ✅ | ✅ | | |

| [Chronosphere](/backends/chronosphere) | Managed | ✅ | ✅ | | | |

| [Clickhouse](/backends/clickhouse) | Self-Hosted | ✅ | ✅ | ✅ | | |

| [Coralogix](/backends/coralogix) | Managed | ✅ | ✅ | ✅ | | | dash0

| [Dash0](/backends/dash0) | Managed | ✅ | ✅ | ✅ | | | datadog

| [Datadog](/backends/datadog) | Managed | ✅ | ✅ | ✅ | | | dynamic

| [Dynamic Destination](/backends/dynamic) | Self-Hosted | ✅ | ✅ | ✅ | ✅ | |

| [Dynatrace](/backends/dynatrace) | Managed | ✅ | ✅ | ✅ | | |

| [Elastic APM](/backends/elasticapm) | Managed | ✅ | ✅ | ✅ | | |

| [Gigapipe](/backends/gigapipe) | Managed | ✅ | ✅ | ✅ | | |

| [Google Cloud Monitoring](/backends/googlecloudmonitoring) | Managed | ✅ | | ✅ | | |

| [Groundcover inCloud](/backends/groundcover) | Managed | ✅ | ✅ | ✅ | | |

| [Honeycomb](/backends/honeycomb) | Managed | ✅ | ✅ | ✅ | | | hyperdx

| [HyperDX](/backends/hyperdx) | Managed | ✅ | ✅ | ✅ | | | instana

| [IBM Instana](/backends/instana) | Managed | ✅ | ✅ | ✅ | | | jaeger

| [Kafka](/backends/kafka) | Self-Hosted | ✅ | ✅ | ✅ | | |

| [KloudMate](/backends/kloudmate) | Managed | ✅ | ✅ | ✅ | | | last9

| [Last9](/backends/last9) | Managed | ✅ | ✅ | ✅ | | |

| [Logz.io](/backends/logzio) | Managed | ✅ | ✅ | ✅ | | | loki

| [Lumigo](/backends/lumigo) | Managed | ✅ | ✅ | ✅ | | |

| [Middleware](/backends/middleware) | Managed | ✅ | ✅ | ✅ | | |

| [New Relic](/backends/newrelic) | Managed | ✅ | ✅ | ✅ | | | observe

| [Observe](/backends/observe) | Managed | ✅ | ✅ | ✅ | | |

| [OneUptime](/backends/oneuptime) | Managed | ✅ | ✅ | ✅ | | |

| [OTLP gRPC](/backends/otlp) | Self-Hosted | ✅ | ✅ | ✅ | ✅ | |

| [OTLP http](/backends/otlphttp) | Self-Hosted | ✅ | ✅ | ✅ | ✅ | |

| [qryn](/backends/qryn) | Self-Hosted | ✅ | ✅ | ✅ | | |

| [Quickwit](/backends/quickwit) | Self-Hosted | ✅ | | ✅ | | | seq

| [Seq](/backends/seq) | Self-Hosted | ✅ | | ✅ | | |

| [SigNoz](/backends/signoz) | Self-Hosted | ✅ | ✅ | ✅ | | | splunk

| [Sumo Logic](/backends/sumologic) | Managed | ✅ | ✅ | ✅ | | |

| [TelemetryHub](/backends/telemetryhub) | Managed | ✅ | ✅ | ✅ | | | tempo

| [Tingyun 基调听云](/backends/tingyun) | Managed | ✅ | ✅ | | | |

| [Uptrace](/backends/uptrace) | Managed | ✅ | ✅ | ✅ | | |

| [VictoriaMetrics Cloud](/backends/victoriametricscloud) | Managed | | ✅ | | | # Alibaba Cloud Source: https://docs.odigos.io/oss/backends/alibabacloud Configuring the Alibaba Cloud backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **ALIBABA\_ENDPOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is required. Do not add the prefix `http://`. * This field is required * **ALIBABA\_TOKEN** `string` : Token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Alibaba Cloud` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `alibabacloud.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: alibabacloud-example namespace: odigos-system spec: data: ALIBABA_ENDPOINT: destinationName: alibabacloud secretRef: name: alibabacloud-secret signals: - TRACES type: alibabacloud --- apiVersion: v1 data: ALIBABA_TOKEN: kind: Secret metadata: name: alibabacloud-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f alibabacloud.yaml ``` # AppDynamics Source: https://docs.odigos.io/oss/backends/appdynamics Configuring the AppDynamics backend (Managed) ### Getting Started

1. [Register](https://accounts.appdynamics.com/trial)/[Login](https://login.appdynamics.com/sso/authenticate) to AppDynamics. 2. Navigate to the OpenTelemetry Configuration page 3. Click on `Processors`, you'll find 3 attributes, locate `appdynamics.controller.account` and copy its value, this is your `Account Name`. 4. Click on `Exporters`, copy the `Endpoint`, and generate an `API Key`. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **APPDYNAMICS\_APPLICATION\_NAME** `string` : Application Name. Will define a namespace in AppDynamics * This field is optional * **APPDYNAMICS\_ACCOUNT\_NAME** `string` : Account Name. * This field is required * **APPDYNAMICS\_ENDPOINT\_URL** `string` : Endpoint URL. * This field is required * **APPDYNAMICS\_API\_KEY** `string` : API Key. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `AppDynamics` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `appdynamics.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: appdynamics-example namespace: odigos-system spec: data: APPDYNAMICS_ACCOUNT_NAME: APPDYNAMICS_ENDPOINT_URL: # Note: The commented fields below are optional. # APPDYNAMICS_APPLICATION_NAME: destinationName: appdynamics secretRef: name: appdynamics-secret signals: - TRACES - METRICS - LOGS type: appdynamics --- apiVersion: v1 data: APPDYNAMICS_API_KEY: kind: Secret metadata: name: appdynamics-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f appdynamics.yaml ``` # AWS CloudWatch Source: https://docs.odigos.io/oss/backends/awscloudwatch Configuring the AWS CloudWatch backend (Managed) ### Getting Started

Make sure that you have the necessary `AWS Credentials` and permissions set up.
AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials). **Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)** The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services. Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables). ```bash theme={null} eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve ``` **Setting up the IAM policy** ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "PushLogs", "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:::log-group::log-stream:" }, { "Sid": "CreateLogGroup", "Effect": "Allow", "Action": "logs:CreateLogGroup", "Resource": "arn:aws:logs:::log-group:" }, { "Sid": "DescribeLogGroups", "Effect": "Allow", "Action": "logs:DescribeLogGroups", "Resource": "arn:aws:logs:::log-group:" } ] } ``` ### Configuring Destination Fields ❌ Traces ✅ Metrics ✅ Logs ❌ Profiles * **AWS\_CLOUDWATCH\_LOG\_GROUP\_NAME** `string` : Log Group Name. The group name of the CloudWatch Logs. If it does not exist it will be created automatically. * This field is required * **AWS\_CLOUDWATCH\_LOG\_STREAM\_NAME** `string` : Log Stream Name. The stream name of the CloudWatch Logs. If it does not exist it will be created automatically. * This field is required * **AWS\_CLOUDWATCH\_REGION** `string` : Region. The AWS region where the log stream is in. Region must be specified if it is not already set in the default credential chain. * This field is optional * **AWS\_CLOUDWATCH\_ENDPOINT** `string` : Endpoint. The CloudWatch Logs service endpoint which the requests are forwarded to. * This field is optional * **AWS\_CLOUDWATCH\_LOG\_RETENTION** `string` : Log Retention. LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. The value is in days, 0 = Never Expire. This value can also be changed later-on via the AWS console (UI) or AWS API, it's not a permanent decision once set. Also, the value here might not reflect the actual retention value in the destination if it was modified or some automatic policy is applied. * This field is optional and defaults to `0` * **AWS\_CLOUDWATCH\_TAGS** `{ key: string; value: string; }[]` : Tags. Tags is the option to set tags for the CloudWatch Log Group. If specified, please add at most 50 tags. Keys must be between 1-128 characters and follow the pattern: (alphanumerics, whitespace, and \_.:/=+-!). Values must be between 1-256 characters and follow the pattern: (alphanumerics, whitespace, and \_.:/=+-!). * This field is optional * **AWS\_CLOUDWATCH\_RAW\_LOG** `boolean` : Raw Log. If set to true, only the log message will be exported to CloudWatch Logs. This needs to be set to true for EMF logs. * This field is optional and defaults to `False` * **AWS\_CLOUDWATCH\_METRICS\_NAMESPACE** `string` : Metrics Namespace. Customized CloudWatch metrics namespace. Metrics in different namespaces are isolated from each other. Please note this is not a cluster namespace. * This field is optional and defaults to `Odigos` * **AWS\_CLOUDWATCH\_METRICS\_DIMENSION\_ROLLUP** `string` : Metrics Dimension Rollup. The option for metrics dimension rollup. * This field is optional and defaults to `NoDimensionRollup` * **AWS\_CLOUDWATCH\_METRICS\_DETAILED** `boolean` : Detailed Metrics. Retain detailed datapoint values in exported metrics (e.g instead of exporting a quantile as a statistical value, preserve the quantile's population). * This field is optional and defaults to `False` * **AWS\_CLOUDWATCH\_RETAIN\_INITIAL\_VALUE\_OF\_DELTA\_METRIC** `boolean` : Retain Initial Value of Delta Metric. This option specifies how the first value of a metric is handled. AWS EMF expects metric values to only contain deltas to the previous value. In the default case the first received value is therefor not sent to AWS but only used as a baseline for follow up changes to this metric. This is fine for high throughput metrics with stable labels. In this case it does not matter if the first value of this metric is discarded. However when your metric describes infrequent events or events with high label cardinality, then the exporter in default configuration would still drop the first occurrence of this metric. With this configuration value set to true the first value of all metrics will instead be send to AWS. * This field is optional and defaults to `False` - If the log group and/or log stream are specified in an EMF log, that EMF log will be exported to that log group and/or log stream (i.e. ignores the log group and log stream defined in the configuration). * The log group and log stream will also be created automatically if they do not already exist. * Example of an EMF log with log group and log stream: ```json theme={null} { "_aws": { "Timestamp": 1574109732004, "LogGroupName": "Foo", "LogStreamName": "Bar", "CloudWatchMetrics": [ { "Namespace": "MyApp", "Dimensions": [ [ "Operation" ] ], "Metrics": [ { "Name": "ProcessingLatency", "Unit": "Milliseconds", "StorageResolution": 60 } ] } ] }, "Operation": "Aggregator", "ProcessingLatency": 100 } ``` - Resource ARNs (Amazon Resource Name (ARN) of the AWS resource running the collector) are currently not supported with the CloudWatch Logs Exporter. ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `AWS CloudWatch` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `cloudwatch.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: cloudwatch-example namespace: odigos-system spec: data: AWS_CLOUDWATCH_LOG_GROUP_NAME: AWS_CLOUDWATCH_LOG_STREAM_NAME: # Note: The commented fields below are optional. # AWS_CLOUDWATCH_REGION: # AWS_CLOUDWATCH_ENDPOINT: # AWS_CLOUDWATCH_LOG_RETENTION: # AWS_CLOUDWATCH_TAGS: # AWS_CLOUDWATCH_RAW_LOG: # AWS_CLOUDWATCH_METRICS_NAMESPACE: # AWS_CLOUDWATCH_METRICS_DIMENSION_ROLLUP: # AWS_CLOUDWATCH_METRICS_DETAILED: # AWS_CLOUDWATCH_RETAIN_INITIAL_VALUE_OF_DELTA_METRIC: destinationName: cloudwatch signals: - METRICS - LOGS type: cloudwatch ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f cloudwatch.yaml ``` # AWS S3 Source: https://docs.odigos.io/oss/backends/awss3 Configuring the AWS S3 backend (Managed) ### Getting Started

**Setting up AWS S3 Bucket** If you haven't already, you need to create an S3 bucket to store your data. You can do it using the [AWS Management Console](https://console.aws.amazon.com/s3/home) or the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html). For example, to create a bucket named `otel-data` in region `us-east-1` using the AWS CLI, you can run the following command: ```bash theme={null} aws s3api create-bucket --bucket otel-data --region us-east-1 ``` Make sure that you have the necessary `AWS Credentials` and permissions set up.
AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials). **Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)** The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services. Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables). ```bash theme={null} eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve ``` **Setting up the IAM policy** ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::", "arn:aws:s3:::/*"] } ] } ``` You can do it with the aws cli. * Replace `otel-data` with the name of your bucket if you used a different name. * The policy name is `odigos-s3` in this example, you can use it or choose a different name. Take note of the policy ARN, you will need it in the next step. ```bash theme={null} aws iam create-policy --policy-name odigos-s3 --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::otel-data", "arn:aws:s3:::otel-data/*"] } ] }' ``` Or use the aws management console to create the policy. The last step is to configure this IAM policy to be assumed by the EKS cluster. Use the following cli command as template and adjust it according to your setup: * make sure to replace the `--attach-policy-arn` value with the ARN of the policy you created in the previous step. * use your EKS cluster name and region. ```bash theme={null} eksctl create iamserviceaccount \ --name default \ --namespace odigos-system \ --cluster otel-cluster \ --region=us-east-1 \ --attach-policy-arn arn:aws:iam::123456789012:policy/odigos-s3 \ --override-existing-serviceaccounts \ --approve ``` **Format** The aws s3 exporter supports two formats: `otlp_json` and `otlp_proto`. Each batch of logs / metrics / spans will be written as a single s3 object in the specified bucket. The content of the object will be the serialized data in the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto), represented as either JSON or Protocol Buffers. For example, the following JSON string represents a single span in `otlp_json` format, from Odigos demo application: ```json theme={null} {"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"service.name","value":{"stringValue":"inventory"}},{"key":"odigos.device","value":{"stringValue":"python"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"k8s.namespace.name","value":{"stringValue":"default"}},{"key":"k8s.deployment.name","value":{"stringValue":"inventory"}},{"key":"k8s.pod.name","value":{"stringValue":"inventory-7b87f7566c-2d6fq"}},{"key":"container.name","value":{"stringValue":"inventory"}},{"key":"k8s.node.name","value":{"stringValue":"ip-192-168-4-178.ec2.internal"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9ee5a0de5802d3f1e7f93440f892509c","spanId":"88a4bfb5bfcaf297","parentSpanId":"f9d7bb9e9243df06","name":"/buy","kind":2,"startTimeUnixNano":"1707600957298456744","endTimeUnixNano":"1707600958303042894","attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"8080"}},{"key":"http.host","value":{"stringValue":"inventory:8080"}},{"key":"http.target","value":{"stringValue":"/buy?id=10"}},{"key":"net.peer.ip","value":{"stringValue":"192.168.13.194"}},{"key":"http.user_agent","value":{"stringValue":"Java/17.0.9"}},{"key":"net.peer.port","value":{"intValue":"49420"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/buy"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]} ``` ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **S3\_BUCKET** `string` : Bucket Name. The name of the bucket where the data will be stored * This field is required * **S3\_REGION** `string` : Bucket Region. The AWS region where the bucket is located * This field is required * **S3\_PARTITION** `string` : Time granularity of S3 Bucket (Deprecated, use Partition Format). Deprecated field for time granularity. Use S3\_PARTITION\_FORMAT for custom formats. * This field is optional and defaults to `minute` * **S3\_PARTITION\_FORMAT** `string` : S3 Partition Format. Custom partition format using strftime syntax (e.g., '%Y/%m/%d/%H/%M' for minute-level, '%Y/%m/%d/%H' for hour-level). If not specified, S3\_PARTITION will be used. * This field is optional * Example: `%Y/%m/%d/%H/%M` * **S3\_MARSHALER** `string` : Marshaller (Data Format). The format in which the data will be encoded. It can be either `otlp_json` or `otlp_proto`. Default is `otlp_json` * This field is required and defaults to `otlp_json` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `AWS S3` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `s3.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: s3-example namespace: odigos-system spec: data: S3_BUCKET: S3_MARSHALER: '' S3_REGION: '' # Note: The commented fields below are optional. # S3_PARTITION: # S3_PARTITION_FORMAT: destinationName: s3 signals: - TRACES - METRICS - LOGS type: s3 ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f s3.yaml ``` # AWS X-Ray Source: https://docs.odigos.io/oss/backends/awsxray Configuring the AWS X-Ray backend (Managed) ### Getting Started xray

Make sure that you have the necessary `AWS Credentials` and permissions set up.
AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials). **Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)** The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services. Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables). ```bash theme={null} eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve ``` **Setting up the IAM policy** ```json theme={null} { "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Action": [ "xray:PutTraceSegments", "xray:PutTelemetryRecords" ], "Resource": [ "*" ] } } ``` ### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **AWS\_XRAY\_REGION** `string` : Region. Send segments to AWS X-Ray service in a specific region. * This field is optional * **AWS\_XRAY\_ENDPOINT** `string` : Endpoint. Optionally override the default X-Ray service endpoint. * This field is optional * **AWS\_XRAY\_PROXY\_ADDRESS** `string` : Proxy Address. Upload segments to AWS X-Ray through a proxy. * This field is optional * **AWS\_XRAY\_RESOURCE\_ARN** `string` : Resource ARN. Amazon Resource Name (ARN) of the AWS resource running the collector. * This field is optional * **AWS\_XRAY\_ROLE\_ARN** `string` : Role ARN. IAM role to upload segments to a different account. * This field is optional * **AWS\_XRAY\_EXTERNAL\_ID** `string` : External ID. Shared identitier used when assuming an IAM role in an external AWS account. * This field is optional * **AWS\_XRAY\_LOG\_GROUPS** `string[]` : Log Groups. List of log group names for CloudWatch. * This field is optional * **AWS\_XRAY\_DISABLE\_SSL** `boolean` : Disable SSL Verification. Enable or disable TLS certificate verification. * This field is optional and defaults to `False` * **AWS\_XRAY\_LOCAL\_MODE** `boolean` : Local Mode. Local mode to skip EC2 instance metadata check. * This field is optional and defaults to `False` * **AWS\_XRAY\_TELEMETRY\_ENABLED** `boolean` : Telemetry - Enabled. Whether telemetry collection is enabled at all. * This field is optional and defaults to `False` * **AWS\_XRAY\_TELEMETRY\_INCLUDE\_METADATA** `boolean` : Telemetry - Include Metadata. Whether to include metadata in the telemetry (InstanceID, Hostname, ResourceARN) * This field is optional and defaults to `False` * **AWS\_XRAY\_TELEMETRY\_HOSTNAME** `string` : Telemetry - Hostname. Sets the Hostname included in the telemetry. * This field is optional * **AWS\_XRAY\_TELEMETRY\_INSTANCE\_ID** `string` : Telemetry - Instance ID. Sets the InstanceID included in the telemetry. * This field is optional * **AWS\_XRAY\_TELEMETRY\_RESOURCE\_ARN** `string` : Telemetry - Resource ARN. Sets the Amazon Resource Name (ARN) included in the telemetry. * This field is optional * **AWS\_XRAY\_TELEMETRY\_CONTRIBUTORS** `string[]` : Telemetry - Contributors. List of X-Ray component IDs contributing to the telemetry (ex. for multiple X-Ray receivers: awsxray/1, awsxray/2) * This field is optional * **AWS\_XRAY\_INDEX\_ALL\_ATTRIBUTES** `boolean` : Index All Attributes. Enable or disable conversion of all OpenTelemetry attributes to X-Ray annotations. * This field is optional and defaults to `False` * **AWS\_XRAY\_INDEXED\_ATTRIBUTES** `string[]` : Indexed Attributes. List of attribute names to be converted to X-Ray annotations. * This field is optional **Traces and logs correlation** AWS X-Ray can be integrated with CloudWatch Logs to correlate traces with logs. For this integration to work, the X-Ray segments must have the AWS Property `cloudwatch_logs` set. This property is set using the AWS X-Ray exporter with the following values that are evaluated in this order: 1. `aws.log.group.arns` resource attribute. 2. `aws.log.group.names` resource attribute. 3. `aws_log_groups` configuration property. `aws.log.group.arns` and `aws.log.group.names` are slice resource attributes, alternatively those resource attributes can be set using the [OTEL\_RESOURCE\_ATTRIBUTES environment variable](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md#specifying-resource-information-via-an-environment-variable). To set multiple log group names / log group arns, you can use `&` to separate them.
For example 3 log groups: `log-group1`, `log-group2`, and `log-group3` are set in the following command: ```bash theme={null} export OTEL_RESOURCE_ATTRIBUTES="aws.log.group.names=log-group1&log-group2&log-group3" ``` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `AWS X-Ray` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `xray.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: xray-example namespace: odigos-system spec: data: {} destinationName: xray signals: - TRACES type: xray ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f xray.yaml ``` # Axiom Source: https://docs.odigos.io/oss/backends/axiom Configuring the Axiom backend (Managed) ### Getting Started axiom

1. [Register](https://app.axiom.co/register)/[Login](https://app.axiom.co/login) to Axiom. 2. Navigate to the [Axiom Datasets](https://app.axiom.co/settings/datasets) page, create a dataset and copy its name. 3. Navigate to the [Axiom API Tokens](https://app.axiom.co/settings/api-tokens) page, and generate a new API Token. ### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **AXIOM\_DATASET** `string` : Dataset. * This field is required * **AXIOM\_API\_TOKEN** `string` : Axiom API token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Axiom` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `axiom.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: axiom-example namespace: odigos-system spec: data: AXIOM_DATASET: destinationName: axiom secretRef: name: axiom-secret signals: - TRACES - LOGS type: axiom --- apiVersion: v1 data: AXIOM_API_TOKEN: kind: Secret metadata: name: axiom-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f axiom.yaml ``` # Azure Blob Storage Source: https://docs.odigos.io/oss/backends/azureblob Configuring the Azure Blob Storage backend (Managed) ### Getting Started

Odigos exports data to Azure Blob Storage in [OTLP format](https://opentelemetry.io/docs/specs/otlp/). Data can be exported either in JSON format or in binary format (protobuf). ### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **AZURE\_BLOB\_ACCOUNT\_NAME** `string` : Account Name. * This field is required * **AZURE\_BLOB\_CONTAINER\_NAME** `string` : Container Name. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Azure Blob Storage` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `azureblob.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: azureblob-example namespace: odigos-system spec: data: AZURE_BLOB_ACCOUNT_NAME: AZURE_BLOB_CONTAINER_NAME: destinationName: azureblob signals: - TRACES - LOGS type: azureblob ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f azureblob.yaml ``` # Azure Monitor Source: https://docs.odigos.io/oss/backends/azuremonitor Configuring the Azure Monitor backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **AZURE\_MONITOR\_CONNECTION\_STRING** `string` : Connection String. The Connection String is the recommended method for configuring the exporter, aligning with Azure Monitor's best practices. If you don't have a Connection String, you can create one in the Azure Portal (after creating an Application Insights resource). * This field is optional * **AZURE\_MONITOR\_INSTRUMENTATION\_KEY** `string` : Instrumentation Key. Application Insights instrumentation key, which can be found in the Application Insights resource in the Azure Portal. While it is currently supported, its use is discouraged and it is slated for deprecation. * This field is optional * **AZURE\_MONITOR\_ENDPOINT** `string` : Endpoint. The endpoint URL where data will be submitted. While this option remains available, it is important to note that the use of the Connection String is recommended, as it encompasses the endpoint information. The direct configuration of the endpoint is considered to be on a deprecation path. * This field is optional Do not configure the `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable. It will override this configuration! ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Azure Monitor` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `azuremonitor.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: azuremonitor-example namespace: odigos-system spec: data: {} destinationName: azuremonitor signals: - TRACES - METRICS - LOGS type: azuremonitor ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f azuremonitor.yaml ``` # Better Stack Source: https://docs.odigos.io/oss/backends/betterstack Configuring the Better Stack backend (Managed) ### Getting Started

### Configuring Destination Fields ❌ Traces ✅ Metrics ✅ Logs ❌ Profiles * **BETTERSTACK\_TOKEN** `string` : Better Stack Source Token. * This field is required We handle the endpoint internally, so you don't need to provide it. * The endpoint for metrics is `https://in-otel.logs.betterstack.com/metrics` * The endpoint for logs is `https://in-otel.logs.betterstack.com:443` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Better Stack` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `betterstack.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: betterstack-example namespace: odigos-system spec: data: {} destinationName: betterstack secretRef: name: betterstack-secret signals: - METRICS - LOGS type: betterstack --- apiVersion: v1 data: BETTERSTACK_TOKEN: kind: Secret metadata: name: betterstack-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f betterstack.yaml ``` # Bonree ONE Source: https://docs.odigos.io/oss/backends/bonree Configuring the Bonree ONE backend (Managed) ### Getting Started bonree

### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **BONREE\_ENDPOINT** `string` : OTLP HTTP Endpoint. APM controller address, obtained from the probe deployment page. * This field is required * **BONREE\_ACCOUNT\_ID** `string` : Account ID. Obtained from the probe deployment page. * This field is required * **BONREE\_ENVIRONMENT\_ID** `string` : Environment ID. The default environment does not need to be configured. * This field is optional ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Bonree ONE` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `bonree.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: bonree-example namespace: odigos-system spec: data: BONREE_ENDPOINT: destinationName: bonree # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: bonree-secret signals: - TRACES - METRICS type: bonree --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # BONREE_ACCOUNT_ID: # BONREE_ENVIRONMENT_ID: # kind: Secret # metadata: # name: bonree-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f bonree.yaml ``` # Causely Source: https://docs.odigos.io/oss/backends/causely Configuring the Causely backend (Managed) ### Getting Started causely

This destination is for the Causely Mediator Service, so you will need to have a Causely instance running and accessible from the k8s cluster running Odigos. ### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **CAUSELY\_URL** `string` : Endpoint. The endpoint URL is the combined `://:` to access your Causely Mediator service. Protocol should be `http`; using `https` or omitting it will automatically be converted to `http`. Hostname should typically follow the format: `mediator.`. Namespace is the k8s namespace where the Causely Mediator service is deployed. Port is optional and defaults to the default OTLP gRPC port `4317` * This field is required * Example: `http://mediator.causely:4317` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Causely` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `causely.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: causely-example namespace: odigos-system spec: data: CAUSELY_URL: destinationName: causely signals: - TRACES - METRICS type: causely ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f causely.yaml ``` # Checkly Source: https://docs.odigos.io/oss/backends/checkly Configuring the Checkly backend (Managed) ### Getting Started checkly

### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **CHECKLY\_ENDOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`. * This field is required * **CHECKLY\_API\_KEY** `string` : API Key. * This field is required Odigos will only filter and export traces to Checkly if their `trace_state` includes the tag `checkly="true"`, which is added by a Checkly SDK. ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Checkly` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `checkly.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: checkly-example namespace: odigos-system spec: data: CHECKLY_ENDOINT: destinationName: checkly secretRef: name: checkly-secret signals: - TRACES type: checkly --- apiVersion: v1 data: CHECKLY_API_KEY: kind: Secret metadata: name: checkly-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f checkly.yaml ``` # Chronosphere Source: https://docs.odigos.io/oss/backends/chronosphere Configuring the Chronosphere backend (Managed) ### Getting Started

Chronosphere is a monitoring and observability platform that provides a scalable, reliable, and cost-effective way to collect, store, and analyze metrics and traces. To send data from Odigos to Chronosphere, [you need to have an account with Chronosphere](https://chronosphere.io/demo-request/) and configure the Chronosphere destination in Odigos with the following details: ### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **CHRONOSPHERE\_DOMAIN** `string` : Chronosphere Company Domain. Company domain in Chronosphere, can be found in the URL of the Chronosphere UI. For example, if your URL is `https://demo-sandbox.chronosphere.io/`, then your company domain is `demo-sandbox` * This field is required * **CHRONOSPHERE\_API\_TOKEN** `string` : API Token. API token generated from your Chronosphere service account in the Chronosphere UI, you should login to your Chronosphere admin account click the `Platform` menu on the left side of the screen, then click `Service Accounts` and create a new service account * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Chronosphere` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `chronosphere.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: chronosphere-example namespace: odigos-system spec: data: CHRONOSPHERE_DOMAIN: destinationName: chronosphere secretRef: name: chronosphere-secret signals: - TRACES - METRICS type: chronosphere --- apiVersion: v1 data: CHRONOSPHERE_API_TOKEN: kind: Secret metadata: name: chronosphere-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f chronosphere.yaml ``` # ClickHouse Source: https://docs.odigos.io/oss/backends/clickhouse Configuring the ClickHouse backend (Self-Hosted) ### Getting Started

ClickHouse is very popular database for storing telemetry data and running efficient queries on it. **Use Cases** * ClickHouse can handle all 3 signals (metrics, logs, traces) in one single database. less overhead to manage multiple databases. * ClickHouse is a columnar database, which is optimized for time-series immutable data like OpenTelemetry telemetry data. * ClickHouse is a distributed database, which can scale horizontally and handle large amounts of data with efficient storage and query performance. * ClickHouse is open-source and has a large community. **Prerequisites** * To use ClickHouse as a destination in Odigos, you need to have a ClickHouse deployment running somewhere and accessible from cluster where Odigos is running. * You should know the service endpoint where ClickHouse listens for incoming client connections. * If you haven't already, create a database in ClickHouse where you want to store the telemetry data. the default database name is `otel` (configurable). To create it, run the following SQL command: `CREATE DATABASE otel;` * Understanding of how to maintain, scale, and optimize your self-hosted ClickHouse deployment, as well as how to fine tune setting based on your queries and use case. **Schema** When using the ClickHouse destination with Odigos, logs metrics and traces are going to be "INSERT INTO" the ClickHouse database. It is important to understand what schema is used, e.g. what table names are used, column names, data types, etc. Then you can run queries on this data, or modify and optimize it for your specific use case. There are 2 modes of operation for ClickHouse destination in Odigos: **Create Schema** In this option, the schema will be automatically created by odigos with reasonable defaults. The benefit of this option is that you can see the value fast, without needing to apply and manage any schema yourself. The downside is that the schema may not be optimized for your specific use case, and may make changes more complicated. To use it: * Odigos UI - When adding a new ClickHouse destination, select the `Create Scheme` checkbox field. * Destination K8s Manifest - Set the `CLICKHOUSE_CREATE_SCHEME` setting to value `true`. The schema which will be used by default can be found [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/a80fdb94cecb1cd3c1fbd7fbe3000396cd588970/exporter/clickhouseexporter/example/default_ddl). * Indexes - The default schema includes indexes for the data, including trace\_id, resource and scope attributes, span/metric/log attributes etc. * TTL is set to 180 days. This means data will be kept in the database for this period of time and then deleted. * Partitioning - The default schema includes partitioning by day, which means data is stored in separate partitions for each day. * Order By - Optimized for trace queries on service\_name + span\_name + time, for logs on service\_name + time, and for metrics on service\_name + metric\_name + attributes + time. This option is not recommended for production workloads: * You may want to adjust the settings to better fit your use case, scale performance requirements, and costs. * Each new exporter will attempt to create the schema, which is less robust and harder to manage than a pre-created schema. **Self Managed Schema** With this option, you are responsible for creating and managing the schema yourself. To use it: * Odigos UI - Unselect the `Create Scheme` checkbox field. * Destination K8s Manifest - Set the `CLICKHOUSE_CREATE_SCHEME` setting to value `false`. The benefit of this option is that you have full control over the schema, and can optimize it for your specific use case. How it works? * Browse to the [default DDL example](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/a80fdb94cecb1cd3c1fbd7fbe3000396cd588970/exporter/clickhouseexporter/example/default_ddl). * Copy the `sql` files to your local machine. * Make any changes you need to the schema. * Run the SQL files in your ClickHouse database to create the schema. This option is recommended for production workloads: * You can optimize the schema for your specific use case, scale performance requirements, and costs. * You can manage the schema in a version control system, and apply changes in a controlled way. * Applying changes to the schema is more robust and easier to manage than attempting to create it on the fly with each new connection. Important Settings: * Indexes - You may want to add or remove indexes based on your queries, to optimize performance and costs. * TTL - You may want to adjust the TTL based on your retention policy. If you need traces for auditing purposes, you may extend it to 365 days. For high-throughput, low-latency systems, you might want to reduce it to 90 days or even less. * Partitioning - partitioning by day is a good default, but in high throughput systems you may want to partition by hour or even minute. You may also consider partitioning by service\_name, or other attributes. * Order By - You may want to adjust the order by clause based on your queries, so that common columns are used first in the query, to optimize performance. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **CLICKHOUSE\_ENDPOINT** `string` : Endpoint. The ClickHouse endpoint is the URL where the ClickHouse server is listening for incoming connections. * This field is required * Example: `http://host:port` * **CLICKHOUSE\_USERNAME** `string` : Username. If Clickhouse Authentication is used, provide the username * This field is optional * **CLICKHOUSE\_PASSWORD** `string` : Password. If Clickhouse Authentication is used, provide the password * This field is optional * **CLICKHOUSE\_CREATE\_SCHEME** `boolean` : Create Schema. Should the destination create the schema for you? Set to `false` if you manage your own schema, or `true` to have Odigos create the schema for you * This field is required and defaults to `True` * **CLICKHOUSE\_DATABASE\_NAME** `string` : Database Name. The name of the Clickhouse Database where the telemetry data will be stored. The Database will not be created when not exists, so make sure you have created it before * This field is required and defaults to `otel` * **CLICKHOUSE\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection * This field is optional and defaults to `False` * **CLICKHOUSE\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification * This field is optional and defaults to `False` * **CLICKHOUSE\_USE\_CUSTOM\_CA** `boolean` : Use Custom CA Certificate. Use a custom CA certificate instead of the system root CA * This field is optional and defaults to `False` * **CLICKHOUSE\_CA\_PEM** `string` : Certificate Authority (CA PEM). CA certificate in PEM format to verify the server * This field is optional * Example: `-----BEGIN CERTIFICATE-----` * **CLICKHOUSE\_TRACES\_TABLE** `string` : Traces Table. Name of the ClickHouse Table to use for storing trace spans. This name should be used in span queries * This field is required and defaults to `otel_traces` * **CLICKHOUSE\_LOGS\_TABLE** `string` : Logs Table. Name of the ClickHouse Table to use for storing logs * This field is required and defaults to `otel_logs` * **CLICKHOUSE\_METRICS\_TABLE\_EXP\_HISTOGRAM** `string` : Metrics Table - Exp. Histogram. Name of the table for storing exponential histogram metrics * This field is required and defaults to `otel_metrics_exponential_histogram` * **CLICKHOUSE\_METRICS\_TABLE\_GAUGE** `string` : Metrics Table - Gauge. Name of the table for storing gauge metrics * This field is required and defaults to `otel_metrics_gauge` * **CLICKHOUSE\_METRICS\_TABLE\_HISTOGRAM** `string` : Metrics Table - Histogram. Name of the table for storing histogram metrics * This field is required and defaults to `otel_metrics_histogram` * **CLICKHOUSE\_METRICS\_TABLE\_SUM** `string` : Metrics Table - Sum. Name of the table for storing sum metrics * This field is required and defaults to `otel_metrics_sum` * **CLICKHOUSE\_METRICS\_TABLE\_SUMMARY** `string` : Metrics Table - Summary. Name of the table for storing summary metrics * This field is required and defaults to `otel_metrics_summary` * **HYPERDX\_LOG\_NORMALIZER** `boolean` : Enable Log Normalizer. Enable HyperDX log normalization processor. This parses JSON from log bodies, infers severity levels, and normalizes log attributes for better querying. * This field is optional and defaults to `False` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Clickhouse` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `clickhouse.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: clickhouse-example namespace: odigos-system spec: data: CLICKHOUSE_CREATE_SCHEME: '' CLICKHOUSE_DATABASE_NAME: '' CLICKHOUSE_ENDPOINT: CLICKHOUSE_LOGS_TABLE: '' CLICKHOUSE_METRICS_TABLE_EXP_HISTOGRAM: '' CLICKHOUSE_METRICS_TABLE_GAUGE: '' CLICKHOUSE_METRICS_TABLE_HISTOGRAM: '' CLICKHOUSE_METRICS_TABLE_SUM: '' CLICKHOUSE_METRICS_TABLE_SUMMARY: '' CLICKHOUSE_TRACES_TABLE: '' # Note: The commented fields below are optional. # CLICKHOUSE_USERNAME: # CLICKHOUSE_TLS_ENABLED: # CLICKHOUSE_INSECURE_SKIP_VERIFY: # CLICKHOUSE_USE_CUSTOM_CA: # HYPERDX_LOG_NORMALIZER: destinationName: clickhouse # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: clickhouse-secret signals: - TRACES - METRICS - LOGS type: clickhouse --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # CLICKHOUSE_CA_PEM: # CLICKHOUSE_PASSWORD: # kind: Secret # metadata: # name: clickhouse-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f clickhouse.yaml ``` # Coralogix Source: https://docs.odigos.io/oss/backends/coralogix Configuring the Coralogix backend (Managed) ### Getting Started

1. Get an [API Key](https://coralogix.com/docs/send-your-data-api-key/). 2. Obtain the [Domain](https://coralogix.com/docs/coralogix-domain/) determined by your Coralogix Account URL. 3. Follow the [Application and Subsystem Names guide](https://coralogix.com/docs/application-and-subsystem-names/) to learn how configure the application and subsystem names for your Coralogix account. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **CORALOGIX\_PRIVATE\_KEY** `string` : Send-Your-Data API Key. The Send-Your-Data API Key is used to authenticate the data sent to Coralogix. * This field is required * **CORALOGIX\_DOMAIN** `string` : Domain. The Coralogix domain to which you want to send the data. * This field is required * **CORALOGIX\_APPLICATION\_NAME** `string` : Application Name. The name of the application that sends the data to Coralogix. * This field is required * **CORALOGIX\_SUBSYSTEM\_NAME** `string` : Subsystem Name. The name of the subsystem that sends the data to Coralogix. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Coralogix` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `coralogix.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: coralogix-example namespace: odigos-system spec: data: CORALOGIX_APPLICATION_NAME: CORALOGIX_DOMAIN: '' CORALOGIX_SUBSYSTEM_NAME: destinationName: coralogix secretRef: name: coralogix-secret signals: - TRACES - METRICS - LOGS type: coralogix --- apiVersion: v1 data: CORALOGIX_PRIVATE_KEY: kind: Secret metadata: name: coralogix-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f coralogix.yaml ``` # Dash0 Source: https://docs.odigos.io/oss/backends/dash0 Configuring the Dash0 backend (Managed) ### Getting Started dash0

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **DASH0\_ENDPOINT** `string` : Dash0 OTLP gRPC Endpoint. The format is `host:port`. Host is required, located in Dash0 UI (OpenTelemetry Collector). Port is optional and defaults to the default OTLP gRPC port `4317`. * This field is required * **DASH0\_TOKEN** `string` : Dash0 Bearer Token. Located in Dash0 UI (OpenTelemetry Collector). * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Dash0` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `dash0.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: dash0-example namespace: odigos-system spec: data: DASH0_ENDPOINT: destinationName: dash0 secretRef: name: dash0-secret signals: - TRACES - METRICS - LOGS type: dash0 --- apiVersion: v1 data: DASH0_TOKEN: kind: Secret metadata: name: dash0-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f dash0.yaml ``` # Datadog Source: https://docs.odigos.io/oss/backends/datadog Configuring the Datadog backend (Managed) ### Getting Started datadog

On the left sidebar, click on your username and then click on **Organization Settings** Organization settings menu

Click on **API Keys** and then click on **New Key** and give it a name. New API key screen

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **DATADOG\_API\_KEY** `string` : API Key. The API key for your Datadog account. * This field is required * **DATADOG\_SITE** `string` : Site. The datadog site you are using (the url of the datadog site you are using). * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Datadog` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `datadog.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: datadog-example namespace: odigos-system spec: data: DATADOG_SITE: '' destinationName: datadog secretRef: name: datadog-secret signals: - TRACES - METRICS - LOGS type: datadog --- apiVersion: v1 data: DATADOG_API_KEY: kind: Secret metadata: name: datadog-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f datadog.yaml ``` # Dynamic Destination Source: https://docs.odigos.io/oss/backends/dynamic Configuring the Dynamic Destination backend (Self-Hosted) ### Getting Started dynamic

Use Dynamic Destination when you need a collector exporter configuration that is not covered by a built-in destination type (for example, OTLP with client mTLS via freeform YAML). Environment variable placeholders such as `${OTLP_CA_PEM}` in `DYNAMIC_CONFIGURATION_DATA` are expanded by the gateway collector from pod environment variables. To inject those values, create a Secret in the **same namespace** as the Destination (the Odigos system namespace) whose keys match the placeholder names, and set `spec.secretRef.name` on the Destination. Without `secretRef`, the instrumentor will not mount the Secret into the gateway and the placeholders remain unset. Use unique env var names across dynamic destinations — they are mounted into the shared gateway without a destination prefix (built-in destinations are automatically scoped). ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ✅ Profiles * **DYNAMIC\_DESTINATION\_TYPE** `string` : Destination Type. The type of OpenTelemetry Collector Exporter to use for the destination. * This field is required * Example: `otlp` * **DYNAMIC\_CONFIGURATION\_DATA** `string` : Config. The raw YAML config for the Collector Exporter for this destination. To inject secrets (e.g. TLS PEMs), use \$ placeholders and set Destination.spec.secretRef to a Secret in the same namespace whose keys match those variable names. Use unique env var names across dynamic destinations — they are mounted into the shared gateway without a destination prefix. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Dynamic Destination` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `dynamic.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: dynamic-example namespace: odigos-system spec: data: DYNAMIC_CONFIGURATION_DATA: DYNAMIC_DESTINATION_TYPE: destinationName: dynamic signals: - TRACES - METRICS - LOGS - PROFILES type: dynamic ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f dynamic.yaml ``` # Dynatrace Source: https://docs.odigos.io/oss/backends/dynatrace Configuring the Dynatrace backend (Managed) ### Getting Started

On the left sidebar click on `Apps`, then under `Manage` select `Access Tokens`: Obtaining access token

Make sure to add the required capabilities for the access tokens (e.g `Ingest OpenTelemetry traces` for traces). ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **DYNATRACE\_URL** `string` : Tenant URL. * This field is required * Example: `https://{your-environment-id}.live.dynatrace.com` * **DYNATRACE\_API\_TOKEN** `string` : API Access Token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Dynatrace` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `dynatrace.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: dynatrace-example namespace: odigos-system spec: data: DYNATRACE_URL: destinationName: dynatrace secretRef: name: dynatrace-secret signals: - TRACES - METRICS - LOGS type: dynatrace --- apiVersion: v1 data: DYNATRACE_API_TOKEN: kind: Secret metadata: name: dynatrace-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f dynatrace.yaml ``` # Elastic APM Source: https://docs.odigos.io/oss/backends/elasticapm Configuring the Elastic APM backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **ELASTIC\_APM\_SERVER\_ENDPOINT** `string` : elastic APM server endpoint. * This field is required * **ELASTIC\_APM\_SECRET\_TOKEN** `string` : elastic APM server token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Elastic APM` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `elasticapm.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: elasticapm-example namespace: odigos-system spec: data: ELASTIC_APM_SERVER_ENDPOINT: destinationName: elasticapm secretRef: name: elasticapm-secret signals: - TRACES - METRICS - LOGS type: elasticapm --- apiVersion: v1 data: ELASTIC_APM_SECRET_TOKEN: kind: Secret metadata: name: elasticapm-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f elasticapm.yaml ``` # Elasticsearch Source: https://docs.odigos.io/oss/backends/elasticsearch Configuring the Elasticsearch backend (Self-Hosted) ### Getting Started

### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **ELASTICSEARCH\_URL** `string` : Elasticsearch URL. Elasticsearch endpoint. Port defaults to `9200` if not specified * This field is required * Example: `http://host:port` * **ES\_TRACES\_INDEX** `string` : Traces Index. The name of the index where traces will be stored * This field is optional and defaults to `trace_index` * **ES\_LOGS\_INDEX** `string` : Logs Index. The name of the index where logs will be stored. * This field is optional and defaults to `log_index` * **ELASTICSEARCH\_BASIC\_AUTH\_ENABLED** `boolean` : Enable HTTP Basic Authentication. * This field is optional and defaults to `False` * **ELASTICSEARCH\_USERNAME** `string` : Username. Username used for HTTP Basic Authentication * This field is optional * **ELASTICSEARCH\_PASSWORD** `string` : Password. Password used for HTTP Basic Authentication * This field is optional * **ELASTICSEARCH\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection (Transport Layer Security) * This field is optional and defaults to `False` * **ELASTICSEARCH\_CA\_PEM** `string` : CA Certificate. When using TLS, provide the CA certificate to verify the server. If empty uses system root CA * This field is optional * Example: `-----BEGIN CERTIFICATE-----` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Elasticsearch` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `elasticsearch.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: elasticsearch-example namespace: odigos-system spec: data: ELASTICSEARCH_URL: # Note: The commented fields below are optional. # ES_TRACES_INDEX: # ES_LOGS_INDEX: # ELASTICSEARCH_BASIC_AUTH_ENABLED: # ELASTICSEARCH_USERNAME: # ELASTICSEARCH_TLS_ENABLED: # ELASTICSEARCH_CA_PEM: destinationName: elasticsearch # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: elasticsearch-secret signals: - TRACES - LOGS type: elasticsearch --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # ELASTICSEARCH_PASSWORD: # kind: Secret # metadata: # name: elasticsearch-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f elasticsearch.yaml ``` # Gigapipe Source: https://docs.odigos.io/oss/backends/gigapipe Configuring the Gigapipe backend (Managed) ### Getting Started qryn

Access your Gigapipe account and generate a **Scoped Token** for Odigos Gigapipe Token

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **QRYN\_API\_SECRET** `string` : API Secret. The API Secret for your Gigapipe Account * This field is required * **QRYN\_API\_KEY** `string` : API Key. The API Token for your Gigapipe Account * This field is required * **QRYN\_URL** `string` : API Url. The API Endpoint for your Gigapipe Account * This field is required * Example: `https://your-account.gigapipe.com` * **QRYN\_RESOURCE\_TO\_TELEMETRY\_CONVERSION** `boolean` : Convert container attributes to labels. * This field is optional and defaults to `True` * **QRYN\_ADD\_EXPORTER\_NAME** `boolean` : Add exporter name to labels. * This field is optional and defaults to `True` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Gigapipe` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `qryn.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: qryn-example namespace: odigos-system spec: data: QRYN_API_KEY: QRYN_URL: # Note: The commented fields below are optional. # QRYN_RESOURCE_TO_TELEMETRY_CONVERSION: # QRYN_ADD_EXPORTER_NAME: destinationName: qryn secretRef: name: qryn-secret signals: - TRACES - METRICS - LOGS type: qryn --- apiVersion: v1 data: QRYN_API_SECRET: kind: Secret metadata: name: qryn-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f qryn.yaml ``` # Google Cloud Monitoring Source: https://docs.odigos.io/oss/backends/googlecloudmonitoring Configuring the Google Cloud Monitoring backend (Managed) ### Getting Started

### Google Cloud Trace API This destination uses the [Google Cloud Trace API](https://cloud.google.com/trace/docs/reference). It is different from the [Google CLoud Telemetry (OTLP) API destination](https://docs.odigos.io/backends/googlecloudotlp). On GCP, Odigos currently supports Standard (non-Autopilot) GKE clusters. ### Authenticating with Google Cloud When running the Google Cloud Exporter on GKE, by default the OpenTelemetry Collector will attempt to use [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate to the Google Cloud Observability backend. This means that for most use cases running on GCP, there is not authentication or service account token required as your workloads are automatically authenticated. However, to authenticate with Cloud Observability when running off-GKE (for example, on a local machine for another cloud provider) you must provide a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) for the OpenTelemetry Collector Exporter. In Odigos, this can be provided in the Application Credentials field of the Google Cloud destination. When the Application Credentials field is set in the UI, the Odigos UI will save the contents of that field as a Kubernetes Secret which is mounted as a volume into the Collector Pod. The location of that volume is then set as the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, instructing the Google Cloud Exporter where to look for its credentials. When creating the destination as a YAML manifest off-GKE, the SecretRef must point to a Secret containing the Service Account token. The token must be set as the `GCP_APPLICATION_CREDENTIALS` key in the referenced Secret. The `GCP_APPLICATION_CREDENTIALS` field does not need to be set on the YAML manifest destination CRD, only in the referenced Secret. This field is used internally by Odigos when creating GCP destinations from the UI. Currently, only one Google Cloud destination can be configured with Application Credentials. See the [Google Cloud OpenTelemetry Collector Exporter docs](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/tree/1ea43a4534606df1d91ee1754e6c5bc31ee8664d/exporter/collector#prerequisite-authenticating) for more information on authentication. ### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **GCP\_TIMEOUT** `string` : Timeout. The request timeout in seconds for the Google Cloud Monitoring API. Defaults to 10 seconds. Increasing this value can help with deadline exceeded errors. * This field is optional * Example: `10s` * **GCP\_PROJECT\_ID** `string` : Project ID. The project ID for the Google Cloud Monitoring API to send data to. Required if running Odigos outside of Google Cloud. Defaults to the detected project ID when running on GCP. * This field is optional * Example: `project-id` * **GCP\_APPLICATION\_CREDENTIALS** `string` : Application Credentials. The JSON GCP Application Credentials file for the GCP project. Either a Workload Identity Federation config file or Service Account Key. Required if running Odigos outside of Google Cloud. * This field is optional * Example: `Credentials config... (optional)` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Google Cloud Monitoring` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `googlecloud.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: googlecloud-example namespace: odigos-system spec: data: {} destinationName: googlecloud # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: googlecloud-secret signals: - TRACES - LOGS type: googlecloud --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # GCP_APPLICATION_CREDENTIALS: # kind: Secret # metadata: # name: googlecloud-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f googlecloud.yaml ``` # Google Cloud (OTLP) Source: https://docs.odigos.io/oss/backends/googlecloudotlp Configuring the Google Cloud (OTLP) backend (Managed) ### Getting Started

### Google Cloud Telemetry API This destination uses the [Google Cloud Telemetry (OTLP) API](https://cloud.google.com/stackdriver/docs/reference/telemetry/overview) to send data as OTLP to GCP Observability using an OTLP/HTTP exporter. It is different from the [Google Cloud Monitoring destination](https://docs.odigos.io/backends/googlecloudmonitoring) which uses the [googlecloud exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/d9769f7c1a0f3bea82c711af551d9b5b44d69e6f/exporter/googlecloudexporter). Ong GCP, Odigos currently supports Standard (non-Autopilot) GKE clusters. ### Authenticating with Google Cloud When running the Google Cloud Exporter on GKE, by default the OpenTelemetry Collector will attempt to use [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate to the Google Cloud Observability backend. This means that for most use cases running on GCP, there is not authentication or service account token required as your workloads are automatically authenticated. However, to authenticate with Cloud Observability when running off-GKE (for example, on a local machine for another cloud provider) you must provide a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) for the OpenTelemetry Collector Exporter. In Odigos, this can be provided in the Application Credentials field of the Google Cloud destination. When the Application Credentials field is set in the UI, the Odigos UI will save the contents of that field as a Kubernetes Secret which is mounted as a volume into the Collector Pod. The location of that volume is then set as the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, instructing the Google Cloud Exporter where to look for its credentials. When creating the destination as a YAML manifest off-GKE, the SecretRef must point to a Secret containing the Service Account token. The token must be set as the `GCP_APPLICATION_CREDENTIALS` key in the referenced Secret. The `GCP_APPLICATION_CREDENTIALS` field does not need to be set on the YAML manifest destination CRD, only in the referenced Secret. This field is used internally by Odigos when creating GCP destinations from the UI. Currently, only one Google Cloud destination can be configured with Application Credentials. ### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **GCP\_PROJECT\_ID** `string` : Project ID. The project ID for the Google Cloud Monitoring API to send data to. This defaults to the Application Credentials project ID (which is automatically detected when running on GCP). Setting this can override the Application Credentials project ID. Required if running Odigos outside of Google Cloud. * This field is optional * Example: `project-id` * **GCP\_APPLICATION\_CREDENTIALS** `string` : Application Credentials. The JSON GCP Application Credentials file for the GCP project. Either a Workload Identity Federation config file or Service Account Key. Required if running Odigos outside of Google Cloud. * This field is optional * Example: `Credentials config... (optional)` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Google Cloud (OTLP)` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `googlecloudotlp.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: googlecloudotlp-example namespace: odigos-system spec: data: {} destinationName: googlecloudotlp # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: googlecloudotlp-secret signals: - TRACES type: googlecloudotlp --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # GCP_APPLICATION_CREDENTIALS: # kind: Secret # metadata: # name: googlecloudotlp-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f googlecloudotlp.yaml ``` # Google Cloud Storage Source: https://docs.odigos.io/oss/backends/googlecloudstorage Configuring Google Cloud Storage Odigos exports data to Google Cloud Storage in [OTLP format](https://opentelemetry.io/docs/specs/otlp/). Data can be exported either in JSON format or in binary format (protobuf). Authentication to Google Cloud Storage is done using [Google Application Default Credentials](https://cloud.google.com/docs/authentication/production). ## Configuring Google Cloud Storage Backend The only required field is the **bucket name**. The bucket must exists and the odigos gateway pod must have write access to it. # Grafana Cloud Loki Source: https://docs.odigos.io/oss/backends/grafanacloudloki Configuring the Grafana Cloud Loki backend (Managed) ### Getting Started

These instructions are for the Grafana Cloud **managed** Loki service. If you run a self hosted Loki instance, please follow the instructions on the [Loki](../backends/loki) page. You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Loki as a **logs** backend. All the connection details can be found at [grafana.com](https://grafana.com) under **My Account** Grafana labs website

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Loki" card and click "Send Logs" Loki managed service configuration

**Loki Labels** Read about setting grafana loki labels in the [Loki](../backends/loki#loki-labels) page. ### Configuring Destination Fields ❌ Traces ❌ Metrics ✅ Logs ❌ Profiles * **GRAFANA\_CLOUD\_LOKI\_ENDPOINT** `string` : Endpoint. The endpoint of your Grafana Cloud Loki instance * This field is required * Example: `https://logs-prod-{REGION/SHARD}.grafana.net/otlp` * **GRAFANA\_CLOUD\_LOKI\_USERNAME** `string` : Username. You can find the loki username in the "Grafana Data Source settings" section as "User" value. The username is a number. * This field is required * Example: `1234567` * **GRAFANA\_CLOUD\_LOKI\_PASSWORD** `string` : Password (Api Token). This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `logs:write` scope * This field is required * Example: `glc_eyJvIj...` * **GRAFANA\_CLOUD\_LOKI\_LABELS** `string[]` : Labels. Use these OpenTelemetry resource attributes as Loki labels for each log record * This field is optional and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]` The supported Loki version is V3. Please make sure you are using the correct version, and that you have enabled native OTLP support. As part of this version, the path `/loki/api/v1/push` has been deprecated and replaced with `/otlp`. ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Grafana Cloud Loki` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `grafanacloudloki.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: grafanacloudloki-example namespace: odigos-system spec: data: GRAFANA_CLOUD_LOKI_ENDPOINT: GRAFANA_CLOUD_LOKI_USERNAME: # Note: The commented fields below are optional. # GRAFANA_CLOUD_LOKI_LABELS: destinationName: grafanacloudloki secretRef: name: grafanacloudloki-secret signals: - LOGS type: grafanacloudloki --- apiVersion: v1 data: GRAFANA_CLOUD_LOKI_PASSWORD: kind: Secret metadata: name: grafanacloudloki-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f grafanacloudloki.yaml ``` # Grafana Cloud Prometheus Source: https://docs.odigos.io/oss/backends/grafanacloudprometheus Configuring the Grafana Cloud Prometheus backend (Managed) ### Getting Started

These instructions are for the Grafana Cloud **managed** Prometheus service. If you run a self hosted Prometheus instance, please follow the instructions on the [Prometheus](../backends/prometheus) page. You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Prometheus as a **metrics** backend. All the connection details can be found at [grafana.com](https://grafana.com) under **My Account** Grafana labs website

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Prometheus" card and click "Send Metrics" Prometheus managed service configuration

Prometheus managed service configuration

**Resource Attributes as Metrics Labels** To use [resource attributes as labels in your data points](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter#setting-resource-attributes-as-metric-labels), you can set the resource attribute names in the `Resource Attributes as Labels` multi-input config option in the destination page in the Odigos UI. **Cluster Static Labels** You can set static labels for each metric data pointsent exported to Grafana Cloud Prometheus from the cluster by setting the `External Labels` configuration with lable keys and values. ### Configuring Destination Fields ❌ Traces ✅ Metrics ❌ Logs ❌ Profiles * **GRAFANA\_CLOUD\_PROMETHEUS\_RW\_ENDPOINT** `string` : Prometheus Remote Write Endpoint. This is the URL of the Prometheus service. From the grafana cloud UI, Prometheus page, make sure you copy the URL from “Remote Write Endpoint” section, and not the first Endpoint which is used for queries * This field is required * Example: `https://{hostname}.grafana.net/api/prom/push` * **GRAFANA\_CLOUD\_PROMETHEUS\_USERNAME** `string` : Username / Instance ID. You can find the Username / Instance ID on the Prometheus page. Value is a number. * This field is required * Example: `12345678` * **GRAFANA\_CLOUD\_PROMETHEUS\_PASSWORD** `string` : Password / Api Token. This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `metrics:write` scope. * This field is required * Example: `glc_eyJvIj...` * **PROMETHEUS\_RESOURCE\_ATTRIBUTES\_LABELS** `string[]` : Resource Attributes Labels. use these OpenTelemetry resource attributes as prometheus labels for each metric data point * This field is optional and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]` * **PROMETHEUS\_RESOURCE\_EXTERNAL\_LABELS** `{ key: string; value: string; }[]` : External Labels. map of labels names and values to be attached to each metric data point * This field is optional ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Grafana Cloud Prometheus` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `grafanacloudprometheus.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: grafanacloudprometheus-example namespace: odigos-system spec: data: GRAFANA_CLOUD_PROMETHEUS_RW_ENDPOINT: GRAFANA_CLOUD_PROMETHEUS_USERNAME: # Note: The commented fields below are optional. # PROMETHEUS_RESOURCE_ATTRIBUTES_LABELS: # PROMETHEUS_RESOURCE_EXTERNAL_LABELS: destinationName: grafanacloudprometheus secretRef: name: grafanacloudprometheus-secret signals: - METRICS type: grafanacloudprometheus --- apiVersion: v1 data: GRAFANA_CLOUD_PROMETHEUS_PASSWORD: kind: Secret metadata: name: grafanacloudprometheus-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f grafanacloudprometheus.yaml ``` # Grafana Cloud Tempo Source: https://docs.odigos.io/oss/backends/grafanacloudtempo Configuring the Grafana Cloud Tempo backend (Managed) ### Getting Started

These instructions are for the Grafana Cloud **managed** Tempo service. If you run a self hosted Tempo instance, please follow the instructions on the [Tempo](../backends/tempo) page. You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Tempo as a **trace** backend. All the connection details can be found at [grafana.com](https://grafana.com) under **My Account** Grafana labs website

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Tempo" card and click "Send Traces" Tempo managed service configuration

### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **GRAFANA\_CLOUD\_TEMPO\_ENDPOINT** `string` : Endpoint. This is the URL of the Tempo service, in the form of `host:port`. You can find the endpoint under the "Sending data to Tempo" section. * This field is required * Example: `tempo-us-central1.grafana.net:443` * **GRAFANA\_CLOUD\_TEMPO\_USERNAME** `string` : Username. You can find the username in the "Sending data to Tempo" section under the "basic\_auth" yaml field. The username is a number. * This field is required * **GRAFANA\_CLOUD\_TEMPO\_PASSWORD** `string` : Password (Api Token). This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `traces:write` scope. * This field is required * Example: `glc_eyJvIj...` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Grafana Cloud Tempo` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `grafanacloudtempo.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: grafanacloudtempo-example namespace: odigos-system spec: data: GRAFANA_CLOUD_TEMPO_ENDPOINT: GRAFANA_CLOUD_TEMPO_USERNAME: destinationName: grafanacloudtempo secretRef: name: grafanacloudtempo-secret signals: - TRACES type: grafanacloudtempo --- apiVersion: v1 data: GRAFANA_CLOUD_TEMPO_PASSWORD: kind: Secret metadata: name: grafanacloudtempo-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f grafanacloudtempo.yaml ``` # GreptimeDB Source: https://docs.odigos.io/oss/backends/greptime Configuring the GreptimeDB backend (Managed) ### Getting Started

### Configuring Destination Fields ❌ Traces ✅ Metrics ❌ Logs ❌ Profiles * **GREPTIME\_ENDPOINT** `string` : OTLP HTTP Endpoint. * This field is required * Example: `https://..aws.greptime.cloud` * **GREPTIME\_DB\_NAME** `string` : Database Name. * This field is required * **GREPTIME\_BASIC\_USERNAME** `string` : Basic Auth Username. The left part (from ":") of the Basic Auth Token. * This field is required * **GREPTIME\_BASIC\_PASSWORD** `string` : Basic Auth Password. The right part (from ":") of the Basic Auth Token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `GreptimeDB` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `greptime.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: greptime-example namespace: odigos-system spec: data: GREPTIME_BASIC_USERNAME: GREPTIME_DB_NAME: GREPTIME_ENDPOINT: destinationName: greptime secretRef: name: greptime-secret signals: - METRICS type: greptime --- apiVersion: v1 data: GREPTIME_BASIC_PASSWORD: kind: Secret metadata: name: greptime-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f greptime.yaml ``` # Groundcover inCloud Source: https://docs.odigos.io/oss/backends/groundcover Configuring the Groundcover inCloud backend (Managed) ### Getting Started

To get an API Key, refer to [these docs](https://docs.groundcover.com/architecture/byoc/ingestion-endpoints#creating-an-ingestion-key). ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **GROUNDCOVER\_ENDPOINT** `string` : Groundcover inCloud Site. The format is `host:port`. Host is required, also known as your `inCloud_Site`, it is part of the configuration provided to you by Groundcover when setting up inCloud Managed. Port is optional, and defaults to the default OpenTelemetry gRPC port `4317`. * This field is required * **GROUNDCOVER\_API\_KEY** `string` : Groundcover API Key. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Groundcover inCloud` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `groundcover.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: groundcover-example namespace: odigos-system spec: data: GROUNDCOVER_ENDPOINT: destinationName: groundcover secretRef: name: groundcover-secret signals: - TRACES - METRICS - LOGS type: groundcover --- apiVersion: v1 data: GROUNDCOVER_API_KEY: kind: Secret metadata: name: groundcover-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f groundcover.yaml ``` # Honeycomb Source: https://docs.odigos.io/oss/backends/honeycomb Configuring the Honeycomb backend (Managed) ### Getting Started

**Where to find the data** * Each Odigos destination send collected telemetry to a **single environment** in Honeycomb. * The collected telemetry will go into Honeycomb [datasets](https://docs.honeycomb.io/configure/datasets/manage/) based on the `service.name` resource attribute. odigos will set it by default to the k8s workload (deployment) name (or as set in source configuration). **Login to Honeycomb**
Go to the **[🔗 website](https://ui.honeycomb.io)** and login/create your honeycomb account. You can start with the Free plan to test sending traces, logs and metrics without needing to provide a credit card. **Select Environment**
Honeycomb uses [environments](https://docs.honeycomb.io/configure/environments/manage/) to partition and organize data. Common environments includes `staging`, `production`, `development`, `integration`, or you might want to include the cluster/unit name to group and isolate telemetry that is unrelated. Each Odigos Honeycomb destination always sends the data to a **single environment**, based on the "API Key" it is configured with. Before creating your API Key, make sure to select in ui the environment you want your odigos collected telemetry to show up in. API Keys settings page in Honeycomb

**Obtaining Honeycomb API Key** From the top left corner click on the ENVIRONMENT, and click the **Settings** configuration gear icon to go to the settings page. API Keys settings page in Honeycomb

From the Settings page, click on the **API Keys** tab, make sure you are on "Ingest" sub-tab and click on the **Create Ingest API Key** blue button on the right. API Keys settings page in Honeycomb

Give your API Key a name (include "odigos" in the name to identify it easily later on), make sure the checkbox for "Can create services/datasets" is checked, and click the **Create** button on the modal bottom right. API Keys settings page in Honeycomb

Copy the ingest key to keyboard and paste it in your Honeycomb Odigos destination configuration (ui or kubernetes manifest). ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **HONEYCOMB\_API\_KEY** `string` : API Key. * This field is required * **HONEYCOMB\_ENDPOINT** `string` : Endpoint. Choose one of the endpoints in the dropdown (`api.honeycomb.io` is the US instance, `api.eu1.honeycomb.io` is EU instance) * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Honeycomb` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `honeycomb.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: honeycomb-example namespace: odigos-system spec: data: HONEYCOMB_ENDPOINT: '' destinationName: honeycomb secretRef: name: honeycomb-secret signals: - TRACES - METRICS - LOGS type: honeycomb --- apiVersion: v1 data: HONEYCOMB_API_KEY: kind: Secret metadata: name: honeycomb-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f honeycomb.yaml ``` # HyperDX Source: https://docs.odigos.io/oss/backends/hyperdx Configuring the HyperDX backend (Managed) ### Getting Started hyperdx

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **HYPERDX\_API\_KEY** `string` : HyperDX API Key. * This field is required * **HYPERDX\_LOG\_NORMALIZER** `boolean` : Enable Log Normalizer. Enable HyperDX log normalization processor. This parses JSON from log bodies, infers severity levels, and normalizes log attributes for better querying. * This field is optional and defaults to `True` We handle the endpoint internally, so you don't need to provide it. * The endpoint is `in-otel.hyperdx.io:4317` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `HyperDX` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `hyperdx.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: hyperdx-example namespace: odigos-system spec: data: {} destinationName: hyperdx secretRef: name: hyperdx-secret signals: - TRACES - METRICS - LOGS type: hyperdx --- apiVersion: v1 data: HYPERDX_API_KEY: kind: Secret metadata: name: hyperdx-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f hyperdx.yaml ``` # IBM Instana Source: https://docs.odigos.io/oss/backends/instana Configuring the IBM Instana backend (Managed) ### Getting Started instana

**Creating Account**
Go to the **[🔗 Website](https://www.ibm.com/products/instana?mhsrc=ibmsearch_a\&mhq=instana) > Account** and click **Log In** **Obtaining Endpoint**
Go to the **[📖 Docs](https://www.ibm.com/docs/en/instana-observability/current?topic=instana-backend#endpoints-of-the-instana-backend-otlp-acceptor) > Endpoints of the Instana backend otlp-acceptor** and find your **Endpoint** **Obtaining Agent Key**
Go to your **📩 Email > Instana Login Details** and copy **Agent-Key** ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **INSTANA\_ENDPOINT** `string` : OTLP gRPC Endpoint. Domain name of the otlp-acceptor component of the Instana backend. * This field is required * Example: `otlp-{color}-saas.instana.io` * **INSTANA\_AGENT\_KEY** `string` : Agent Key. The agent key of the Instana for targeting the Instana backend. * This field is required For ingesting logs via OpenTelemetry, an add-on to the Instana license is required. It’s advisable to contact your IBM Sales representative to include this add-on in your account. ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `IBM Instana` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `instana.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: instana-example namespace: odigos-system spec: data: INSTANA_ENDPOINT: destinationName: instana secretRef: name: instana-secret signals: - TRACES - METRICS - LOGS type: instana --- apiVersion: v1 data: INSTANA_AGENT_KEY: kind: Secret metadata: name: instana-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f instana.yaml ``` # Jaeger Source: https://docs.odigos.io/oss/backends/jaeger Configuring the Jaeger backend (Self-Hosted) ### Getting Started jaeger

Version v1.35 of Jaeger introduced the ability to receive OpenTelemetry trace data via the OpenTelemetry Protocol (OTLP). This allows to create a new Jaeger backend by simply specifying the Jaeger OTLP gRPC unencrypted URL. ### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **JAEGER\_URL** `string` : Jaeger OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317` * This field is required * **JAEGER\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection * This field is optional and defaults to `False` * **JAEGER\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA * This field is optional * Example: `-----BEGIN CERTIFICATE-----` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Jaeger` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `jaeger.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: jaeger-example namespace: odigos-system spec: data: JAEGER_URL: # Note: The commented fields below are optional. # JAEGER_TLS_ENABLED: # JAEGER_CA_PEM: destinationName: jaeger signals: - TRACES type: jaeger ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f jaeger.yaml ``` # Kafka Source: https://docs.odigos.io/oss/backends/kafka Configuring the Kafka backend (Self-Hosted) ### Getting Started kafka

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **KAFKA\_PROTOCOL\_VERSION** `string` : Protocol Version. Kafka protocol version. * This field is required * Example: `2.0.0` * **KAFKA\_BROKERS** `string[]` : Brokers. The list of kafka brokers * This field is optional and defaults to `["localhost:9092"]` * **KAFKA\_RESOLVE\_CANONICAL\_BOOTSTRAP\_SERVERS\_ONLY** `boolean` : Resolve Canonical Bootstrap Servers Only. Whether to resolve then reverse-lookup broker IPs during startup. * This field is optional and defaults to `False` * **KAFKA\_CLIENT\_ID** `string` : Client ID. The client ID to configure the Sarama Kafka client with. The client ID will be used for all produce requests. * This field is optional and defaults to `sarama` * **KAFKA\_TOPIC** `string` : Topic. The name of the default kafka topic to export to (default = otlp\_spans for traces, otlp\_metrics for metrics, otlp\_logs for logs). * This field is optional * **KAFKA\_TOPIC\_FROM\_ATTRIBUTE** `string` : Topic from Attribute. Specify the resource attribute whose value should be used as the message's topic. * This field is optional * **KAFKA\_ENCODING** `string` : Encoding. The encoding of the traces sent to kafka. * This field is optional and defaults to `otlp_proto` * **KAFKA\_PARTITION\_TRACES\_BY\_ID** `boolean` : Partition Traces by ID. Configures the exporter to include the trace ID as the message key in trace messages sent to kafka. Please note: this setting does not have any effect on Jaeger encoding exporters since Jaeger exporters include trace ID as the message key by default. * This field is optional and defaults to `False` * **KAFKA\_PARTITION\_METRICS\_BY\_RESOURCE\_ATTRIBUTES** `boolean` : Partition Metrics by Resource Attributes. Configures the exporter to include the hash of sorted resource attributes as the message partitioning key in metric messages sent to kafka. * This field is optional and defaults to `False` * **KAFKA\_PARTITION\_LOGS\_BY\_RESOURCE\_ATTRIBUTES** `boolean` : Partition Logs by Resource Attributes. Configures the exporter to include the hash of sorted resource attributes as the message partitioning key in log messages sent to kafka. * This field is optional and defaults to `False` * **KAFKA\_AUTH\_METHOD** `string` : Auth Method. The auth method to use. * This field is required and defaults to `none` * **KAFKA\_USERNAME** `string` : Username. The username to use. * This field is optional * **KAFKA\_PASSWORD** `string` : Password. The password to use. * This field is optional * **KAFKA\_METADATA\_FULL** `boolean` : Metadata Full. Whether to maintain a full set of metadata. When disabled, the client does not make the initial request to broker at the startup. * This field is optional and defaults to `False` * **KAFKA\_METADATA\_MAX\_RETRY** `string` : Metadata Max Retry. The number of retries to get metadata. * This field is optional and defaults to `3` * **KAFKA\_METADATA\_BACKOFF\_RETRY** `string` : Metadata Backoff Retry. How long to wait between metadata retries. * This field is optional and defaults to `250ms` * **KAFKA\_TIMEOUT** `string` : Timeout. Is the timeout for every attempt to send data to the backend. * This field is optional and defaults to `5s` * **KAFKA\_RETRY\_ON\_FAILURE\_ENABLED** `boolean` : Enable Retry on Failure. * This field is optional and defaults to `True` * **KAFKA\_RETRY\_ON\_FAILURE\_INITIAL\_INTERVAL** `string` : Initial Interval. Time to wait after the first failure before retrying; ignored if `enabled` is `false`. * This field is optional and defaults to `5s` * **KAFKA\_RETRY\_ON\_FAILURE\_MAX\_INTERVAL** `string` : Max Interval. Is the upper bound on backoff; ignored if `enabled` is `false`. * This field is optional and defaults to `30s` * **KAFKA\_RETRY\_ON\_FAILURE\_MAX\_ELAPSED\_TIME** `string` : Max Elapsed Time. Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false`. * This field is optional and defaults to `120s` * **KAFKA\_PRODUCER\_MAX\_MESSAGE\_BYTES** `string` : Producer Max Message Bytes. The maximum permitted size of a message in bytes. * This field is optional and defaults to `1000000` * **KAFKA\_PRODUCER\_REQUIRED\_ACKS** `string` : Producer Required Acks. Controls when a message is regarded as transmitted. * This field is optional and defaults to `1` * **KAFKA\_PRODUCER\_COMPRESSION** `string` : Producer Compression. The compression used when producing messages to kafka. * This field is optional and defaults to `none` * **KAFKA\_PRODUCER\_FLUSH\_MAX\_MESSAGES** `string` : Producer Flush Max Messages. The maximum number of messages the producer will send in a single broker request. * This field is optional and defaults to `0` The destination topic can be defined in a few different ways and takes priority in the following order: 1. When `topic_from_attribute` is configured, and the corresponding attribute is found on the ingested data, the value of this attribute is used. 2. If a prior component in the collector pipeline sets the topic on the context via the `topic.WithTopic` function (from the [github.com/open-telemetry/opentelemetry-collector-contrib/pkg/kafka/topic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/kafka/topic) package), the value set in the context is used. 3. Finally, the topic configuration is used as a default/fallback destination. ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Kafka` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `kafka.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: kafka-example namespace: odigos-system spec: data: KAFKA_AUTH_METHOD: '' KAFKA_PROTOCOL_VERSION: # Note: The commented fields below are optional. # KAFKA_BROKERS: # KAFKA_RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY: # KAFKA_CLIENT_ID: # KAFKA_TOPIC: # KAFKA_TOPIC_FROM_ATTRIBUTE: # KAFKA_ENCODING: # KAFKA_PARTITION_TRACES_BY_ID: # KAFKA_PARTITION_METRICS_BY_RESOURCE_ATTRIBUTES: # KAFKA_PARTITION_LOGS_BY_RESOURCE_ATTRIBUTES: # KAFKA_USERNAME: # KAFKA_METADATA_FULL: # KAFKA_METADATA_MAX_RETRY: # KAFKA_METADATA_BACKOFF_RETRY: # KAFKA_TIMEOUT: # KAFKA_RETRY_ON_FAILURE_ENABLED: # KAFKA_RETRY_ON_FAILURE_INITIAL_INTERVAL: # KAFKA_RETRY_ON_FAILURE_MAX_INTERVAL: # KAFKA_RETRY_ON_FAILURE_MAX_ELAPSED_TIME: # KAFKA_PRODUCER_MAX_MESSAGE_BYTES: # KAFKA_PRODUCER_REQUIRED_ACKS: # KAFKA_PRODUCER_COMPRESSION: # KAFKA_PRODUCER_FLUSH_MAX_MESSAGES: destinationName: kafka # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: kafka-secret signals: - TRACES - METRICS - LOGS type: kafka --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # KAFKA_PASSWORD: # kind: Secret # metadata: # name: kafka-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f kafka.yaml ``` # KloudMate Source: https://docs.odigos.io/oss/backends/kloudmate Configuring the KloudMate backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **KLOUDMATE\_API\_KEY** `string` : KloudMate API Key. * This field is required We handle the endpoint internally, so you don't need to provide it. * The endpoint is `https://otel.kloudmate.com:4318` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `KloudMate` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `kloudmate.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: kloudmate-example namespace: odigos-system spec: data: {} destinationName: kloudmate secretRef: name: kloudmate-secret signals: - TRACES - METRICS - LOGS type: kloudmate --- apiVersion: v1 data: KLOUDMATE_API_KEY: kind: Secret metadata: name: kloudmate-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f kloudmate.yaml ``` # Last9 Source: https://docs.odigos.io/oss/backends/last9 Configuring the Last9 backend (Managed) ### Getting Started last9

[Click here](https://app.last9.io/integrations?category=all\&integration=OpenTelemetry) to visit the Last9 OpenTelemetry integration page. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **LAST9\_OTLP\_ENDPOINT** `string` : Last9 OpenTelemetry Endpoint. * This field is required * **LAST9\_OTLP\_BASIC\_AUTH\_HEADER** `string` : Basic Auth Header. * This field is required * Example: `Basic ...` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Last9` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `last9.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: last9-example namespace: odigos-system spec: data: LAST9_OTLP_ENDPOINT: destinationName: last9 secretRef: name: last9-secret signals: - TRACES - METRICS - LOGS type: last9 --- apiVersion: v1 data: LAST9_OTLP_BASIC_AUTH_HEADER: kind: Secret metadata: name: last9-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f last9.yaml ``` # Lightstep Source: https://docs.odigos.io/oss/backends/lightstep Configuring the Lightstep backend (Managed) ### Getting Started

**Obtaining Lightstep Access Token**
Go to **⚙️ > Access Tokens** and click **Create New** ### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **LIGHTSTEP\_ACCESS\_TOKEN** `string` : Access Token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Lightstep` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `lightstep.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: lightstep-example namespace: odigos-system spec: data: {} destinationName: lightstep secretRef: name: lightstep-secret signals: - TRACES type: lightstep --- apiVersion: v1 data: LIGHTSTEP_ACCESS_TOKEN: kind: Secret metadata: name: lightstep-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f lightstep.yaml ``` # Logz.io Source: https://docs.odigos.io/oss/backends/logzio Configuring the Logz.io backend (Managed) ### Getting Started logzio

**Finding Logz.io Region** You can find your account’s region by selecting **⚙️ > Settings > General** from the top menu. Your two-letter region code is at the start of the region designation. Another option is to sign in to your Logz.io account and look at the URL in the address bar. If you see `app.logz.io`, your account is hosted in the US East (Northern Virginia). All other regions have a two-letter region code. For example, if you see `app-eu.logz.io`, then your account is in the Europe (Frankfurt) region. **Obtaining Tokens** From your account, go to the **Manage Tokens > Data shipping** tokens tab of your Operations workspace. It can be reached by selecting **⚙️ > Settings > Manage tokens** in the ADMIN ZONE section of the side navigation menu. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **LOGZIO\_REGION** `string` : Region. The region of your Logz.io account * This field is required * **LOGZIO\_TRACING\_TOKEN** `string` : Tracing token. The token you obtained from Logz.io * This field is optional * **LOGZIO\_METRICS\_TOKEN** `string` : Metrics token. The token you obtained from Logz.io * This field is optional * **LOGZIO\_LOGS\_TOKEN** `string` : Logs token. The token you obtained from Logz.io * This field is optional ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Logz.io` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `logzio.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: logzio-example namespace: odigos-system spec: data: LOGZIO_REGION: '' destinationName: logzio # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: logzio-secret signals: - TRACES - METRICS - LOGS type: logzio --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # LOGZIO_LOGS_TOKEN: # LOGZIO_METRICS_TOKEN: # LOGZIO_TRACING_TOKEN: # kind: Secret # metadata: # name: logzio-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f logzio.yaml ``` # Loki Source: https://docs.odigos.io/oss/backends/loki Configuring the Loki backend (Self-Hosted) ### Getting Started loki

These instructions are for the **self hosted** Loki instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Loki](../backends/grafanacloudloki) page. This destination is for self hosted Loki instances. **Loki Labels** Loki labels are used to "index" logs and allow efficient filtering in the Grafana UI. You can read more about them [here](https://grafana.com/docs/loki/latest/get-started/labels/). You can configure your grafana cloud loki labels in the `Labels` multi-input config option in the destination page in the Odigos UI. * Labels should be names of OpenTelemetry resource attribute or log record attribute keys that you want to index, for example: `"k8s.namespace.name"`. * To avoid performance issues, it is recommended to limit the number of labels to a reasonable amount. * [Use only low cardinality values](https://grafana.com/docs/loki/latest/get-started/labels/#cardinality). e.g. use only OpenTelemetry attributes for which you expect a small number of unique values like `http.response.status_code` but **not** `network.peer.address`. * If the label is not present in a log record, it will be ignored. * The default labels if not set are: `k8s.container.name, k8s.pod.name, k8s.namespace.name` ### Configuring Destination Fields ❌ Traces ❌ Metrics ✅ Logs ❌ Profiles * **LOKI\_URL** `string` : Endpoint. If scheme is not provided, it defaults to `http`. If port is not provided, it defaults to `3100`. If path is not provided, it defaults to `/otlp`. * This field is required * Example: `http://:/otlp` * **LOKI\_SCOPE\_ORG\_ID** `string` : X-Scope-OrgID. use this X-Scope-OrgID if you need to scope the data to a specific org. Multiple orgs can be chained using `|` * This field is optional * Example: `org1|org2|org3` * **LOKI\_USERNAME** `string` : Basic Auth Username. username for basic authentication * This field is optional * Example: `` * **LOKI\_PASSWORD** `string` : Basic Auth Password. password for basic authentication * This field is optional * Example: `` * **LOKI\_LABELS** `string[]` : Labels. use these OpenTelemetry resource attributes as loki labels for each log record * This field is required and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]` The supported Loki version is V3. Please make sure you are using the correct version, and that you have enabled native OTLP support. As part of this version, the path `/loki/api/v1/push` has been deprecated and replaced with `/otlp`. ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Loki` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `loki.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: loki-example namespace: odigos-system spec: data: LOKI_LABELS: '' LOKI_URL: # Note: The commented fields below are optional. # LOKI_SCOPE_ORG_ID: # LOKI_USERNAME: destinationName: loki # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: loki-secret signals: - LOGS type: loki --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # LOKI_PASSWORD: # kind: Secret # metadata: # name: loki-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f loki.yaml ``` # Lumigo Source: https://docs.odigos.io/oss/backends/lumigo Configuring the Lumigo backend (Managed) ### Getting Started lumigo

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **LUMIGO\_ENDPOINT** `string` : Lumigo OTLP HTTP Endpoint. * This field is required and defaults to `https://ga-otlp.lumigo-tracer-edge.golumigo.com` * **LUMIGO\_TOKEN** `string` : Lumigo Authorization Token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Lumigo` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `lumigo.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: lumigo-example namespace: odigos-system spec: data: LUMIGO_ENDPOINT: '' destinationName: lumigo secretRef: name: lumigo-secret signals: - TRACES - METRICS - LOGS type: lumigo --- apiVersion: v1 data: LUMIGO_TOKEN: kind: Secret metadata: name: lumigo-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f lumigo.yaml ``` # Middleware Source: https://docs.odigos.io/oss/backends/middleware Configuring the Middleware backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **MW\_TARGET** `string` : Endpoint. * This field is required * **MW\_API\_KEY** `string` : API Key. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Middleware` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `middleware.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: middleware-example namespace: odigos-system spec: data: MW_TARGET: destinationName: middleware secretRef: name: middleware-secret signals: - TRACES - METRICS - LOGS type: middleware --- apiVersion: v1 data: MW_API_KEY: kind: Secret metadata: name: middleware-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f middleware.yaml ``` # New Relic Source: https://docs.odigos.io/oss/backends/newrelic Configuring the New Relic backend (Managed) ### Getting Started

**Obtaining a License Key**
On the left sidebar, click on your account name and select **API Keys** and then click on **Create a key**. New Relic API keys

For key type select **Ingest - License**, give a name to your key and press **Create a key**. Create an API key screen in New Relic

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **NEWRELIC\_API\_KEY** `string` : License Key. * This field is required * **NEWRELIC\_ENDPOINT** `string` : Endpoint. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `New Relic` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `newrelic.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: newrelic-example namespace: odigos-system spec: data: NEWRELIC_ENDPOINT: '' destinationName: newrelic secretRef: name: newrelic-secret signals: - TRACES - METRICS - LOGS type: newrelic --- apiVersion: v1 data: NEWRELIC_API_KEY: kind: Secret metadata: name: newrelic-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f newrelic.yaml ``` # Observe Source: https://docs.odigos.io/oss/backends/observe Configuring the Observe backend (Managed) ### Getting Started observe

**Creating an ingest token**
Data is sent to Observe's OTLP collection endpoint, which requires a **Datastream ingest token** (a user/API token will be rejected with `401`). Create one in the Observe UI under **Datastreams**, following [Configure your own OTel collector without Kubernetes](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment). **Collection endpoint**
By default the endpoint is `https://.collect.observeinc.com/v2/otel`. For regional tenants or other overrides, set the optional `OBSERVE_ENDPOINT` field. See [Configure your own OTel collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) for the endpoint format. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **OBSERVE\_CUSTOMER\_ID** `string` : Customer ID. The Customer ID is the first part of your Observe URL. For example, if your Observe URL is `https://.observe.com`, then the Customer ID is ``. * This field is required * **OBSERVE\_TOKEN** `string` : Token. A Datastream ingest token (not a user/API token). Create one in the Observe UI under "Datastreams". See [https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment) for how to create an ingest token. * This field is required * **OBSERVE\_ENDPOINT** `string` : Collection Endpoint. Optional. The full OTLP/HTTP collection endpoint. Set this to override the default, for example for a regional tenant such as `https://.collect.eu1.observeinc.com/v2/otel`. When empty, it is derived from the Customer ID. See [https://docs.observeinc.com/docs/configure-your-own-otel-collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) for the endpoint format. * This field is optional * Example: `https://.collect.observeinc.com/v2/otel` By default the endpoint is derived from your Customer ID as `https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/v2/otel`. To send to a different endpoint (for example a regional tenant), set the optional `OBSERVE_ENDPOINT` field. For details on the collection endpoint and creating an ingest token, see Observe's guides for [configuring your own OTel collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) and [configuring it without Kubernetes](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment). ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Observe` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `observe.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: observe-example namespace: odigos-system spec: data: OBSERVE_CUSTOMER_ID: # Note: The commented fields below are optional. # OBSERVE_ENDPOINT: destinationName: observe secretRef: name: observe-secret signals: - TRACES - METRICS - LOGS type: observe --- apiVersion: v1 data: OBSERVE_TOKEN: kind: Secret metadata: name: observe-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f observe.yaml ``` # OneUptime Source: https://docs.odigos.io/oss/backends/oneuptime Configuring the OneUptime backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **ONEUPTIME\_INGESTION\_KEY** `string` : Telemetry Ingestion Key. The Telemetry Ingestion Key is used to authenticate your data. You can find it in the OneUptime dashboard under Project Settings > Ingestion Keys. * This field is required We handle the endpoint internally, so you don't need to provide it. * The endpoint is `https://oneuptime.com/otlp` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `OneUptime` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `oneuptime.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: oneuptime-example namespace: odigos-system spec: data: {} destinationName: oneuptime secretRef: name: oneuptime-secret signals: - TRACES - METRICS - LOGS type: oneuptime --- apiVersion: v1 data: ONEUPTIME_INGESTION_KEY: kind: Secret metadata: name: oneuptime-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f oneuptime.yaml ``` # OpenObserve Source: https://docs.odigos.io/oss/backends/openobserve Configuring the OpenObserve backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **OPEN\_OBSERVE\_ENDPOINT** `string` : OTLP HTTP Endpoint. * This field is required * Example: `https://api.openobserve.ai/api/` * **OPEN\_OBSERVE\_API\_KEY** `string` : API Key. Do not include the `Basic` prefix. * This field is required * **OPEN\_OBSERVE\_STREAM\_NAME** `string` : Stream Name. * This field is required and defaults to `default` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `OpenObserve` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `openobserve.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: openobserve-example namespace: odigos-system spec: data: OPEN_OBSERVE_ENDPOINT: OPEN_OBSERVE_STREAM_NAME: '' destinationName: openobserve secretRef: name: openobserve-secret signals: - TRACES - LOGS type: openobserve --- apiVersion: v1 data: OPEN_OBSERVE_API_KEY: kind: Secret metadata: name: openobserve-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f openobserve.yaml ``` # Oracle Cloud Source: https://docs.odigos.io/oss/backends/oracle Configuring the Oracle Cloud backend (Managed) ### Getting Started oracle

### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **ORACLE\_ENDPOINT** `string` : OTLP HTTP Endpoint. The data upload endpoint of the Oracle APM domain. * This field is required * **ORACLE\_DATA\_KEY** `string` : Data Key. * This field is required * **ORACLE\_DATA\_KEY\_TYPE** `string` : Data Key Type. The type of the data key, relevant for Traces. Metrics are always authenticated with a private data key. * This field is required and defaults to `private` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Oracle Cloud` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `oracle.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: oracle-example namespace: odigos-system spec: data: ORACLE_DATA_KEY_TYPE: '' ORACLE_ENDPOINT: destinationName: oracle secretRef: name: oracle-secret signals: - TRACES - METRICS type: oracle --- apiVersion: v1 data: ORACLE_DATA_KEY: kind: Secret metadata: name: oracle-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f oracle.yaml ``` # OTLP gRPC Source: https://docs.odigos.io/oss/backends/otlp Configuring the OTLP gRPC backend (Self-Hosted) ### Getting Started otlp

For advanced users trying to implement complex observability pipelines, Odigos support sending data to any OTLP gRPC unencrypted endpoint. Note that if your backend expects [OTLP over http](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) you should use the [OTLP http](../backends/otlphttp) destination instead. If your backend is supported natively in Odigos, it's recommended to use the native integration. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ✅ Profiles * **OTLP\_GRPC\_ENDPOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`. * This field is required * Example: `host:port` * **OTLP\_GRPC\_COMPRESSION** `string` : Destination Compression Type. Compression type to use for the destination. The default is `none`. The compression type must be supported by the destination. * This field is optional and defaults to `none` * **OTLP\_GRPC\_HEADERS** `{ key: string; value: string; }[]` : Headers. Headers is the option to set custom GRPC headers for OTLP GRPC destination. If specified, please provide each header in the format: key:value. Multiple headers can be added. Keys must be non-empty strings and follow standard HTTP header conventions. Values must be non-empty strings and may include alphanumerics, whitespace, and standard punctuation. * This field is optional * **OTLP\_GRPC\_OAUTH2\_ENABLED** `boolean` : Enable OAuth2. Enable OAuth2 client credentials authentication * This field is optional and defaults to `False` * **OTLP\_GRPC\_OAUTH2\_CLIENT\_ID** `string` : OAuth2 Client ID. OAuth2 client identifier for client credentials flow * This field is optional * Example: `your-client-id` * **OTLP\_GRPC\_OAUTH2\_CLIENT\_SECRET** `string` : OAuth2 Client Secret. OAuth2 client secret for client credentials flow * This field is optional * Example: `your-client-secret` * **OTLP\_GRPC\_OAUTH2\_TOKEN\_URL** `string` : OAuth2 Token URL. OAuth2 token endpoint URL for obtaining access tokens * This field is optional * Example: `https://example.com/oauth2/token` * **OTLP\_GRPC\_OAUTH2\_SCOPES** `string` : OAuth2 Scopes. Comma-separated list of OAuth2 scopes to request (e.g., "api.metrics,api.traces") * This field is optional * Example: `api.metrics,api.traces` * **OTLP\_GRPC\_OAUTH2\_AUDIENCE** `string` : OAuth2 Audience. OAuth2 audience parameter for token requests * This field is optional * Example: `api.example.com` * **OTLP\_GRPC\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection * This field is optional and defaults to `False` * **OTLP\_GRPC\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA * This field is optional * Example: `-----BEGIN CERTIFICATE-----` * **OTLP\_GRPC\_MTLS\_ENABLED** `boolean` : Enable mTLS. Use a client certificate and key for mutual TLS authentication with the destination * This field is optional and defaults to `False` * **OTLP\_GRPC\_CLIENT\_CERT\_PEM** `string` : Client Certificate. Client certificate in PEM format for mutual TLS authentication * This field is optional * Example: `-----BEGIN CERTIFICATE-----` * **OTLP\_GRPC\_CLIENT\_KEY\_PEM** `string` : Client Key. Client private key in PEM format for mutual TLS authentication * This field is optional * Example: `-----BEGIN PRIVATE KEY-----` * **OTLP\_GRPC\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification * This field is optional and defaults to `False` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `OTLP gRPC` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `otlp.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: otlp-example namespace: odigos-system spec: data: OTLP_GRPC_ENDPOINT: # Note: The commented fields below are optional. # OTLP_GRPC_COMPRESSION: # OTLP_GRPC_HEADERS: # OTLP_GRPC_OAUTH2_ENABLED: # OTLP_GRPC_OAUTH2_CLIENT_ID: # OTLP_GRPC_OAUTH2_TOKEN_URL: # OTLP_GRPC_OAUTH2_SCOPES: # OTLP_GRPC_OAUTH2_AUDIENCE: # OTLP_GRPC_TLS_ENABLED: # OTLP_GRPC_CA_PEM: # OTLP_GRPC_MTLS_ENABLED: # OTLP_GRPC_INSECURE_SKIP_VERIFY: destinationName: otlp # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: otlp-secret signals: - TRACES - METRICS - LOGS - PROFILES type: otlp --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # OTLP_GRPC_CLIENT_CERT_PEM: # OTLP_GRPC_CLIENT_KEY_PEM: # OTLP_GRPC_OAUTH2_CLIENT_SECRET: # kind: Secret # metadata: # name: otlp-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f otlp.yaml ``` # OTLP http Source: https://docs.odigos.io/oss/backends/otlphttp Configuring the OTLP http backend (Self-Hosted) ### Getting Started

For advanced users trying to implement complex observability pipelines, Odigos support sending data to any OTLP http endpoint. Note that if your backend expects [OTLP over gRPC](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) you should use the [OTLP gRPC](../backends/otlp) destination instead. If your backend is supported natively in Odigos, it's recommended to use the native integration. #### Using Basic Authentication This section is relevant if your OTLP http endpoint requires [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) (username and password). To configure basic authentication, use the optional config options `Basic Auth Username` and `Basic Auth Password`. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ✅ Profiles * **OTLP\_HTTP\_ENDPOINT** `string` : OTLP http Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`. * This field is required * Example: `http://host:port` * **OTLP\_HTTP\_BASIC\_AUTH\_USERNAME** `string` : Basic Auth Username. in case the otlp receiver requires basic auth, this is the username * This field is optional * Example: `username` * **OTLP\_HTTP\_BASIC\_AUTH\_PASSWORD** `string` : Basic Auth Password. in case the otlp receiver requires basic auth, this is the password * This field is optional * Example: `password` * **OTLP\_HTTP\_OAUTH2\_ENABLED** `boolean` : Enable OAuth2. Enable OAuth2 client credentials authentication * This field is optional and defaults to `False` * **OTLP\_HTTP\_OAUTH2\_CLIENT\_ID** `string` : OAuth2 Client ID. OAuth2 client identifier for client credentials flow * This field is optional * Example: `your-client-id` * **OTLP\_HTTP\_OAUTH2\_CLIENT\_SECRET** `string` : OAuth2 Client Secret. OAuth2 client secret for client credentials flow * This field is optional * Example: `your-client-secret` * **OTLP\_HTTP\_OAUTH2\_TOKEN\_URL** `string` : OAuth2 Token URL. OAuth2 token endpoint URL for obtaining access tokens * This field is optional * Example: `https://example.com/oauth2/token` * **OTLP\_HTTP\_OAUTH2\_SCOPES** `string` : OAuth2 Scopes. Comma-separated list of OAuth2 scopes to request (e.g., "api.metrics,api.traces") * This field is optional * Example: `api.metrics,api.traces` * **OTLP\_HTTP\_OAUTH2\_AUDIENCE** `string` : OAuth2 Audience. OAuth2 audience parameter for token requests * This field is optional * Example: `api.example.com` * **OTLP\_HTTP\_COMPRESSION** `string` : Destination Compression Type. Compression type to use for the destination. The default is `none`. The compression type must be supported by the destination. * This field is optional and defaults to `none` * **OTLP\_HTTP\_HEADERS** `{ key: string; value: string; }[]` : Headers. Headers is the option to set custom HTTP headers for OTLP HTTP destination. If specified, please provide each header in the format: key:value. Multiple headers can be added. Keys must be non-empty strings and follow standard HTTP header conventions. Values must be non-empty strings and may include alphanumerics, whitespace, and standard punctuation. * This field is optional * **OTLP\_HTTP\_TRACES\_ENDPOINT** `string` : OTLP http traces Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/traces` will be used. * This field is optional * Example: `http://host:port/v1/traces` * **OTLP\_HTTP\_METRICS\_ENDPOINT** `string` : OTLP http metrics Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/metrics` will be used. * This field is optional * Example: `http://host:port/v1/metrics` * **OTLP\_HTTP\_LOGS\_ENDPOINT** `string` : OTLP http logs Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/logs` will be used. * This field is optional * Example: `http://host:port/v1/logs` * **OTLP\_HTTP\_PROFILES\_ENDPOINT** `string` : OTLP http profiles Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/profiles` will be used. * This field is optional * Example: `http://host:port/v1/profiles` * **OTLP\_HTTP\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection * This field is optional and defaults to `False` * **OTLP\_HTTP\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA * This field is optional * Example: `-----BEGIN CERTIFICATE-----` * **OTLP\_HTTP\_MTLS\_ENABLED** `boolean` : Enable mTLS. Use a client certificate and key for mutual TLS authentication with the destination * This field is optional and defaults to `False` * **OTLP\_HTTP\_CLIENT\_CERT\_PEM** `string` : Client Certificate. Client certificate in PEM format for mutual TLS authentication * This field is optional * Example: `-----BEGIN CERTIFICATE-----` * **OTLP\_HTTP\_CLIENT\_KEY\_PEM** `string` : Client Key. Client private key in PEM format for mutual TLS authentication * This field is optional * Example: `-----BEGIN PRIVATE KEY-----` * **OTLP\_HTTP\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification * This field is optional and defaults to `False` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `OTLP http` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `otlphttp.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: otlphttp-example namespace: odigos-system spec: data: OTLP_HTTP_ENDPOINT: # Note: The commented fields below are optional. # OTLP_HTTP_BASIC_AUTH_USERNAME: # OTLP_HTTP_OAUTH2_ENABLED: # OTLP_HTTP_OAUTH2_CLIENT_ID: # OTLP_HTTP_OAUTH2_TOKEN_URL: # OTLP_HTTP_OAUTH2_SCOPES: # OTLP_HTTP_OAUTH2_AUDIENCE: # OTLP_HTTP_COMPRESSION: # OTLP_HTTP_HEADERS: # OTLP_HTTP_TRACES_ENDPOINT: # OTLP_HTTP_METRICS_ENDPOINT: # OTLP_HTTP_LOGS_ENDPOINT: # OTLP_HTTP_PROFILES_ENDPOINT: # OTLP_HTTP_TLS_ENABLED: # OTLP_HTTP_CA_PEM: # OTLP_HTTP_MTLS_ENABLED: # OTLP_HTTP_INSECURE_SKIP_VERIFY: destinationName: otlphttp # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: otlphttp-secret signals: - TRACES - METRICS - LOGS - PROFILES type: otlphttp --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # OTLP_HTTP_BASIC_AUTH_PASSWORD: # OTLP_HTTP_CLIENT_CERT_PEM: # OTLP_HTTP_CLIENT_KEY_PEM: # OTLP_HTTP_OAUTH2_CLIENT_SECRET: # kind: Secret # metadata: # name: otlphttp-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f otlphttp.yaml ``` # Prometheus Source: https://docs.odigos.io/oss/backends/prometheus Configuring the Prometheus backend (Self-Hosted) ### Getting Started

These instructions are for the **self hosted** Prometheus instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Prometheus](../backends/grafanacloudprometheus) page. **Enabling Remote Write in Prometheus**
Currently, Odigos supports Prometheus via the Remote Write API. In order to enable remote write in Prometheus, set [the following feature flag](https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver). #### Using Authentication To enable authentication for Prometheus remote write, check the `Use Authentication` checkbox. Odigos supports two authentication methods: * **Bearer Token Authentication**: Use the `Bearer Token` field to provide a bearer token for authentication. This method is used if basic auth username is not provided. * **Basic Authentication**: Use the `Basic Auth Username` and `Basic Auth Password` fields if your Prometheus remote write endpoint requires [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). Basic auth takes precedence if username is provided. ### Configuring Destination Fields ❌ Traces ✅ Metrics ❌ Logs ❌ Profiles * **PROMETHEUS\_REMOTEWRITE\_URL** `string` : Remote Write URL. The remote write endpoint. If the URL ends with a known remote write path (`/api/v1/write` for Prometheus/Thanos or `/api/v1/push` for Mimir) it is used as-is; otherwise `/api/v1/write` is appended. * This field is required * **PROMETHEUS\_RESOURCE\_ATTRIBUTES\_LABELS** `string[]` : Resource Attributes Labels (Deprecated). deprecated. will be removed soon. * This field is optional * **PROMETHEUS\_USE\_AUTHENTICATION** `boolean` : Use Authentication. * This field is optional and defaults to `False` * **PROMETHEUS\_BEARER\_TOKEN** `string` : Bearer Token. use this bearer token if your prometheus remote write endpoint requires authentication * This field is optional * Example: `your-bearer-token` * **PROMETHEUS\_BASIC\_AUTH\_USERNAME** `string` : Basic Auth Username. username for basic authentication (if your prometheus remote write endpoint requires basic auth) * This field is optional * Example: `username` * **PROMETHEUS\_BASIC\_AUTH\_PASSWORD** `string` : Basic Auth Password. password for basic authentication (if your prometheus remote write endpoint requires basic auth) * This field is optional * Example: `password` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Prometheus` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `prometheus.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: prometheus-example namespace: odigos-system spec: data: PROMETHEUS_REMOTEWRITE_URL: # Note: The commented fields below are optional. # PROMETHEUS_RESOURCE_ATTRIBUTES_LABELS: # PROMETHEUS_USE_AUTHENTICATION: # PROMETHEUS_BASIC_AUTH_USERNAME: destinationName: prometheus # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: prometheus-secret signals: - METRICS type: prometheus --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # PROMETHEUS_BASIC_AUTH_PASSWORD: # PROMETHEUS_BEARER_TOKEN: # kind: Secret # metadata: # name: prometheus-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f prometheus.yaml ``` # Pyroscope Source: https://docs.odigos.io/oss/backends/pyroscope Send continuous profiles from Odigos to Grafana Pyroscope ### Getting Started

[Grafana Pyroscope](https://grafana.com/oss/pyroscope/) is an open-source continuous profiling backend. Odigos exports profile data over OTLP to Pyroscope so the same eBPF profiler that powers in-product views can also feed long-term, queryable storage. This destination only emits the **Profiles** signal. The destination must include `PROFILES`, and continuous profiling must be enabled for the agent you use: * **K8s Agent**: [Continuous Profiling overview](/enterprise/instrumentations/profiling/overview) and [getting started](/enterprise/instrumentations/profiling/getting-started) * **VM Agent**: [Continuous Profiling overview](/vmagent/instrumentations/profiling/overview) and [getting started](/vmagent/instrumentations/profiling/getting-started) **Pyroscope version**
Use a Pyroscope build that accepts OTLP profiles on the endpoint you configure. If you are unsure, upgrade to a recent release before configuring Odigos. **Quickstart: self-hosted Pyroscope**
If you don't already run Pyroscope, the upstream Helm chart gets you a working instance in a couple of commands: ```bash theme={null} helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install pyroscope grafana/pyroscope \ --namespace pyroscope --create-namespace ``` The distributor will be reachable in-cluster at `pyroscope.pyroscope:4040` — use that as the `PYROSCOPE_URL` value below. Provide the distributor's `host:port` only. Do **not** add `/otlp` or other path suffixes — Odigos sends OTLP profiles to `/v1development/profiles` on the host you configure. **TLS**
Toggle `PYROSCOPE_TLS_ENABLED` when Pyroscope is fronted by TLS. For plain traffic, leave it off. **Verifying the export**
After applying the destination and sending traffic to a profiled source, open the Pyroscope UI (`kubectl -n pyroscope port-forward svc/pyroscope 4040`) and confirm that profiles appear under your service. ### Configuring Destination Fields ❌ Traces ❌ Metrics ❌ Logs ✅ Profiles * **PYROSCOPE\_URL** `string` : Pyroscope Endpoint. Hostname and port of the Pyroscope distributor HTTP API (e.g. `pyroscope.pyroscope:4040`). Odigos sends OTLP profiles to `/v1development/profiles` on this host; do not add an `/otlp` path. * This field is required * Example: `host:port` * **PYROSCOPE\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection * This field is optional and defaults to `False` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Pyroscope` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `pyroscope.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: pyroscope-example namespace: odigos-system spec: data: PYROSCOPE_URL: # Note: The commented fields below are optional. # PYROSCOPE_TLS_ENABLED: destinationName: pyroscope signals: - PROFILES type: pyroscope ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f pyroscope.yaml ``` # qryn Source: https://docs.odigos.io/oss/backends/qryn Configuring the qryn backend (Self-Hosted) ### Getting Started qryn-oss

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **QRYN\_OSS\_URL** `string` : API Url. The API Endpoint for for your qryn instance * This field is required * Example: `http://qryn.local:3100` * **QRYN\_OSS\_PASSWORD** `string` : Basic auth password. The HTTP Basic Auth password for your qryn instance * This field is optional * **QRYN\_OSS\_USERNAME** `string` : Basic auth username. The HTTP Basic Auth username for your qryn instance * This field is optional * **QRYN\_OSS\_RESOURCE\_TO\_TELEMETRY\_CONVERSION** `boolean` : Convert container attributes to labels. * This field is optional and defaults to `True` * **QRYN\_OSS\_ADD\_EXPORTER\_NAME** `boolean` : Add exporter name to labels. * This field is optional and defaults to `True` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `qryn` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `qryn-oss.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: qryn-oss-example namespace: odigos-system spec: data: QRYN_OSS_URL: # Note: The commented fields below are optional. # QRYN_OSS_USERNAME: # QRYN_OSS_RESOURCE_TO_TELEMETRY_CONVERSION: # QRYN_OSS_ADD_EXPORTER_NAME: destinationName: qryn-oss # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: qryn-oss-secret signals: - TRACES - METRICS - LOGS type: qryn-oss --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # QRYN_OSS_PASSWORD: # kind: Secret # metadata: # name: qryn-oss-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f qryn-oss.yaml ``` # Quickwit Source: https://docs.odigos.io/oss/backends/quickwit Configuring the Quickwit backend (Self-Hosted) ### Getting Started

You'll find more details here: [https://quickwit.io/docs/log-management/otel-service](https://quickwit.io/docs/log-management/otel-service) ### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **QUICKWIT\_URL** `string` : Quickwit OTLP gRPC Endpoint. The unencrypted gRPC endpoint of the Quickwit OTLP receiver (indexer component). The default Quickwit gRPC port if not changed is `7281` * This field is required * Example: `quickwit.example.com:7281` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Quickwit` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `quickwit.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: quickwit-example namespace: odigos-system spec: data: QUICKWIT_URL: destinationName: quickwit signals: - TRACES - LOGS type: quickwit ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f quickwit.yaml ``` # Sentry Source: https://docs.odigos.io/oss/backends/sentry Configuring the Sentry backend ## Obtaining a Sentry DSN To use Sentry, you need to obtain a DSN (Data Source Name) from your Sentry account. You can find this in the Sentry UI under "Client Keys (DSN)". ## Configuring Sentry Backend Add a new backend with the DSN from your Sentry account. # Seq Source: https://docs.odigos.io/oss/backends/seq Configuring the Seq backend (Self-Hosted) ### Getting Started seq

* [Deploy Seq into Kubernetes using Docker](https://docs.datalust.co/docs/getting-started-with-docker) * [Deploy Seq into Kubernetes using Helm](https://docs.datalust.co/docs/using-helm) ### Configuring Destination Fields ✅ Traces ❌ Metrics ✅ Logs ❌ Profiles * **SEQ\_ENDPOINT** `string` : OTLP HTTP Endpoint. The format is `http://host:port`, `host` is required and equals the cluster internal DNS record of the Seq instance, `port` is required and equals any of the exposed ports of the Seq instance (defaults to 5341), do not add the rest of the path `/ingest/otlp/v1/*`. * This field is required * **SEQ\_API\_KEY** `string` : API Key. The API key is generated in the Seq instance. The API key must have the `Ingest` permission. * This field is optional ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Seq` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `seq.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: seq-example namespace: odigos-system spec: data: SEQ_ENDPOINT: destinationName: seq # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: seq-secret signals: - TRACES - LOGS type: seq --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # SEQ_API_KEY: # kind: Secret # metadata: # name: seq-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f seq.yaml ``` # SignalFx Source: https://docs.odigos.io/oss/backends/signalfx Configuring the SignalFx backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **SIGNALFX\_ACCESS\_TOKEN** `string` : Access Token. The authentication token provided by Splunk Observability Cloud. * This field is required * **SIGNALFX\_REALM** `string` : Realm. The SignalFx realm where your data will be sent (e.g., us0, us1, eu0). * This field is required * **SIGNALFX\_CA\_PEM** `string` : CA Certificate (PEM). Optional CA certificate in PEM format for TLS verification. * This field is optional * **SIGNALFX\_INSECURE\_SKIP\_VERIFY** `boolean` : Skip TLS Verification. If enabled, TLS certificate verification will be skipped. Not recommended for production. * This field is optional ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `SignalFx` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `signalfx.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: signalfx-example namespace: odigos-system spec: data: SIGNALFX_REALM: # Note: The commented fields below are optional. # SIGNALFX_INSECURE_SKIP_VERIFY: destinationName: signalfx # Uncomment the 'secretRef' below if you are using the optional Secret. # secretRef: # name: signalfx-secret signals: - TRACES - METRICS type: signalfx --- # The following Secret is optional. Uncomment the entire block if you need to use it. # apiVersion: v1 # data: # SIGNALFX_ACCESS_TOKEN: # SIGNALFX_CA_PEM: # kind: Secret # metadata: # name: signalfx-secret # namespace: odigos-system # type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f signalfx.yaml ``` # SigNoz Source: https://docs.odigos.io/oss/backends/signoz Configuring the SigNoz backend (Self-Hosted) ### Getting Started signoz

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **SIGNOZ\_URL** `string` : Collector Endpoint. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `SigNoz` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `signoz.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: signoz-example namespace: odigos-system spec: data: SIGNOZ_URL: destinationName: signoz signals: - TRACES - METRICS - LOGS type: signoz ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f signoz.yaml ``` # Splunk (SAPM) (Deprecated) Source: https://docs.odigos.io/oss/backends/splunk Configuring the Splunk (SAPM) (Deprecated) backend (Managed) ### Getting Started splunk

**Obtaining Access Token**
Go to **Settings** then **Access Tokens** and click **Create New Token** Splunk Access Tokens menu

Give the token a name and select **Ingest Token** for the authorization scope and click create. Create Access Token form in Splunk

### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token. * This field is required * **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`. * This field is required * **SPLUNK\_OTLP\_COMPRESSION** `string` : Compression Type. Compression type to use for the destination. The default is `none`. * This field is optional and defaults to `none` * **SPLUNK\_OTLP\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection * This field is optional and defaults to `False` * **SPLUNK\_OTLP\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA * This field is optional * Example: `-----BEGIN CERTIFICATE-----` * **SPLUNK\_OTLP\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification * This field is optional and defaults to `False` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Splunk (OTLP)` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `splunkotlp.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: splunkotlp-example namespace: odigos-system spec: data: SPLUNK_REALM: # Note: The commented fields below are optional. # SPLUNK_OTLP_COMPRESSION: # SPLUNK_OTLP_TLS_ENABLED: # SPLUNK_OTLP_CA_PEM: # SPLUNK_OTLP_INSECURE_SKIP_VERIFY: destinationName: splunkotlp secretRef: name: splunkotlp-secret signals: - TRACES type: splunkotlp --- apiVersion: v1 data: SPLUNK_ACCESS_TOKEN: kind: Secret metadata: name: splunkotlp-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f splunkotlp.yaml ``` # Splunk (SAPM) (Deprecated) Source: https://docs.odigos.io/oss/backends/splunksapm Configuring the Splunk (SAPM) (Deprecated) backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs * **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token. * This field is required * **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](../cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Splunk (SAPM) (Deprecated)` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `splunksapm.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: splunksapm-example namespace: odigos-system spec: data: SPLUNK_REALM: destinationName: splunksapm secretRef: name: splunksapm-secret signals: - TRACES type: splunksapm --- apiVersion: v1 data: SPLUNK_ACCESS_TOKEN: kind: Secret metadata: name: splunksapm-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f splunksapm.yaml ``` # Sumo Logic Source: https://docs.odigos.io/oss/backends/sumologic Configuring the Sumo Logic backend (Managed) ### Getting Started

Follow the [following instructions](https://help.sumologic.com/docs/send-data/hosted-collectors/http-source/otlp/) to obtain a OTLP/HTTP Source URL. ### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **SUMOLOGIC\_COLLECTION\_URL** `string` : Source URL. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Sumo Logic` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `sumologic.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: sumologic-example namespace: odigos-system spec: data: {} destinationName: sumologic secretRef: name: sumologic-secret signals: - TRACES - METRICS - LOGS type: sumologic --- apiVersion: v1 data: SUMOLOGIC_COLLECTION_URL: kind: Secret metadata: name: sumologic-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f sumologic.yaml ``` # TelemetryHub Source: https://docs.odigos.io/oss/backends/telemetryhub Configuring the TelemetryHub backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **TELEMETRY\_HUB\_API\_KEY** `string` : API Key. * This field is required We handle the endpoint internally, so you don't need to provide it. * The endpoint is `https://otlp.telemetryhub.com:4317` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `TelemetryHub` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `telemetryhub.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: telemetryhub-example namespace: odigos-system spec: data: {} destinationName: telemetryhub secretRef: name: telemetryhub-secret signals: - TRACES - METRICS - LOGS type: telemetryhub --- apiVersion: v1 data: TELEMETRY_HUB_API_KEY: kind: Secret metadata: name: telemetryhub-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f telemetryhub.yaml ``` # Tempo Source: https://docs.odigos.io/oss/backends/tempo Configuring the Tempo backend (Self-Hosted) ### Getting Started tempo

These instructions are for the **self hosted** Tempo instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Tempo](../backends/grafanacloudtempo) page. ### Configuring Destination Fields ✅ Traces ❌ Metrics ❌ Logs ❌ Profiles * **TEMPO\_URL** `string` : Endpoint. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Tempo` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `tempo.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: tempo-example namespace: odigos-system spec: data: TEMPO_URL: destinationName: tempo signals: - TRACES type: tempo ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f tempo.yaml ``` # Tingyun 基调听云 Source: https://docs.odigos.io/oss/backends/tingyun Configuring the Tingyun 基调听云 backend (Managed) ### Getting Started tingyun

### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **TINGYUN\_ENDPOINT** `string` : OTLP HTTP Endpoint. * This field is required * **TINGYUN\_LICENSE\_KEY** `string` : License Key. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Tingyun 基调听云` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `tingyun.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: tingyun-example namespace: odigos-system spec: data: TINGYUN_ENDPOINT: destinationName: tingyun secretRef: name: tingyun-secret signals: - TRACES - METRICS type: tingyun --- apiVersion: v1 data: TINGYUN_LICENSE_KEY: kind: Secret metadata: name: tingyun-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f tingyun.yaml ``` # Traceloop Source: https://docs.odigos.io/oss/backends/traceloop Configuring the Traceloop backend (Managed) ### Getting Started

### Configuring Destination Fields ✅ Traces ✅ Metrics ❌ Logs ❌ Profiles * **TRACELOOP\_ENDPOINT** `string` : Traceloop OTLP HTTP Endpoint. * This field is required and defaults to `https://api.traceloop.com` * **TRACELOOP\_API\_KEY** `string` : Traceloop API Key. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Traceloop` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `traceloop.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: traceloop-example namespace: odigos-system spec: data: TRACELOOP_ENDPOINT: '' destinationName: traceloop secretRef: name: traceloop-secret signals: - TRACES - METRICS type: traceloop --- apiVersion: v1 data: TRACELOOP_API_KEY: kind: Secret metadata: name: traceloop-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f traceloop.yaml ``` # Uptrace Source: https://docs.odigos.io/oss/backends/uptrace Configuring the Uptrace backend (Managed) ### Getting Started uptrace

### Configuring Destination Fields ✅ Traces ✅ Metrics ✅ Logs ❌ Profiles * **UPTRACE\_DSN** `string` : Data Source Name (DSN). * This field is required * **UPTRACE\_ENDPOINT** `string` : Endpoint. Overwrite Uptrace endpoint when self-hosting * This field is optional and defaults to `https://otlp.uptrace.dev:4317` ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `Uptrace` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `uptrace.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: uptrace-example namespace: odigos-system spec: data: UPTRACE_DSN: # Note: The commented fields below are optional. # UPTRACE_ENDPOINT: destinationName: uptrace signals: - TRACES - METRICS - LOGS type: uptrace ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f uptrace.yaml ``` # VictoriaMetrics Cloud Source: https://docs.odigos.io/oss/backends/victoriametricscloud Configuring the VictoriaMetrics Cloud backend (Managed) ### Getting Started

**Creating Account**
Go to the **[🔗 Cloud Console](https://console.victoriametrics.cloud) > Sign Up** **Setting Up**
Go to **⚙️ > Deployments** and click **Start using VictoriaMetrics Cloud** **Obtaining Access Token**
Go to **⚙️ > Deployments** and click **Overview**
Then click **Access Tokens** and copy one of the keys **Obtaining Endpoint**
Go to **⚙️ > Integrations** and click **OpenTelemetry > Operator**
Scroll to **Kubernetes commandline command** and copy the **endpoint** ### Configuring Destination Fields ❌ Traces ✅ Metrics ❌ Logs ❌ Profiles * **VICTORIA\_METRICS\_CLOUD\_ENDPOINT** `string` : OTLP HTTP Endpoint. * This field is required * **VICTORIA\_METRICS\_CLOUD\_TOKEN** `string` : Access Token. * This field is required ### Adding Destination to Odigos There are two primary methods for configuring destinations in Odigos: ##### **Using the UI** Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI ```bash theme={null} odigos ui ``` Click on `Add Destination`, select `VictoriaMetrics Cloud` and follow the on-screen instructions ##### **Using Kubernetes manifests** Save the YAML below to a file (e.g. `victoriametricscloud.yaml`) ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: victoriametricscloud-example namespace: odigos-system spec: data: VICTORIA_METRICS_CLOUD_ENDPOINT: destinationName: victoriametricscloud secretRef: name: victoriametricscloud-secret signals: - METRICS type: victoriametricscloud --- apiVersion: v1 data: VICTORIA_METRICS_CLOUD_TOKEN: kind: Secret metadata: name: victoriametricscloud-secret namespace: odigos-system type: Opaque ``` Apply the YAML using `kubectl` ```bash theme={null} kubectl apply -f victoriametricscloud.yaml ``` # odigos Source: https://docs.odigos.io/oss/cli/odigos ## odigos Automate OpenTelemetry Observability in Kubernetes ### Synopsis Odigos is primarily focused on automating OpenTelemetry observability pipelines for traces, metrics, and logs, without the need for extensive code changes. The core of Odigos functionality lies in the Kubernetes operators it deploys within your cluster, enabling seamless observability. Key Features of Odigos: * Automatic creation of OpenTelemetry observability pipelines. * Simplified tracing, metrics, and log collection. * Enhanced visibility into your Kubernetes services. * Streamlined Kubernetes operations with observability at the forefront. Get started with Odigos today to effortlessly improve the observability of your Kubernetes services! ### Options ``` -h, --help help for odigos --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos cleanup](/cli/odigos_cleanup) - Remove Odigos Sources created by the user. * [odigos describe](/cli/odigos_describe) - Show details of a specific odigos entity * [odigos diagnose](/cli/odigos_diagnose) - Diagnose Client Cluster * [odigos install](/cli/odigos_install) - Install and upgrade Odigos * [odigos pro](/cli/odigos_pro) - Manage Odigos onprem tier for enterprise users * [odigos profile](/cli/odigos_profile) - Manage presets of applied profiles to your odigos installation * [odigos sources](/cli/odigos_sources) - Manage Odigos Sources in a cluster * [odigos ui](/cli/odigos_ui) - Start the Odigos UI * [odigos uninstall](/cli/odigos_uninstall) - Uninstall Odigos * [odigos uninstall-deprecated](/cli/odigos_uninstall-deprecated) - Revert all the changes made by the `odigos install` command. This command will uninstall Odigos from your cluster. It will delete all Odigos objects and rollback any metadata changes made to your objects. * [odigos upgrade](/cli/odigos_upgrade) - Upgrade Odigos * [odigos version](/cli/odigos_version) - Print odigos version. # odigos describe Source: https://docs.odigos.io/oss/cli/odigos_describe ## odigos describe Show details of a specific odigos entity ### Synopsis This command can be used for troubleshooting and observing odigos state for its various entities.
It is similar to `kubectl describe` but for Odigos entities. ``` odigos describe [flags] ``` ### Examples ``` # Describe a source of kind deployment and name myservice in the default namespace odigos describe source deployment myservice -n default # Describe a source of kind deploymentconfig and name myservice in the default namespace (OpenShift) odigos describe source deploymentconfig myservice -n default Output: Name: myservice Kind: deployment Namespace: default Labels: Instrumented: true Workload: instrumented Namespace: unset Decision: Workload is instrumented because there is an enabled Source for this workload Troubleshooting: https://docs.odigos.io/architecture/troubleshooting#1-source-configuration Instrumentation Config: Created at 2024-07-30 19:00:40 +0300 IDT Runtime inspection details: Created at 2024-07-30 19:00:40 +0300 IDT Detected Containers: - Container Name: myservice Language: javascript Relevant Environment Variables: - NODE_OPTIONS : --require /var/odigos/nodejs/autoinstrumentation.js Instrumentation Device: Status: Successfully applied instrumentation device to pod template - Container Name: myservice Instrumentation Devices: javascript-native-community Pods (Total 1, Running 1): Pod Name: myservice-ffd68d8c-qqmxl Pod Phase: Running Pod Node Name: kind-control-plane Containers: - Container Name: myservice Instrumentation Devices: javascript-native-community Instrumentation Instances: - Healthy: true Reason: Healthy ``` ### Options ``` -h, --help help for describe -r, --remote use odigos ui service in the cluster to describe the entity ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes * [odigos describe source](/cli/odigos_describe_source) - Show details of a specific odigos source # odigos diagnose Source: https://docs.odigos.io/oss/cli/odigos_diagnose ## odigos diagnose Diagnose Client Cluster ### Synopsis Retrieves Logs of all Odigos components in the odigos-system namespace and CRDs of Actions, instrumentation resources and more. The results will be saved in a compressed file for further troubleshooting. The file will be saved in this format: odigos\_debug\_ddmmyyyyhhmmss.tar.gz ``` odigos diagnose [flags] ``` ### Options ``` --all-workload-namespaces Collect instrumented source workloads from all namespaces -h, --help help for diagnose --odigos-configmaps Include Odigos ConfigMaps in the diagnose output (default true) --odigos-crds Include Odigos CRDs in the diagnose output (default true) --odigos-logs Include Odigos component pod logs in the diagnose output (default true) --odigos-metrics Include Odigos Prometheus metrics in the diagnose output (default true) --odigos-profiles Include Odigos pprof profiles in the diagnose output (default true) --source-workload-namespaces strings Collect instrumented source workloads from specific namespaces (comma-separated) ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # odigos install Source: https://docs.odigos.io/oss/cli/odigos_install ## odigos install Install and upgrade Odigos ### Synopsis This subcommand installs or upgrades Odigos in your Kubernetes cluster. It installs Kubernetes components that auto-instrument your applications with OpenTelemetry and send traces, metrics, and logs to any telemetry backend. ``` odigos install [flags] ``` ### Examples ``` # Install or upgrade Odigos open-source in your cluster with the default values odigos install # Install or upgrade Odigos on-prem tier for enterprise users odigos install --set onPremToken=${ODIGOS_TOKEN} # Install or upgrade Odigos and set specific values odigos install --set collectorGateway.minReplicas=5 --set collectorGateway.maxReplicas=10 # Install or upgrade Odigos using a local values.yaml file odigos install --values ./values.yaml # Reset all values to chart defaults (opt out of reuse) odigos install --reset-then-reuse-values=false ``` ### Options ``` --chart string Helm chart to install (repo/name, local path, or URL) (default "odigos/odigos") --chart-version string Override Helm chart version (defaults to CLI-baked version) -h, --help help for install --ns string Target Kubernetes namespace (default "odigos-system") --release-name string Helm release name (default "odigos") --reset-then-reuse-values Reset to chart defaults, then reuse values from the previous release (default: true). (default true) --set strings Set values on the command line (key=value) -f, --values string Path to a custom values.yaml file ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # odigos profile Source: https://docs.odigos.io/oss/cli/odigos_profile ## odigos profile Manage presets of applied profiles to your odigos installation ### Synopsis This command can be used to interact with the applied profiles in your odigos installation. ``` odigos profile [flags] ``` ### Examples ``` # Enable payload collection for all supported workloads and instrumentation libraries in the cluster odigos profile add full-payload-collection # Remove the full-payload-collection profile from the cluster odigos profile remove full-payload-collection ``` ### Options ``` -a, --available list all available profiles to use -h, --help help for profile ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # odigos sources Source: https://docs.odigos.io/oss/cli/odigos_sources ## odigos sources Manage Odigos Sources in a cluster ### Synopsis This command can be used to create, delete, or update Sources to configure workload or namespace auto-instrumentation ### Examples ``` # Create a Source "foo-source" for deployment "foo" in namespace "default" odigos sources create foo-source --workload-kind=Deployment --workload-name=foo --workload-namespace=default -n default # Update all existing Sources in namespace "default" to disable instrumentation odigos sources update --disable-instrumentation -n default # Delete all Sources in group "mygroup" odigos sources delete --group mygroup --all-namespaces ``` ### Options ``` -h, --help help for sources ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes * [odigos sources create](/cli/odigos_sources_create) - Create an Odigos Source * [odigos sources delete](/cli/odigos_sources_delete) - Delete Odigos Sources * [odigos sources disable](/cli/odigos_sources_disable) - Disable a source for Odigos instrumentation. * [odigos sources enable](/cli/odigos_sources_enable) - Enable a source for Odigos instrumentation. * [odigos sources status](/cli/odigos_sources_status) - Show the status of all Odigos Sources * [odigos sources update](/cli/odigos_sources_update) - Update Odigos Sources # odigos ui Source: https://docs.odigos.io/oss/cli/odigos_ui ## odigos ui Start the Odigos UI ### Synopsis Start the Odigos UI. This will start a web server that you can access in your browser and enables you to manage and configure Odigos. ``` odigos ui [flags] ``` ### Examples ``` # Start the Odigos UI on http://localhost:3000 odigos ui # Start the Odigos UI on specific port if 3000 is already in use odigos ui --port 3456 # Start the Odigos UI and have it manage and configure a specific cluster odigos ui --kubeconfig ``` ### Options ``` --address string Address to serve the UI on (default "localhost") -h, --help help for ui --port int Port to listen on (default 3000) --remote-port int Port to forward to the remote UI (default 3000) ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # odigos uninstall Source: https://docs.odigos.io/oss/cli/odigos_uninstall ## odigos uninstall Uninstall Odigos ### Synopsis Revert all the changes made by the odigos install command. This command will uninstall Odigos from your cluster. It will delete all Odigos objects and rollback any metadata changes made to your objects. ``` odigos uninstall [flags] ``` ### Examples ``` # Uninstall Odigos odigos uninstall ``` ### Options ``` -h, --help help for uninstall --ns string Target Kubernetes namespace (default "odigos-system") --release-name string Helm release name (default "odigos") ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # odigos upgrade Source: https://docs.odigos.io/oss/cli/odigos_upgrade ## odigos upgrade Upgrade Odigos ### Synopsis Upgrades (or installs) Odigos in your Kubernetes cluster. This command behaves identically to 'odigos install' and uses the same Helm-based flow. ``` odigos upgrade [flags] ``` ### Examples ``` # Upgrade Odigos odigos upgrade # Upgrade Odigos with custom values odigos upgrade --set collectorGateway.maxReplicas=10 # Reset all values to chart defaults (opt out of reuse) odigos upgrade --reset-then-reuse-values=false ``` ### Options ``` --chart string Helm chart to install (repo/name, local path, or URL) (default "odigos/odigos") --chart-version string Override Helm chart version (defaults to CLI-baked version) -h, --help help for upgrade --ns string Target Kubernetes namespace (default "odigos-system") --release-name string Helm release name (default "odigos") --reset-then-reuse-values Reset to chart defaults, then reuse values from the previous release (default: true). (default true) --set strings Set values on the command line (key=value) -f, --values string Path to a custom values.yaml file ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # odigos version Source: https://docs.odigos.io/oss/cli/odigos_version ## odigos version Print odigos version. ### Synopsis This command is used to print the Odigos version. Both the CLI version and the Odigos components in your cluster will be printed. ``` odigos version [flags] ``` ### Examples ``` # Print the version of odigos CLI and Odigos deployment in your cluster odigos version ``` ### Options ``` --cli prints only the CLI version --cluster prints only the version of odigos deployed in the cluster -h, --help help for version ``` ### Options inherited from parent commands ``` --kube-context string (optional) name of the kubeconfig context to use --kubeconfig string (optional) absolute path to the kubeconfig file (default "KUBECONFIG") -v, --verbose enable verbose output ``` ### SEE ALSO * [odigos](/cli/odigos) - Automate OpenTelemetry Observability in Kubernetes # Custom Resources Source: https://docs.odigos.io/oss/custom-resources The different components of the Odigos observability control plane work together to achieve observability for the cluster. The interaction between the components is performed via the Kubernetes API server. The instrumentor is a [Kubernetes Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). It coordinates by writing and watching the odigos custom resources in the Kubernetes API server. Odigos adds the following custom resources to the Kubernetes API server: ## Source This resource defines namespaces and workloads to instrument. It has the following `spec` fields: | Field | Description | Type | Default | Required | | - | - | - | - | - | | `disableInstrumentation` | Excludes the specified workload from instrumentation. | `bool` | `false` | Optional | | `workload.name` | Name of the workload for this Source. | `string` | n/a | Required | | `workload.namespace` | Namespace of the workload for this Source. | `string` | n/a | Required | | `workload.kind` | Kind of the workload for this Source. | `WorkloadKind` | n/a | Required | Valid values for `WorkloadKind` are: * `Deployment` * `DaemonSet` * `StatefulSet` * `Namespace` These values are case-sensitive. ## Destination This custom resource is used to define the backend destinations for the observability data. Destinations can be either vendors (Datadog, Honeycomb, etc) or on premise. The destination object holds any data that is needed to connect to the backend system. Notice that sensitive fields such as API keys are stored in a Kubernetes secret and referenced by the destination object. ## CollectorsGroup The CollectorsGroup object is used to define a group of collectors that have a shared role. For example, a collectors group might include collectors that are deployed as DaemonSet that are responsible for collecting telemetry data, or collectors that are deployed as Deployment and are responsible for shipping the data to the selected destinations. # Debugging Odigos Source: https://docs.odigos.io/oss/debugging Different components of Odigos can be debugged in different ways. This page describes how to debug each component. ## Odiglet First make sure you have a running installation of Odigos on a local Kind cluster. From the root of the Odigos repository, run: ```bash theme={null} TAG= make debug-odiglet ``` This command will replace the running Odiglet binary and run it in a container on the Kind cluster. The new Odiglet will not start until a debug session is attached to it Now all that left is to attach a remote debugger (using VSCode / GoLand) to `localhost:2345` # Istio Service Mesh Tracing Source: https://docs.odigos.io/oss/instrumentations/advanced/Istio Odigos can collect the distributed traces emitted by the [Istio](https://istio.io/) service mesh and route them through the same pipeline as your application traces. Once enabled, the Envoy sidecars report a span for every meshed hop;connection-level latency, retries, response codes, and mTLS — over OTLP to the Odigos collector, where they are correlated with the spans Odigos already generates for your applications. The result is a single trace that spans both the network and application layers. This page covers how to point an **existing** Istio mesh at Odigos. It is not a guide to installing or operating Istio. For mesh setup, see the [Istio installation guide](https://istio.io/latest/docs/setup/install/helm/). ## Prerequisites **Kubernetes 1.26+ is required** — the `odigos-data-collection-local-traffic` service depends on `internalTrafficPolicy: Local`, which became GA in 1.26. Istio's control plane (`istiod`) must be installed in your cluster. Follow the [Istio installation guide](https://istio.io/latest/docs/setup/install/helm/). Istio only emits spans for workloads that have an Envoy sidecar. Label the namespaces you want traced and restart their workloads so the sidecar is injected: ```bash theme={null} kubectl label namespace istio-injection=enabled kubectl rollout restart deployment -n ``` For the full set of injection options, see Istio's [sidecar injection](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/) documentation. ## Configuration Add an OpenTelemetry extension provider to Istio's `MeshConfig` that points at the Odigos node collector's OTLP/HTTP endpoint. For background on this field, see Istio's reference for [enabling OpenTelemetry tracing](https://istio.io/latest/docs/tasks/observability/distributed-tracing/opentelemetry/). Apply it using whichever method you use to manage your mesh: Add the `meshConfig` block to the values for the `istiod` chart (for example, `istiod-values.yaml`): ```yaml istiod-values.yaml theme={null} meshConfig: enableTracing: true extensionProviders: - name: odigos-otel opentelemetry: service: odigos-data-collection-local-traffic.odigos-system.svc.cluster.local port: 4318 http: path: "/v1/traces" timeout: 5s resource_detectors: environment: {} ``` ```bash theme={null} helm upgrade istiod istio/istiod -n istio-system --reuse-values -f istiod-values.yaml ``` Define the `meshConfig` block in an `IstioOperator` configuration file (for example, `istio-otel.yaml`): ```yaml istio-otel.yaml theme={null} apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: meshConfig: enableTracing: true extensionProviders: - name: odigos-otel opentelemetry: service: odigos-data-collection-local-traffic.odigos-system.svc.cluster.local port: 4318 http: path: "/v1/traces" timeout: 5s resource_detectors: environment: {} ``` ```bash theme={null} istioctl install -f istio-otel.yaml ``` Defining the provider on its own does not start any tracing — it only tells Istio where Odigos is. Requires **Kubernetes 1.26+**. Below that, `odigos-data-collection-local-traffic` does not exist and Istio has no endpoint to reach. Create a `Telemetry` resource in the Istio root namespace (usually `istio-system`) that references the provider. This is what actually turns on span reporting. See the [Istio Telemetry API](https://istio.io/latest/docs/tasks/observability/telemetry/) for scoping options. ```yaml theme={null} apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: mesh-default namespace: istio-system spec: tracing: - providers: - name: odigos-otel randomSamplingPercentage: 100 ``` Leave `randomSamplingPercentage` at `100` so every meshed hop is reported. If you need to reduce volume, do it with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) rather than at the Envoy layer — that way the decision sees the full trace (errors, latency, and downstream spans) instead of being made blindly per request at the sidecar. The Odigos collector only exports data when at least one destination is configured. Because Envoy emits Istio spans on behalf of the mesh rather than any specific Odigos-instrumented workload, the collector routes them through the **default data stream** — so make sure the destination you want the Istio traces to reach is assigned to the default data stream. Istio produces spans only for L7 HTTP traffic on Service ports named `http` / `http-*` (or with `appProtocol: http`). Plain TCP protocols such as Redis or AMQP will not generate mesh spans. ## Validation After traffic flows through a meshed workload, look for Envoy spans in your tracing backend. They appear under service names in the form `.` (for example, `frontend.default`), distinct from the application service names Odigos reports. You can identify them by: * `otel.scope.name` = `envoy` * `component` = `proxy` * `istio.*` resource attributes such as `istio.canonical_service` and `istio.mesh_id` ### Example Real meshed traces interleave Istio and Odigos spans — every meshed hop adds a pair of Istio sidecar spans (one on each side of the wire) that wrap the callee's application work, which itself can fan out into more application spans and another mesh hop. A two-hop call through `frontend → checkout → payments` lays out like this: Horizontal position is wall-clock time; label indentation follows the true parent/child nesting. A child bar always starts after and ends within its parent. ``` 0ms 50 100 150 200ms ├────────┼────────┼────────┼────────┤ frontend · POST /checkout ████████████████████████████████████ frontend · Istio→checkout ┏━━━━━━━━━━━━━━━━━━━━━━━━━┓ checkout · Istio server ┏━━━━━━━━━━━━━━━━━━━┓ checkout · POST /checkout ██████████████████████ checkout · SELECT cart ██████ checkout · Istio→payments ┏━━━━━━━━━┓ payments · Istio server ┏━━━━━━━┓ payments · POST /charge █████████ payments · INSERT txn ██████ Legend: █ Odigos (app) ┏━┓ Istio (mesh) ``` Each Istio client/server pair brackets the next service's application work. The gap between the two Istio spans is **on-the-wire time** (DNS, TCP, mTLS handshakes, retries); the gap between the Istio server span and the application span is **proxy overhead**. With each additional meshed hop, another sidecar pair is added — both costs become directly attributable instead of being absorbed into a single application-latency number. Istio spans are additive — they do not replace the application spans Odigos generates. Each meshed HTTP hop yields an Istio client span, an Istio server span, and the application span(s), all sharing the same trace. # NGINX Ingress Tracing Source: https://docs.odigos.io/oss/instrumentations/advanced/Nginx Odigos can collect the distributed traces emitted by the [Ingress-NGINX Controller](https://kubernetes.github.io/ingress-nginx/) and route them through the same pipeline as your application traces. Once enabled, the controller reports a span for every request that enters the cluster through it — route, upstream, response code, and latency — over OTLP to the Odigos collector, where they are correlated with the spans Odigos already generates for your applications. The result is a single trace that begins at the edge and follows the request all the way through your services. The community Ingress-NGINX project has been **retired** as of March 2026 — it no longer receives releases, bug fixes, or security patches, so running it carries growing security risk. See the [Kubernetes retirement announcement](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/). For new deployments, migrate to the [Gateway API](https://gateway-api.sigs.k8s.io/) instead. This page covers how to point an **existing** Ingress-NGINX deployment at Odigos. It is not a guide to installing or operating the controller. For controller setup, see the [Ingress-NGINX installation guide](https://kubernetes.github.io/ingress-nginx/deploy/). Tracing happens at the **Ingress-NGINX Controller**, not in your application's nginx pods. Stock `nginx` images lack the OpenTelemetry module, but the controller bundles it and turns a few ConfigMap keys into the right OTel directives. Put the workloads you want traced behind an Ingress and enable OTel on the controller. ## Prerequisites The `odigos-data-collection-local-traffic` service depends on `internalTrafficPolicy: Local`, which became GA in 1.26. The Ingress-NGINX Controller must be installed in your cluster, with the OpenTelemetry module available. Follow the [Ingress-NGINX installation guide](https://kubernetes.github.io/ingress-nginx/deploy/). The controller only emits spans for traffic that flows through it. Make sure the workloads you want traced are fronted by an `Ingress` resource that routes to their `Service`. For background on routing, see the [Ingress-NGINX user guide](https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/). ## Configuration Configure the Ingress-NGINX Controller to export OpenTelemetry traces to the Odigos node collector's OTLP/gRPC endpoint. For background on these keys, see the controller's [OpenTelemetry documentation](https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/). Choose one of the methods below based on how you manage the controller. If you installed it with Helm, use Helm values so a later `helm upgrade` does not overwrite the settings; otherwise apply the ConfigMap directly. Enable the OTel module and set the exporter through the `ingress-nginx` chart values: ```bash theme={null} helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx --create-namespace \ --set controller.opentelemetry.enabled=true \ --set controller.config.enable-opentelemetry="true" \ --set controller.config.otlp-collector-host="odigos-data-collection-local-traffic.odigos-system.svc.cluster.local" \ --set controller.config.otlp-collector-port="4317" \ --set controller.config.otel-service-name="nginx-ingress" \ --set controller.config.opentelemetry-trust-incoming-span="true" \ --set controller.config.opentelemetry-operation-name="HTTP \$request_method \$uri" ``` Add the exporter keys to the controller's ConfigMap (named `ingress-nginx-controller` in the `ingress-nginx` namespace by default — match whatever your controller reads via its `--configmap` flag): ```yaml ingress-nginx-controller-cm.yaml theme={null} apiVersion: v1 kind: ConfigMap metadata: name: ingress-nginx-controller namespace: ingress-nginx data: enable-opentelemetry: "true" otlp-collector-host: "odigos-data-collection-local-traffic.odigos-system.svc.cluster.local" otlp-collector-port: "4317" otel-service-name: "nginx-ingress" opentelemetry-trust-incoming-span: "true" opentelemetry-operation-name: "HTTP $request_method $uri" ``` After changing the ConfigMap, roll the controller so it reloads: ```bash theme={null} kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller ``` **OTLP/gRPC only.** The Ingress-NGINX OTel module speaks OTLP gRPC on port **4317**. There is no HTTP/protobuf (4318) option — pointing it at 4318 will silently fail to export. The `odigos-data-collection-local-traffic` service accepts OTLP/gRPC on 4317. By default the controller samples every request through the ingress, so no sampler settings are needed. If you need to reduce volume, do it with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) rather than at the controller — that way the decision sees the full trace (errors, latency, and downstream spans) instead of being made blindly per request at the edge. Enabling OTel on the controller does not trace every route by default. Activate tracing on a route with per-`Ingress` annotations. For the full set of options, see the controller's [OpenTelemetry documentation](https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/). ```yaml theme={null} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: frontend namespace: default annotations: nginx.ingress.kubernetes.io/enable-opentelemetry: "true" nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true" spec: rules: - host: frontend.example.com http: paths: - path: / pathType: Prefix backend: service: name: frontend port: number: 80 ``` The Odigos collector only exports data when at least one destination is configured. Because the controller emits ingress spans on behalf of the edge rather than any specific Odigos-instrumented workload, the collector routes them through the **default data stream** — so make sure the destination you want the NGINX traces to reach is assigned to the default data stream. ## Validation After traffic flows through the ingress, look for the controller's spans in your tracing backend. They appear under the service name you set in `otel-service-name` (for example, `nginx-ingress`), distinct from the application service names Odigos reports. Confirm the path end to end. Neither the NGINX OTel module nor the Odigos collector logs anything per span, so **no OTel errors in the logs is the healthy state** — any OTel output is an error. To confirm spans are actually arriving, check your destination. * **Controller logs** — confirm there are no export errors: ```bash theme={null} kubectl -n ingress-nginx logs deployment/ingress-nginx-controller | grep -i otel ``` If the collector is unreachable, the gRPC exporter logs an `UNAVAILABLE` error on every flush: ```text theme={null} [Error] File: .../otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "failed to connect to all addresses; last error: ... ipv4::4317: Connection refused" ``` That almost always means a wrong `otlp-collector-host`/`otlp-collector-port`, port 4318 instead of 4317, or a NetworkPolicy blocking egress to `odigos-system`. * **Collector logs** — the collector is also quiet on success. Watch for export errors toward your destination: ```bash theme={null} kubectl -n odigos-system logs -l odigos.io/collector-role=DATA_COLLECTION --tail=50 ``` A misconfigured or unreachable destination shows up as repeated exporter errors. Depending on where the failure happens, you will see either `Exporting failed. Dropping data.` or `Exporting failed. Rejecting data.`: ```text theme={null} error exporterhelper/queue_sender.go:50 Exporting failed. Dropping data. {"otelcol.component.id": "otlp/", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "...", "dropped_items": 5} ``` * **Your destination** — open your backend and look for a service named after `otel-service-name` (for example, `nginx-ingress`). ### Example Real traces interleave the NGINX ingress span with the Odigos application spans — the ingress span wraps the upstream's application work, which itself can fan out into more application spans. A request through `ingress → frontend → checkout` lays out like this: Horizontal position is time; indentation shows parent/child nesting. ``` 0ms 50 100 150 200ms ├────────┼────────┼────────┼────────┤ nginx-ingress · GET / ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ frontend · GET / ███████████████████████████████████ frontend · GET /checkout ████████████████████████ checkout · GET /checkout ██████████████████ checkout · SELECT cart ██████ Legend: █ Odigos (app) ┏━┓ NGINX (ingress) ``` The ingress span wraps all the application work behind it. The gap before the first application span is **time spent at the edge** (routing and connecting upstream); everything inside is application latency. NGINX ingress spans are additive — they do not replace the application spans Odigos generates. Each traced request yields an ingress span plus the application span(s), all sharing the same trace, as long as `opentelemetry-trust-incoming-span` is enabled so context propagates to your workloads. ## Tuning | Key | Effect | | - | - | | `otel-service-name` | The service name spans show up under. Set per environment. | | `opentelemetry-operation-name` | Template for the span name. `HTTP $request_method $uri` produces names like `HTTP GET /checkout`; supports NGINX variables. | | `otel-sampler` | Leave at the default (`AlwaysOn`), which captures 100%, and configure sampling with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) instead. | | `opentelemetry-trust-incoming-span` | Whether to continue trace context from inbound requests. Keep `true` so ingress spans link to upstream callers and downstream workloads. | | `nginx.ingress.kubernetes.io/enable-opentelemetry` (annotation) | Per-`Ingress` enablement. Remove to stop tracing a specific route. | ## Troubleshooting * **No spans at all** — the most common cause is a wrong `otlp-collector-host`. The `odigos-data-collection-local-traffic` service must exist in `odigos-system` and accept OTLP/gRPC. Verify with `kubectl get svc -n odigos-system`. * **Set 4318 and nothing arrives** — the module is gRPC-only; use 4317. * **OTel keys added to an application nginx ConfigMap did nothing** — expected; that is the stock-nginx ConfigMap, not the controller's. OTel config belongs in the controller ConfigMap. * **Config changes ignored** — roll the controller: `kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller`. # Agent Mount Method Source: https://docs.odigos.io/oss/instrumentations/configuration/mount-method For Odigos agents to run inside instrumented pod containers, certain files must be mounted into the container. In Kubernetes, these files are mounted under `/var/odigos` directory, with subdirectories for each agent. There are few mechanisms to achieve this, which are explained in this document. ## Mount Method This section is for advanced users and odigos administrators. It is recommended to use the default settings, unless you have specific requirements or run into an issue. ## Supported Mount Methods Odigos supports 4 mount methods, which can be used depending on the user preference, cluster policies, integration with existing tools, etc. ### 1. VirtualDevice This is the default mount method #### Pod Manifest Additions For agents and languages that requires filesystem mounts, odigos webhook will add the following: * For each instrumented container in the pod spec, odigos will add a resource requirement under the `spec.containers[].resources` field: ```yaml theme={null} resources: limits: instrumentation.odigos.io/generic: "1" requests: instrumentation.odigos.io/generic: "1" ``` #### Caveats * With [Karpenter](../../setup/odigos-with-karpenter), provisioning can fail because Karpenter does not know about the `instrumentation.odigos.io/generic` extended resource when simulating new nodes. Support depends on your Karpenter version and provider: when [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is available and enabled, apply an overlay that adds this capacity (see the [Karpenter guide](../../setup/odigos-with-karpenter#virtual-device-nodeoverlay)). Otherwise use [`k8s-init-container`](#3-InitContainer), or [`k8s-host-path`](#2-HostPath) / [`k8s-csi-driver`](#4-CSI-Driver) with the Karpenter integration steps. * Odiglet daemonset needs to be running on a node for instrumented pods to be scheduled on that node. ### 2. HostPath This method is an opt-in configuration option which can be used if the virtual device method is not suitable for your cluster. If it is supported in the cluster, it is preferred to use over the VirtualDevice method which requires odiglet component to run. **Enabling HostPath** * Profile: `odigos profile add mount-method-k8s-host-path` * Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-host-path` * Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-host-path`, or use helm cli `--set instrumentor.mountMethod=k8s-host-path` flag with helm upgrade. * Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-host-path`. #### Pod Manifest Additions For agents and languages that requires filesystem mounts, odigos webhook will add the following: * Volume to the pod spec, under the `spec.volumes` field: ```yaml theme={null} - hostPath: path: /var/odigos type: "" name: odigos-agent ``` * VolumeMount to the instrumented container specs, under the `spec.containers[].volumeMounts` field: ```yaml theme={null} - mountPath: /var/odigos/{agent_sub_dir} name: odigos-agent readOnly: true subPath: {agent_sub_dir} ``` #### Caveats * The "HostPath" option should be enabled in the cluster. Some policy tools may block this, like [Open Policy Agent](https://www.openpolicyagent.org/) or [Kyverno](https://kyverno.io/). If blocked, use the VirtualDevice method, or ask the cluster administrator to whitelist the Odigos agent host path. * With [Karpenter](../../setup/odigos-with-karpenter), additional configuration is required. By default Odigos adds node affinity for the odiglet-installed label, which blocks Karpenter from provisioning nodes. Enable `karpenter.enabled` and configure the NodePool `startupTaints` as described in the [Karpenter guide](../../setup/odigos-with-karpenter). ### 3. InitContainer This is an opt-in configuration method designed for scenarios where using hostPath is not feasible due to security policies or virtual device limitations. For agents and languages that require filesystem mounts, Odigos will inject an InitContainer that copies the necessary files into a shared volume mounted by the application container. **Enabling InitContainer** * Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-init-container` * Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-init-container`, or use helm cli `--set instrumentor.mountMethod=k8s-init-container` flag with helm upgrade. * Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-init-container`. #### Pod Manifest Additions Odigos will add an InitContainer that copies the necessary files for instrumentation, depending on the application language. These files will be placed into a shared volume that is also mounted by the application container. * Volume to the pod spec, under the `spec.volumes` field: ```yaml theme={null} - hostPath: path: /var/odigos type: "" name: odigos-agent ``` #### Caveats The InitContainer method must be enabled in the cluster and introduces a slight delay during pod initialization. This approach is ideal for users who prioritize avoiding hostPath or virtual device usage over startup speed. Works with [Karpenter](../../setup/odigos-with-karpenter) out of the box. No additional Karpenter configuration is required. #### Init container image pull secrets Kubernetes 1.35+ enables [`KubeletEnsureSecretPulledImages`](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) by default. The kubelet [re-verifies pull credentials](https://kubernetes.io/docs/concepts/containers/images/#ensure-image-pull-credential-verification) even when the agents image is already cached on the node. The injected `odigos-agents` init container therefore needs the same pull secret in the **application** namespace, not only in the Odigos namespace. This applies when you use `k8s-init-container` with a private registry: * the enterprise registry (`registry.odigos.io` via `odigos-enterprise-registry`, including `externalOnpremPullSecret` / `externalOnpremTokenSecret`) * a private mirrored registry (`imagePrefix` + `imagePullSecrets`) On Kubernetes 1.35+, the pod webhook adds those secret **names** to instrumented pods. You must create the same-named dockerconfig secret in every instrumented namespace yourself (External Secrets, Reflector, or a GitOps manifest). Below 1.35, odiglet pre-pulls the agents image onto each node and kubelet will use that cache without per-pod credentials. ### 4. CSI Driver This is an opt-in configuration method that uses the Container Storage Interface (CSI) standard for mounting instrumentation files. It provides improved user experience over the VirtualDevice method by ensuring pods wait gracefully when the mount system isn't ready. **Enabling CSI Driver** * Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-csi-driver` * Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-csi-driver`, or use helm cli `--set instrumentor.mountMethod=k8s-csi-driver` flag with helm upgrade. * Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-csi-driver`. #### Pod Manifest Additions For agents and languages that requires filesystem mounts, odigos webhook will add the following: * Volume to the pod spec, under the `spec.volumes` field: ```yaml theme={null} - name: odigos-agent csi: driver: odigos.csi.driver volumeAttributes: type: instrumentation ``` * VolumeMount to the instrumented container specs, under the `spec.containers[].volumeMounts` field: ```yaml theme={null} - mountPath: /var/odigos/{agent_sub_dir} name: odigos-agent readOnly: true subPath: {agent_sub_dir} ``` #### Benefits * **Better User Experience**: Pods wait in "Creating" state until CSI driver is ready. * **Standard Compliance**: Uses Kubernetes CSI standard for storage integration. * **Same Functionality**: Provides identical mount behavior as the VirtualDevice method. #### Caveats * Requires CSI support in the cluster (available in Kubernetes 1.13+). * Like VirtualDevice, requires odiglet daemonset to be running on nodes for instrumented pods to be scheduled. * With [Karpenter](../../setup/odigos-with-karpenter), additional configuration is required. By default Odigos adds node affinity for the odiglet-installed label, which blocks Karpenter from provisioning nodes. Enable `karpenter.enabled` and configure the NodePool `startupTaints` as described in the [Karpenter guide](../../setup/odigos-with-karpenter). # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/dotnet/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Add the following dependencies to your project by running: ```bash theme={null} dotnet add package OpenTelemetry ``` ## Creating Activity To create a new `Activity`, use the `ActivitySource` class. ```csharp theme={null} using System.Diagnostics; public static class Telemetry { // Name it after the service name for your app. // It can come from a config file, constants file, etc. public static readonly ActivitySource MyActivitySource = new(TelemetryConstants.ServiceName); public static void MyMethod() { using (var activity = MyActivitySource.StartActivity("MyMethod")) { // Your code here } } } ``` ## Additional Information For more use cases, see the [OpenTelemetry .NET API documentation](https://opentelemetry.io/docs/languages/net/instrumentation/#creating-activities). # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/dotnet/native Native Instrumentation is currently the default setting of Odigos **Open-Source** version. eBPF-Based Instrumentation is currently not supported for this language. ## Supported Versions Supported versions of the .NET Framework range from **4.6.2** to **10.0.3**. The following CPU architectures are supported: * amd64 (x86\_64) * arm64 ## Instrumentation Libraries The following .NET modules will be auto instrumented by Odigos: ### Application Frameworks | Framework | Tracing Support | Metrics Support | Notes | | - | :-: | :-: | - | | [ASP.NET](https://docs.microsoft.com/aspnet/overview) | Yes | Yes | | | [ASP.NET Core](https://docs.microsoft.com/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-6.0) | Yes | Yes | | ### Databases | Library | Tracing Support | Metrics Support | Notes | | - | - | - | - | | [Microsoft.Data.SqlClient](https://www.nuget.org/packages/Microsoft.Data.SqlClient) | Yes | | | | [System.Data.SqlClient](https://www.nuget.org/packages/System.Data.SqlClient) | Yes | | | | MySql.Data | Partial | | This is the official MySQL library. | | [MySqlConnector](https://mysqlconnector.net/) | Yes | | Seems to be the [recommended library for MariaDB](https://mariadb.com/docs/connectors/mariadb-connector-net/mariadb-connector-net-guide). | | [Npgsql](https://www.npgsql.org/) | Yes | | | | [Microsoft.Data.SqlLite](https://docs.microsoft.com/dotnet/standard/data/sqlite/?tabs=netcore-cli) | | | | | [MongoDB.Driver](https://www.nuget.org/packages/mongodb.driver) | Yes | | | | [StackExchange.Redis](https://stackexchange.github.io/StackExchange.Redis/) | Partial | | | | [Elasticsearch-net](https://github.com/elastic/elasticsearch-net) | Yes | | 8.0.0+ only. | | [Oracle.ManagedDataAccess.Core](https://www.nuget.org/packages/Oracle.ManagedDataAccess.Core) | Yes | | 23.4.0+ only. | | [Oracle.ManagedDataAccess](https://www.nuget.org/packages/Oracle.ManagedDataAccess) | Yes | | 23.4.0+ only. | | [AWSSDK.DynamoDBv2](https://www.nuget.org/packages/AWSSDK.DynamoDBv2) | | | | | [Microsoft.Azure.Cosmos](https://www.nuget.org/packages/Microsoft.Azure.Cosmos) | | | | ### Inter-process communication (IPC) | Library | Tracing Support | Metrics Support | Notes | | - | - | - | - | | [HttpClient](https://docs.microsoft.com/dotnet/api/system.net.http.httpclient) | Yes | Yes | | | [HttpWebRequest](https://docs.microsoft.com/dotnet/api/system.net.httpwebrequest) | Yes | Yes | | | [WCF](https://docs.microsoft.com/dotnet/framework/wcf/whats-wcf) | Yes (Client and Service side on .NET Framework) | | | | [CoreWCF](https://github.com/CoreWCF/CoreWCF) | | | | | [RestSharp](https://restsharp.dev/) | | | This library may be implicitly supported by instrumenting the underlying HttpClient or HttpWebRequest. | | [gRPC-dotnet](https://github.com/grpc/grpc-dotnet) | Partial | | Client and service support should be added. | | [GraphQL](https://www.nuget.org/packages/GraphQL/) | Partial | | | | [GraphQL Client](https://github.com/graphql-dotnet/graphql-client) | | | | | [RabbitMQ](https://www.nuget.org/packages/RabbitMQ.Client) | Yes | | `RabbitMQ.Client` 7.0.0+ has native support for traces. | | [Kafka](https://www.nuget.org/packages/Confluent.Kafka) | | | | | [NServiceBus](https://docs.particular.net/nservicebus/) | Yes | Yes | | | [MassTransit](https://masstransit-project.com/) | Yes | | | ## Automatic Testing The following scenarios are automatically tested with every release: | .NET Version | LibC type | CPU | | - | - | - | | 6.0 | glibc | amd64 | | 6.0 | musl | amd64 | | 8.0 | glibc | amd64 | | 8.0 | musl | amd64 | # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/golang/ebpf eBPF-Based Instrumentation is currently the default setting of Odigos **Open-Source** version, and is also available with the Odigos **Enterprise** plan. Native Instrumentation is currently not supported for this language. ## Supported Versions Odigos uses the official [opentelemetry-go-instrumentation](https://github.com/open-telemetry/opentelemetry-go-instrumentation) OpenTelemetry Auto Instrumentation using eBPF, thus it supports the same golang versions as this project. * Go runtime versions **1.19** through **1.26** are supported. Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0. The required Linux capabilities depend on your kernel version: * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`. * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`. We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`. ## Traces Odigos will automatically instrument your golang services to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Go official auto Instrumentation Libraries. ## Instrumentation Libraries The following go modules will be auto instrumented by Odigos: ### HTTP Frameworks * [`github.com/gin-gonic/gin`](https://pkg.go.dev/github.com/gin-gonic/gin) HTTP web framework ⭐️ * [`net/http`](https://pkg.go.dev/net/http) go standard library for http server and client Please note that in order for the `http.route` parameter to be properly extracted by Go auto instrumentation, you should use Go 1.22 or above and declare a template handler as described in [this article](https://go.dev/blog/routing-enhancements). ### HTTP Clients * [`net/http`](https://pkg.go.dev/net/http) go standard library for http server and client ### Database Clients, ORMs, and data access libraries * [`github.com/gocql/gocql`](https://pkg.go.dev/github.com/gocql/gocql) ⭐️ versions `>= v1.0.0`. database client library for Apache Cassandra * [`github.com/elastic/go-elasticsearch/v8`](https://pkg.go.dev/github.com/elastic/go-elasticsearch/v8) ⭐️ versions >= v8.0.0 * [`database/sql`](https://pkg.go.dev/database/sql) go standard library for SQL database access ### Messaging Systems Clients * [`github.com/rabbitmq/amqp091-go`](https://pkg.go.dev/github.com/rabbitmq/amqp091-go) ⭐️ versions `>= v1.4.0`. messaging client for RabbitMQ * [`github.com/segmentio/kafka-go`](https://pkg.go.dev/github.com/segmentio/kafka-go) versions `>= v0.4.1`. messaging client for Apache Kafka * [`github.com/apache/pulsar-client-go`](https://pkg.go.dev/github.com/apache/pulsar-client-go) ⭐️ versions `>= v0.12.0`. messaging client for Apache Pulsar * [`github.com/IBM/sarama`](https://github.com/IBM/sarama) ⭐️ versions `>= v1.40.0`. messaging client for Apache Kafka ### RPC (Remote Procedure Call) * [`google.golang.org/grpc`](https://pkg.go.dev/google.golang.org/grpc) versions `>= v1.14.0`. rpc client and server for gRPC framework ## Custom Instrumentations We support adding custom eBPF-based instrumentations for arbitrary Go functions and invokable sites that exist in your application code or its dependencies. By defining custom instrumentation rules, you can extend Odigos's auto-instrumentation capabilities and cover your specific use cases and application logic. To learn more about creating custom eBPF-based instrumentations for Go applications, refer to the [Custom eBPF Instrumentation for Go Applications](../../pipeline/rules/custominstrumentation) guide. ## About Go Offsets Auto-instrumentation for Go with eBPF works by using eBPF uprobes to dynamically read variables in memory at runtime. For example, when instrumenting the `net/http` package Odigos is able to determine the `http.request.method` by probing the [`Method`](https://pkg.go.dev/net/http#Request) field in the current `Request`. Doing so relies on knowing the exact memory location of this field. This is known as the field's "offset" (because it refers to the field's location as a number of bytes offset from the start of the current struct). With this information, our eBPF code can access the value directly. However, most production Go programs do not preserve this offset information at compile time. Only Go programs compiled with [debug information](https://go.dev/doc/gdb) will include this information by default. To work with all Go programs, Odigos ships with a precompiled list of known offsets for all currently supported versions of instrumented libraries. This precompiled list is updated to be up-to-date with every release of Odigos. But, this means that you may encounter errors instrumenting applications that use very newly released versions of dependencies. If this occurs, it will be resolved by updating to the next release of Odigos, or alternatively upgrading the locally-stored offsets with the [`odigos pro update-offsets`](../../../enterprise/cli/odigos_pro_update-offsets) command. ### Updating Offsets Between Versions It is possible to pull the latest Go library support for Odigos without updating the entire Odigos installation. There are 3 ways to do so: 1. Manual updates (with [`odigos pro update-offsets`](../../../enterprise/cli/odigos_pro_update-offsets)). 2. Direct automatic updates (recommended, see below). 3. Mirrored image automatic updates (for disconnected environments, see below). In both automatic modes, a Kubernetes CronJob is created which updates the locally stored Go library offsets used by the Odiglet. The differences between these modes and how to enable them are described below. #### Direct automatic updates In "direct" mode, the CronJob will pull the latest Go library offsets directly from the Odigos cloud server and apply it to the cluster. This option consumes the fewest resources but requires that your cluster has direct internet access to this file hosted on the web. If you have restricted Internet egress policies, see "mirrored image updates" below. Enable automatic updates on any standard cron schedule by setting the `goAutoOffsetsCron` option during installation or upgrade. For example, enable nightly updates at midnight with `odigos install --set goAutoOffsetsCron="0 0 * * *"`. To disable automatic updates, set the cron value to an empty string with `odigos install --set goAutoOffsetsCron=""`. Alternatively, you can set these values in your `values.yaml` file for Helm installations: ```yaml theme={null} goAutoOffsetsCron: "0 0 * * *" # Enable nightly updates at midnight goAutoOffsetsMode: "direct" # Use direct mode ``` #### Mirrored image automatic updates For clusters with restricted traffic policies, you have the option to [mirror the Go library updates as an image](../../../setup/docker-registry#install-odigos-images-from-a-custom-docker-registry). In this mode, the CronJob will run an image that contains the latest Go library offsets bundled into the image itself. This mode does not require direct Internet access from the cluster, but it does require you to mirror this image into your own registry and maintain the latest version on your nodes. To enable mirrored image updates, run `odigos install --set goAutoOffsetsMode="image"`. Then use the `goAutoOffsetsCron` option described above to set the schedule. To disable mirrored image updates, run `odigos install --set goAutoOffsetsMode="direct"` (to switch back to direct mode), or set the `goAutoOffsetsCron` setting to an empty string (`""`) as described above to disable automatic updates entirely. # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/golang/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Odigos fully supports the OpenTelemetry Go eBPF Auto SDK. See the [official docs on the Auto SDK](https://opentelemetry.io/docs/zero-code/go/autosdk/) for information on using it. Add the following dependencies to your project by running: ```bash theme={null} go get go.opentelemetry.io/otel \ go.opentelemetry.io/otel/trace ``` ## Spans Spans are the OpenTelemetry representation of operations in the execution of your code. You should record a span if you think the operation is meaningful and that observing it will help you understand the behavior of your system and operate it. A span includes a name, start and end time, and key-value attributes for detailed information about the execution. ## Tracer In OpenTelemetry, spans are created using the `Tracer` object which you can acquire from the OpenTelemetry API. Each tracer is associated with a name and optional version. These name and version are recorded as "instrumentation scope" into all spans created by this tracer. Humans viewing the trace and data processors downstream can use this information to understand the origin of each span. It is best practice to use the go module and package name for the tracer name. ```go theme={null} package main import ( "context" "go.opentelemetry.io/otel" ) var tracer = otel.Tracer("github.com///") ``` **Note**: Manually initializing a global TracerProvider will conflict with the Go Auto SDK's ability to link eBPF spans to custom spans and lead to broken trace IDs. If you are enriching your Odigos application with custom spans, do not initialize a global TracerProvider. See the [OpenTelemetry docs](https://opentelemetry.io/docs/zero-code/go/autosdk/) for more information. ## Creating Spans ```go theme={null} func my_function(ctx context.Context) { // Span start time is set here ctx, span := tracer.Start(ctx, "my_function") // Span end time is set here defer span.End() // ...Do some work, use ctx if needed when calling next functions } ``` Important Notes: 1. **Always End a span**: Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function. 2. **Propagate and use a valid context object**: When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace. 3. **Pass the context object downstream**: When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow. 4. **Assign meaningful names to spans**: Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context. 5. **Avoid dynamic, high cardinality data in span names**: Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace. ### Recording Span Attributes Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting Examples: * User ID, organization ID, Account ID or other identifiers. * Inputs - the relevant parameters or configuration that influenced the operation. * Outputs - the result of the operation. * Entities - the entities or resources that the operation interacted with. Attribute names are lowercased strings in the form `my_application.my_attribute`, example: `my_service.user_id`. Read more [here](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/) To record attributes, use the `SetAttributes` method on the span object. ```go theme={null} import ( "context" "go.opentelemetry.io/otel/attribute" ) func my_function(ctx context.Context, userId string) { _, span := tracer.Start(ctx, "my_function") defer span.End() span.SetAttributes(attribute.String("my_service.user_id", userId)) // ...Do some work } ``` Important Notes: 1. **Be cautious when recording data**: Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces. 2. **Attribute cost considerations**: Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data. 3. **Use static names for attributes**: Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute\_name) to provide clear context for downstream consumers. 4. **Adhere to OpenTelemetry semantic conventions**: Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand. ### Recording Errors To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts. If an operation you are tracing fails, you can mark the span's status as an error and record the error details within the span. Here's how you can do it: ```go theme={null} import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/codes" ) func my_function(ctx context.Context, userId string) { _, span := tracer.Start(ctx, "my_function") defer span.End() err := doSomeWork() // ...Do some work if err != nil { span.SetStatus(codes.Error, "Operation failed") span.RecordError(err) } } ``` ## Additional Information For more use cases, see the [OpenTelemetry Go API documentation](https://opentelemetry.io/docs/languages/go/instrumentation/#creating-spans). # Go Hooks Source: https://docs.odigos.io/oss/instrumentations/golang/go-hooks Go hooks are currently available with the Odigos **Enterprise** plan.
[Contact us](https://odigos.io/) for more information. Odigos provides Go functions to augment and enhance auto-instrumentation. These functions are intended to be used in conjunction with auto-instrumentation to provide enriched functionality and interaction for developers running in an Odigos environment. # eBPF-Based Instrumentation Source: https://docs.odigos.io/oss/instrumentations/java/ebpf eBPF-Based Instrumentation is currently available with the Odigos **Enterprise** plan.
[Contact us](https://odigos.io/) for more information. For Native Instrumentation [click here](../../instrumentations/java/native). # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/java/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Add the following dependencies to your project: ```xml theme={null} io.opentelemetry opentelemetry-bom 1.35.0 pom import io.opentelemetry opentelemetry-api ``` ```groovy theme={null} dependencyManagement { imports { mavenBom("io.opentelemetry:opentelemetry-bom:1.35.0") } } dependencies { implementation("org.springframework.boot:spring-boot-starter-web"); implementation("io.opentelemetry:opentelemetry-api"); } ``` If you are not using Spring and its `io.spring.dependency-management` dependency management plugin, install the OpenTelemetry BOM and API using Gradle dependencies only. ```groovy theme={null} dependencies { implementation(platform("io.opentelemetry:opentelemetry-bom:1.35.0")); implementation("io.opentelemetry:opentelemetry-api"); } ``` ## Creating Spans To create a span, use the `Tracer` interface. The `Tracer` interface is the main entry point for the OpenTelemetry API. It allows you to create spans and manage the current span. Acquire a `Tracer` instance using the `OpenTelemetry` class: ```java theme={null} import io.opentelemetry.api.GlobalOpenTelemetry; Tracer tracer = GlobalOpenTelemetry.getTracer("instrumentation-scope-name", "instrumentation-scope-version"); ``` Now you can create a span using the `Tracer` instance: ```java theme={null} Span span = tracer.spanBuilder("span-name").startSpan(); try (Scope scope = span.makeCurrent()) { // Your code goes here. } finally { span.end(); } ``` ## Additional Information For more use cases, see the [OpenTelemetry Java API documentation](https://opentelemetry.io/docs/languages/java/instrumentation/#create-spans). # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/java/native Native Instrumentation is currently the default setting of Odigos **Open-Source** version. eBPF-Based Instrumentation is currently available with the Odigos Enterprise plan. Contact us for more information. ## Supported Versions Odigos uses the official OpenTelemetry instrumentation libraries and SDK components, thus it supports the same Java versions as this project. * Java Virtual Machine runtime versions **8** and above are supported. ## Traces Odigos will automatically instrument your Java sources to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Java Official Instrumentation Libraries. If you use one of the supported logger libraries, the trace id and span id will be auto-injected into your log records allowing signals correlations between logs and traces. ## Instrumentation Libraries The following Java modules will be auto instrumented by Odigos: ### HTTP Frameworks * [`play-ws`](https://central.sonatype.com/artifact/com.typesafe.play/play-ws_3) `versions >=1.0.0 <2.1.0` * [`jaxrs`](https://central.sonatype.com/artifact/com.jslsolucoes/jaxrs) `versions >=0.5` * [`resteasy-jaxrs`](https://central.sonatype.com/artifact/org.jboss.resteasy/resteasy-jaxrs) `versions >=3.0.0 <4.0.0` * [`jersey-server`](https://central.sonatype.com/artifact/com.sun.jersey/jersey-server) `versions >=2.0.0 <3.0.0` * [`jersey-container-servlet`](https://central.sonatype.com/artifact/org.glassfish.jersey.containers/jersey-container-servlet) `versions >=2.0.0 <3.0.0` * [`spring-web`](https://central.sonatype.com/artifact/org.springframework/spring-web) `versions >=3.1.0` * [`spring-webflux`](https://central.sonatype.com/artifact/org.springframework/spring-webflux) `versions >=5.0.0` * [`spring-webmvc`](https://central.sonatype.com/artifact/org.springframework/spring-webmvc) `versions >=3.1.0` * [`javalin`](https://central.sonatype.com/artifact/io.javalin/javalin) `versions >=5.0.0` * [`finatra-http`](https://central.sonatype.com/artifact/com.twitter/finatra-http_2.13) `versions >=2.9.0` * [`ktor-server-core`](https://central.sonatype.com/artifact/io.ktor/ktor-server-core) `versions >=2.0.0` ### HTTP Clients * [`okhttp`](https://central.sonatype.com/artifact/com.squareup.okhttp/okhttp) `versions >=3.0.0` * [`httpasyncclient`](https://central.sonatype.com/artifact/org.apache.httpcomponents/httpasyncclient) `versions >=4.1` * [`reactor-netty-http`](https://central.sonatype.com/artifact/io.projectreactor.netty/reactor-netty-http) `versions >=1.0.0` * [`cxf-rt-frontend-jaxrs`](https://central.sonatype.com/artifact/org.apache.cxf/cxf-rt-frontend-jaxrs) `versions >=3.2 <4.0.0` ### Database Clients, ORMs, and Data Access Libraries * [`jedis`](https://central.sonatype.com/artifact/redis.clients/jedis) `versions >=1.4.0 <4.0.0` * [`couchbase-java-client`](https://central.sonatype.com/artifact/org.testcontainers/couchbase) `versions >=2.0.0` * [`spring-data-commons`](https://central.sonatype.com/artifact/org.springframework.data/spring-data-commons) `versions >=1.8.0` * [`spring-batch-core`](https://central.sonatype.com/artifact/org.springframework.batch/spring-batch-core) `versions >=3.0.0 <5.0.0` * [`elasticsearch-java`](https://central.sonatype.com/artifact/co.elastic.clients/elasticsearch-java) `versions >=7.16 <7.17.20`, `versions >=8 <8.10` * [`elasticsearch-rest-client`](https://central.sonatype.com/artifact/org.elasticsearch.client/elasticsearch-rest-client) `versions >=6.4 <7.0` * [`elasticsearch-transport`](https://central.sonatype.com/artifact/org.elasticsearch/elasticsearch-transport-wares) `versions >=5.0 <8.0` * [`elasticsearch-rest`](https://central.sonatype.com/artifact/org.apache.camel/camel-elasticsearch-rest) `versions >=7.0` ### Messaging Systems Clients * [`spring-rabbit`](https://central.sonatype.com/artifact/org.springframework.amqp/spring-rabbit) `versions >=1.6.4 <2.1.1` * [`spring-kafka`](https://central.sonatype.com/artifact/org.springframework.kafka/spring-kafka) `versions >=2.7` * [`rocketmq-client-java`](https://central.sonatype.com/artifact/org.apache.rocketmq/rocketmq-client-java) `versions >=4.0.0` * [`reactor-kafka`](https://central.sonatype.com/artifact/io.projectreactor.kafka/reactor-kafka) `versions >=1.0.0` * [`spring-jms`](https://central.sonatype.com/artifact/org.springframework/spring-jms) `versions >=2.0` ### RPC (Remote Procedure Call) * [`dubbo`](https://central.sonatype.com/artifact/com.alibaba/dubbo) `versions >=2.7` * [`grpc-core`](https://central.sonatype.com/artifact/io.grpc/grpc-core) `versions >=1.6.0` * [`finagle-http`](https://central.sonatype.com/artifact/com.twitter/finagle-http_2.12) `versions >=23.11.0` ### Security * [`spring-security-config`](https://central.sonatype.com/artifact/org.springframework.security/spring-security-config) `versions >=6.0.0` ### Container Orchestration * [`kubernetes-client`](https://central.sonatype.com/artifact/io.fabric8/kubernetes-client) `versions >=7.0.0` ### Java Core Modules * [`spring-aop`](https://central.sonatype.com/artifact/org.springframework/spring-aop) `versions >=1.2` * [`spring-context`](https://central.sonatype.com/artifact/org.springframework/spring-context) `versions >=4.0.0 <6.0.0` * [`spring-core`](https://central.sonatype.com/artifact/org.springframework/spring-core) `versions >=2.0.` * [`spring-boot-actuator-autoconfigure`](https://central.sonatype.com/artifact/org.springframework.boot/spring-boot-actuator-autoconfigure) `versions >=2.0.0` * [`spring-cloud-gateway-core`](https://central.sonatype.com/artifact/org.springframework.cloud/spring-cloud-gateway-core) `versions >=2.0.0` * [`spring-ws-core`](https://central.sonatype.com/artifact/org.springframework.ws/spring-ws-core) `versions >=2.0.0` * [`spring-integration-core`](https://central.sonatype.com/artifact/org.springframework.integration/spring-integration-core) `versions >=4.1.0` * [`reactor-core`](https://central.sonatype.com/artifact/io.projectreactor/reactor-core) `versions >=3.1.0` * [`reactor-netty`](https://central.sonatype.com/artifact/io.projectreactor.netty/reactor-netty) `versions >=0.7.0` # eBPF-Based Instrumentation Source: https://docs.odigos.io/oss/instrumentations/nodejs/ebpf eBPF-Based Instrumentation is currently available with the Odigos **Enterprise** plan.
[Contact us](https://odigos.io/) for more information. For Native Instrumentation [click here](../../instrumentations/nodejs/native). # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/nodejs/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Add the following npm packages to your service by running: ```bash theme={null} npm install @opentelemetry/api@1.9.0 ``` Odigos agent implements OpenTelemetry API version 1.9.0. Any version greater than 1.9.0 may not be compatible with Odigos agent and fail to produce data.
Please do not use caret range ~~`@opentelemetry/api@^1.9.0`~~ for this dependency in your package.json to avoid pulling in incompatible version. ## Creating Spans To enhance your traces with spans for your custom business logic operations, you need to create a span in your code and wrap the operation you want to observe. The `tracer` object is the entry point for the OpenTelemetry API. You can create a `tracer` object using the `getTracer` method from the `@opentelemetry/api` package. ```javascript theme={null} const opentelemetry = require('@opentelemetry/api'); const tracer = opentelemetry.trace.getTracer( 'instrumentation-scope-name', 'instrumentation-scope-version', ); function myFunction() { const span = tracer.startSpan('myFunction'); // Your code here span.end(); } ``` ```typescript theme={null} import opentelemetry from '@opentelemetry/api'; const tracer = opentelemetry.trace.getTracer( 'instrumentation-scope-name', 'instrumentation-scope-version', ); function myFunction() { const span = tracer.startSpan('myFunction'); // Your code here span.end(); } ``` Make sure to replace `instrumentation-scope-name` and `instrumentation-scope-version` with the name and version of your instrumented file. Important Notes: 1. **Always End a span**: Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function. 2. **Propagate and use a valid context object**: When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace. 3. **Pass the context object downstream**: When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow. 4. **Assign meaningful names to spans**: Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context. 5. **Avoid dynamic, high cardinality data in span names**: Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace. ## Additional Information For more use cases, see the [OpenTelemetry JavaScript API documentation](https://opentelemetry.io/docs/languages/js/instrumentation/#create-spans). # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/nodejs/native Native Instrumentation is currently the default setting of Odigos **Open-Source** version. eBPF-Based Instrumentation is currently available with the Odigos Enterprise plan. Contact us for more information. ## Supported Versions Odigos uses the official OpenTelemetry instrumentation libraries and SDK components, thus it supports the same Node.js versions as the OpenTelemetry Node.js. * Node.js runtime versions **14** and above are supported. ## Traces Odigos will automatically instrument your Node.js sources to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Node.js Official Instrumentation Libraries. If you use one of the supported logger libraries, the trace id and span id will be auto-injected into your log records allowing signals correlations between logs and traces. ## Instrumentation Libraries The following npm packages will be auto instrumented by Odigos: ### Node.js Core Modules: * `node:dns` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:fs` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:http` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:https` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:net` [`Node.js`](https://nodejs.org/) versions `>=14` ### HTTP Frameworks: * [`connect`](https://www.npmjs.com/package/connect) versions `>=3.0.0 <4` * [`express`](https://www.npmjs.com/package/express) versions `>=4.0.0 <5` * [`fastify`](https://www.npmjs.com/package/fastify) versions `>=3.0.0 <5` * [`graphql`](https://www.npmjs.com/package/graphql) versions `>=14.0.0 <17` * [`@hapi/hapi`](https://www.npmjs.com/package/@hapi/hapi) versions `>=17.0.0 <22` * [`koa`](https://www.npmjs.com/package/koa) versions `>=2.0.0 <3` * [`@koa/router`](https://www.npmjs.com/package/@koa/router) versions `>=8.0.0` * [`@nestjs/core`](https://www.npmjs.com/package/@nestjs/core) versions `>=4.0.0 <11` * `node:http` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:https` [`Node.js`](https://nodejs.org/) versions `>=14` * [`restify`](https://www.npmjs.com/package/restify) versions `>=4.0.0 <12` * [`router`](https://www.npmjs.com/package/router) versions `>=1.0.0 <2` ### HTTP Clients: * `node:http` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:https` [`Node.js`](https://nodejs.org/) versions `>=14` * [`undici`](https://www.npmjs.com/package/undici) versions `>=5.12.0` ### RPC (Remote Procedure Call): * [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) versions `>=2.308.0 <3` * [`@aws-sdk/client-*`](https://www.npmjs.com/package/@aws-sdk/client-*) versions `>=3.0.0 <4` * [`@grpc/grpc-js`](https://www.npmjs.com/package/@grpc/grpc-js) versions `^1.0.0` ### Messaging Systems Clients: * [`amqplib`](https://www.npmjs.com/package/amqplib) versions `>=0.5.5 <1` * [`kafkajs`](https://www.npmjs.com/package/kafkajs) versions `>=0.3.0 <3` * `node:http` [`Node.js`](https://nodejs.org/) versions `>=14` * `node:https` [`Node.js`](https://nodejs.org/) versions `>=14` * [`socket.io`](https://www.npmjs.com/package/socket.io) versions `>=2.0.0 <5` ### Database Clients, ORMs, and data access libraries: * [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) versions `>=2.308.0 <3` * [`@aws-sdk/client-*`](https://www.npmjs.com/package/@aws-sdk/client-*) versions `>=3.0.0 <4` * [`cassandra-driver`](https://www.npmjs.com/package/cassandra-driver) versions `>=4.4.0 <5` * [`dataloader`](https://www.npmjs.com/package/dataloader) versions `>=2.0.0 <3` * [`generic-pool`](https://www.npmjs.com/package/generic-pool) versions `>=2.0.0 <4` * [`ioredis`](https://www.npmjs.com/package/ioredis) versions `>=2.0.0 <6` * [`knex`](https://www.npmjs.com/package/knex) versions `>=0.10.0 <4` * [`lru-memoizer`](https://www.npmjs.com/package/lru-memoizer) versions `>=1.3.0 <3` * [`memcached`](https://www.npmjs.com/package/memcached) versions `>=2.2.0 <3` * [`mongodb`](https://www.npmjs.com/package/mongodb) versions `>=3.3.0 <7` * [`mongoose`](https://www.npmjs.com/package/mongoose) versions `>=5.9.7 <7` * [`mysql`](https://www.npmjs.com/package/mysql) versions `>=2.0.0 <3` * [`mysql2`](https://www.npmjs.com/package/mysql2) versions `>=1.4.2 <4` * [`pg-pool`](https://www.npmjs.com/package/pg-pool) versions `>=2.0.0 <4` * [`pg`](https://www.npmjs.com/package/pg) versions `>=8.0.0 <9` * [`redis`](https://www.npmjs.com/package/redis) versions `>=2.6.0` * [`tedious`](https://www.npmjs.com/package/tedious) versions `>=1.11.0 <18` ### Loggers: Automatic injection of trace context (trace id and span id) into log records for the following loggers: * [`bunyan`](https://www.npmjs.com/package/bunyan) versions `>=1.0.0 <2` * [`pino`](https://www.npmjs.com/package/pino) versions `>=5.14.0 <10` * [`winston`](https://www.npmjs.com/package/winston) versions `>=1.0.0 <4` # OpenTelemetry eBPF Instrumentation (OBI) Source: https://docs.odigos.io/oss/instrumentations/obi OBI is OpenTelemetry’s eBPF-based auto-instrumentation. Odigos can run **[OpenTelemetry eBPF Instrumentation (OBI)](https://opentelemetry.io/docs/zero-code/obi)** for workloads where the `opentelemetry-ebpf-instrumentation` distro is selected or a Metrics `InstrumentationRule` is configured. Upstream concepts, requirements, and capabilities. Source, releases, and issues. ## Traces OBI tracing can be enabled by selecting it as a container override for specific workloads. This can be configured either from the Odigos UI or directly on a [Source object](/oss/pipeline/sources/introduction). ### Enabling OBI from the UI Select the source you want to instrument. In the source drawer, click the edit (pencil) icon next to the detected container. Edit detected container in the source drawer

Edit detected container in the source drawer

In the **Otel Distro Name** drop-down, choose **`opentelemetry-ebpf-instrumentation`** and click the check mark to save. Select opentelemetry-ebpf-instrumentation as the Otel distro

Select opentelemetry-ebpf-instrumentation as the Otel distro

After refreshing, the detected container should report that it is instrumented with `opentelemetry-ebpf-instrumentation`. Source instrumented with OBI

To remove OBI instrumentation, open the runtime override config again and click the delete (trash) icon next to the **Otel Distro Name** field. Remove OBI override from runtime config

By default, OBI does not need to restart your containers to begin instrumenting them. Because of this, choosing OBI as your container override will not trigger a pod restart. In some cases, however, your specific application may still require a pod restart (for example, when switching from a pre-existing instrumentation to OBI or vice-versa). You can trigger a pod restart using the **Restart** button in the source drawer. ### Enabling OBI from a Source object If editing a [Source object](/oss/pipeline/sources/introduction) directly, OBI is enabled through the `containerOverrides` field. For example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-workload namespace: default spec: workload: kind: DaemonSet name: sample-app namespace: default containerOverrides: - containerName: foo otelDistroName: opentelemetry-ebpf-instrumentation ``` In this case, `containerName` is the name of the container to instrument in this workload, and the required `otelDistroName` is **`opentelemetry-ebpf-instrumentation`**. ### Trace support When OBI is the selected distro, odiglet attaches the process to OBI's trace selector when instrumentation loads and removes it when instrumentation stops. OBI captures distributed traces for supported protocols (HTTP, gRPC, databases, messaging, and more) without code changes. DNS lookup traces are enabled by default when using the OBI distro. Learn more in the upstream docs: Context propagation, compatibility, and Kubernetes requirements. Protocol, database, messaging, and Go library instrumentation coverage. Odigos configures OBI header-based context propagation (`context_propagation: headers`). For encrypted traffic and multi-service propagation limits, see the upstream distributed tracing guide linked above. ### Limitations OBI's instrumentation is based on HTTP header propagation. For encrypted traffic, context propagation will not work unless both workloads (client and server) are instrumented with OBI as explained in [the official OBI docs](https://opentelemetry.io/docs/zero-code/obi/distributed-traces/#context-propagation-at-network-level). ## Metrics Upstream OBI is split into several observability pipelines. In Odigos, each signal is wired independently so traces and metrics can be enabled without requiring the same distro or lifecycle on every process. | Signal | OBI pipeline | Enabled in Odigos when | Applies to | | - | - | - | - | | **Traces** | [AppO11y](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main/pkg/appolly) (application protocol tracing) | The `opentelemetry-ebpf-instrumentation` distro is selected for the container. DNS lookup traces are included by default for the OBI distro | Processes instrumented with the OBI distro only | | **Network metrics** | [NetO11y](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main/pkg/netolly) (point-to-point flow bytes) and [StatsO11y](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main/pkg/statsolly) (TCP statistics such as RTT and failed connections) | An [InstrumentationRule](/oss/pipeline/rules/introduction) with `networkMetrics` set (optionally scoped via `scopes`) | Any process, regardless of language or distro (Go, Java, OBI, etc.) | All OBI telemetry that Odigos collects is exported over OTLP to the odiglet node collector (`localhost:4317`), then follows the same Odigos pipeline as other workload telemetry. For consistency and discoverability alongside other Odigos telemetry, the node collector rewrites the `obi` prefix of OBI metric names to `odigos`. Metrics named `obi.*` are exported as `odigos.*` (and the Prometheus form `obi_*` as `odigos_*`). For example, `obi.network.flow.bytes` is exported as `odigos.network.flow.bytes`. Metric names referenced in the OBI documentation below use the `obi` prefix; account for this rename when building dashboards or queries. OBI does not need to be selected as the traces distro for an app to use OBI metrics, see below. ### Configuration **Network metrics infrastructure (cluster-wide):** OBI network flow collection requires odiglet `hostNetwork`. Enable the infrastructure in Helm by setting `metricsSources.networkMetrics.enabled` to `true` (it defaults to disabled): ```yaml theme={null} metricsSources: networkMetrics: enabled: true ``` **Enabling network metrics:** Create an `InstrumentationRule` in the Odigos installation namespace (typically `odigos-system`) with a `networkMetrics` block. See [Network Metrics](/oss/pipeline/rules/networkmetrics) for configuration options, examples, and verification steps. Multiple rules merge with OR semantics (if any rule enables a signal, it is enabled). Workload scoping via `scopes` is optional — omit it to apply cluster-wide. ### Network metrics Network metrics describe bytes sent and received between endpoints (pods, services, nodes, and related Kubernetes metadata), along with TCP-level statistics such as round-trip time and failed connection attempts. Enable cluster infrastructure with `metricsSources.networkMetrics.enabled: true`, then enable collection with an InstrumentationRule `networkMetrics` block (cluster-wide or scoped via `scopes`). Network metrics are collected per process. In environments where a workload runs many processes, enabling network metrics can significantly increase the number of emitted metric series and the overall cardinality of your metrics. Monitor your metrics cardinality after enabling network metrics, and scope the rule via `scopes` to limit it to the workloads that need it. Flow and inter-zone metric families, attributes, and aggregation behavior. OpenTelemetry and Prometheus names for `obi.network.flow.bytes` and related series. ### Stats metrics Stats metrics (`obi.stat.tcp.rtt`, `obi.stat.tcp.failed.connections`, and related series) are collected together with network flow metrics when the Network Metrics rule is enabled. Metric names and attributes for `obi.stat.tcp.rtt`, `obi.stat.tcp.failed.connections`, and related series. How OBI groups metrics into `network`, `stats`, `stats_tcp_rtt`, and related feature flags. Network flow and TCP stats metrics export interval defaults to 60 seconds. # Overview Source: https://docs.odigos.io/oss/instrumentations/overview Odigos will automatically apply instrumentation to your selected workloads to record observability signals from your services such as collecting traces, metrics, and logs without any code changes. ## Supported Runtimes Odigos provides automatic instrumentations for the following runtimes: ## Enrich with OpenTelemetry APIs Odigos automatically generates data for open-source libraries and frameworks. Additional trace spans, metrics datapoints and log records can be added using the OpenTelemetry APIs. Odigos will automatically capture this data and deliver it to the chosen [destination](../backends-overview) alongside the automatically generated data. No need to configure the OpenTelemetry SDK, Odigos will automatically configure it for you. Select a language to learn how to enrich your data with OpenTelemetry APIs: ## Instrumentation Flow: 1. User selects a workload to auto instrument by creating a `Source` object (per workload or per namespace). 2. Odigos `instrumentor` component watches for changes with `Source` objects, and creates a relative `InstrumentationConfig` object. 3. Odigos `odiglet` component watches for changes with the `InstrumentationConfig` objects, and runs a runtime inspection on running pods to detect the programming language for every container. 4. Odigos `instrumentor` component watches for changes with the `InstrumentationConfig` object, and adds a resource request into each pod spec of all relevant containers in the workload manifest. These resource requests are called `Instrumentation Devices`. 5. Kubernetes detects the changes in the manifest, and rollout-restarts the pods with the new `Instrumentation Devices`. 6. New pods are scheduled and started. Odiglet resolves the resource request by mounting the auto instrumentation code and relevant environment variables into the container. 7. The auto instrumentation code starts the OpenTelemetry SDK and sends telemetry data to the odigos pipeline. ## Auto-rollback and Recover If pods enter CrashLoopBackOff or ImagePullBackOff after instrumentation is injected, Odigos **disables the agent** and restarts the workload **without** instrumentation. The UI may show that Odigos detected a crash and rolled back instrumentation to protect the workload. That rollback is **not** a Kubernetes Deployment undo. Odigos does not restore a previous ReplicaSet or Helm revision. It only removes Odigos agents from the workload. **Recover** (in the UI) means **try instrumentation again**. It clears the rollback latch and rolls the workload **with** agents. Clicking Recover does not remove remaining instrumentation or roll back the Deployment a second time. Disable automatic uninstrumentation with: ```bash theme={null} odigos install --set autoRollback.disabled=true ``` or Helm `autoRollback.disabled=true`. # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/php/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Add the following dependencies to your project by running: ```bash theme={null} composer require open-telemetry/api ``` ## Creating a Span To create a new span, use the `TracerProvider`. ```php theme={null} getTracer('demo'); $app = AppFactory::create(); $app->get('/rolldice', function (Request $request, Response $response) use ($tracer) { $span = $tracer ->spanBuilder('manual-span') ->startSpan(); $result = random_int(1,6); $response->getBody()->write(strval($result)); $span ->addEvent('rolled dice', ['result' => $result]) ->end(); return $response; }); $app->run(); ``` ## Additional Information For more use cases, see the [OpenTelemetry PHP API documentation](https://opentelemetry.io/docs/languages/php). # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/php/native This language instrumentation is using OpenTelemetry native-php SDK and instrumentation libraries. ## Supported Versions The minimal supported version of PHP is **8.1.0**. The maximal supported version of PHP is **8.4.x**, when PHP releases a new version we'll add the support for it. PHP version **8.0** has been deprecated and is no longer supported. PHP version **8.1** has limited library support and will be the next version to be deprecated. PHP version **8.2+** is the recommended version for new projects. ## Instrumentation Libraries The following PHP modules will be auto instrumented by Odigos: ### PHP Standards Recommendations (PSR): * [`psr/3 (log)`](https://packagist.org/packages/psr/log) versions `>=1` * [`psr/6 (cache)`](https://packagist.org/packages/psr/cache) versions `>=1` * [`psr/16 (simple-cache)`](https://packagist.org/packages/psr/simple-cache) versions `>=1` * [`psr/14 (event-dispatcher)`](https://packagist.org/packages/psr/event-dispatcher) versions `>=1` * [`psr/15 (http-server-middleware)`](https://packagist.org/packages/psr/http-server-middleware) versions `>=1` * [`psr/18 (http-client)`](https://packagist.org/packages/psr/http-client) versions `>=1` ### HTTP Clients: * [`php-http/httplug`](https://packagist.org/packages/php-http/httplug) versions `>=2` * [`guzzlehttp/guzzle`](https://packagist.org/packages/guzzlehttp/guzzle) versions `>=7` * [`ext-curl`](https://www.php.net/manual/en/book.curl.php) versions `*` `ext-curl` not supported for PHP 8.1 ### Frameworks: We're working on adding support for `laravel`, `cakephp`, `wordpress`, `yii2` and `codeigniter` frameworks. * [`slim/slim`](https://packagist.org/packages/slim/slim) versions `>=4` * [`symfony/http-kernel`](https://packagist.org/packages/symfony/http-kernel) versions `*` * [`symfony/http-client-contracts`](https://packagist.org/packages/symfony/http-client-contracts) versions `*` ### Databases: * [`ext-mongodb`](https://www.php.net/manual/en/book.mongodb.php) versions `>=1.13` * [`ext-mysqli`](https://www.php.net/manual/en/book.mysqli.php) versions `*` `ext-mysqli` not supported for PHP 8.1 * [`ext-pdo`](https://www.php.net/manual/en/book.pdo.php) versions `*` `ext-pdo` not supported for PHP 8.1 * [`doctrine/dbal`](https://packagist.org/packages/doctrine/dbal) versions `>=3` `doctrine/dbal` not supported for PHP 8.1 ### Messaging Systems: * [`ext-amqp`](https://pecl.php.net/package/amqp) versions `*` `ext-amqp` not supported for PHP 8.1 * [`ext-rdkafka`](https://pecl.php.net/package/rdkafka) versions `*` `ext-rdkafka` not supported for PHP 8.1 ### Atificial Intelligence: * [`openai-php/client`](https://packagist.org/packages/openai-php/client) versions `*` # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/python/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Install the API from PyPI using pip: ```bash theme={null} pip install opentelemetry-api ``` ## Creating Spans To create a span, use the `tracer` object from the `opentelemetry.trace` module. The `tracer` object is a factory for creating spans. ```python theme={null} from opentelemetry import trace tracer = trace.get_tracer(__name__) def my_function(): with tracer.start_as_current_span("my_function") as span: print("Hello world!") ``` Important Notes: 1. **Assign meaningful names to spans**: Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context. 2. **Avoid dynamic, high cardinality data in span names**: Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace. ### Recording Span Attributes Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting Examples: * User ID, organization ID, Account ID or other identifiers. * Inputs - the relevant parameters or configuration that influenced the operation. * Outputs - the result of the operation. * Entities - the entities or resources that the operation interacted with. Attribute names are lowercased strings in the form `my_application.my_attribute`, example: `my_service.user_id`. Read more [here](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/) To record attributes, use the `set_attribute` method on the span object. ```python theme={null} def my_function(arg: str): with tracer.start_as_current_span("my_function") as span: span.set_attribute("argument_name", arg) ``` Important Notes: 1. **Be cautious when recording data**: Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces. 2. **Attribute cost considerations**: Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data. 3. **Use static names for attributes**: Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute\_name) to provide clear context for downstream consumers. 4. **Adhere to OpenTelemetry semantic conventions**: Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand. ### Recording Errors To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts. If an operation you are tracing fails, you can mark the span's status as an error and record the error details within the span. Here's how you can do it: * An exception has been raised to demonstrate an error that occurred in your code. ```python theme={null} def my_function(): with tracer.start_as_current_span("my_function") as span: try: print("Hello world!") raise Exception("Some Exception") except Exception as e: span.record_exception(e) span.set_status(trace.Status(trace.StatusCode.ERROR, str(e))) ``` ## Advanced Capabilities The Odigos Python agent extends the standard OpenTelemetry SDK with capabilities that apply to **all spans** — including custom spans you create with the API above. These features are configured remotely through OpAMP and take effect without restarting your application. ### Head Sampling The Python agent supports [head sampling](../../pipeline/sampling/head-and-tail-sampling#head-sampling-agent) at the root span. Odigos pushes [Noisy Operation](../../pipeline/sampling/rules/noisy-operation) rules to the agent, which evaluates them through a custom `OdigosSampler` wired into the `TracerProvider`. Custom spans participate in the same sampling pipeline as auto-instrumented spans: * **Root spans** — If your custom span is the entry point of a trace (no active parent), head-sampling rules can match it and decide whether the entire trace is kept or dropped. * **Child spans** — Follow parent-based sampling: if the parent trace is sampled, custom child spans are recorded; if not, they are dropped (or recorded for metrics only — see [Span metrics mode](#span-metrics-mode) below). * **Context propagation** — Use `tracer.start_as_current_span(...)` (as shown above) so custom spans attach to the active trace and inherit the sampling decision. Head-sampling rules match HTTP server spans (`SpanKind.SERVER`) and HTTP client spans (`SpanKind.CLIENT`) using OpenTelemetry semantic conventions. The sampler accepts both stable and legacy attribute names — for example `http.request.method` and `http.method`, `server.address` and `http.host`. Server rules match on route (with `{param}` wildcard support), route prefix, and method. Client rules match on server address, templated path, path prefix, and method. When multiple rules match, the **lowest keep percentage** wins. Even when a trace is dropped, the agent attaches an `odigos` [tracestate](https://www.w3.org/TR/trace-context/#tracestate-header) entry so downstream services and the Odigos tail sampler can coordinate on the same decision. For rule configuration and evaluation order, see [Sampling rules overview](../../pipeline/sampling/rules/overview). ### Dry Run Mode When cluster-wide dry run (`samplingDryRun` in [OdigosConfiguration](../../api-reference/odigos.io.v1alpha1)) is enabled, the Python agent still exports every span but annotates the sampling decision in tracestate (`dry:t` if the trace would have been kept, `dry:f` if it would have been dropped). Use dry run to validate Noisy rules before committing to real drops. ### Span Metrics Mode The global `spanMetricsMode` setting controls what happens to spans that head sampling marks as not sampled: | Mode | Not-sampled spans | | - | - | | `sampled-spans-only` (default) | Dropped entirely — no trace export, no span metrics | | `all-spans` | Recorded with `RECORD_ONLY` — excluded from trace export but forwarded for span metrics computation | This applies to custom spans the same way it applies to auto-instrumented spans. If you rely on span metrics for RED dashboards, set `spanMetricsMode: all-spans` in your Odigos configuration so unsampled traffic still contributes to rate and error metrics. ### Agent Diagnostics Debug logging for the Odigos Python agent and its OpenTelemetry components can be toggled at runtime through an [InstrumentationRule](../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-InstrumentationRule) with `agentDiagnostics`: ```yaml theme={null} spec: agentDiagnostics: odigosLogLevel: debug openTelemetryComponentsLogLevel: debug ``` Supported levels are `error`, `warn`, `info`, and `debug`. Remote config from OpAMP **fully overrides** the startup environment variables — if a level is not set in the remote config, that logger is silenced. For the window before the first OpAMP message arrives, you can set startup log levels with environment variables: | Variable | Logger | Values | | - | - | - | | `ODIGOS_LOG_LEVEL` | Odigos agent (`odigos`) | `ALL`, `VERBOSE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `NONE` | | `OTEL_LOG_LEVEL` | OpenTelemetry SDK components | Same as above | At `debug` level, the sampler logs each rule evaluation — useful when troubleshooting why a custom or auto-instrumented span was kept or dropped. ## Additional Information For more use cases, see the [OpenTelemetry Python API documentation](https://opentelemetry.io/docs/languages/python/instrumentation/#creating-spans). # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/python/native Native Instrumentation is currently the default setting of Odigos **Open-Source** version. eBPF-Based Instrumentation is currently not supported for this language. ## Supported Versions Odigos uses the official [opentelemetry-python-instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation) OpenTelemetry Auto Instrumentation, thus it supports the same Python versions as this project. * In the enterprise version, Odigos leverages eBPF to enhance performance in the Python instrumentation process. * Odigos ships **two separate Python distros** — one for Python 3.8 and one for Python 3.9 and above. Odigos selects the correct distro automatically based on the runtime version of your workload. * **Python 3.8** uses a dedicated legacy distro with OpenTelemetry instrumentation `0.54b1`. * **Python 3.9 through 3.14** uses the main distro with the latest OpenTelemetry instrumentation. ## Traces Odigos will automatically instrument your Python services to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Python official auto Instrumentation Libraries. ## Instrumentation Libraries The following Python modules will be auto instrumented by Odigos. Supported libraries differ between the Python 3.8 and Python 3.9+ distros: The Python 3.8 distro includes the libraries below. It does not include Gen AI instrumentations other than OpenAI. ### Database Clients, ORMs, and Data Access Libraries * [`aiopg`](https://pypi.org/project/aiopg/) versions `aiopg >= 0.13.0, < 2.0.0` * [`dbapi`](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/dbapi/dbapi.html) * [`mysql`](https://pypi.org/project/mysql-connector-python/) version `mysql-connector-python >= 8.0, < 10.0` * [`mysqlclient`](https://pypi.org/project/mysqlclient/) version `mysqlclient < 3` * [`psycopg`](https://pypi.org/project/psycopg/) versions `psycopg >= 3.1.0` * [`psycopg2`](https://pypi.org/project/psycopg2/) versions `psycopg2 >= 2.7.3.1`, `psycopg2-binary >= 2.7.3.1` * [`pymemcache`](https://pypi.org/project/pymemcache/) versions `pymemcache >= 1.3.5, < 5` * [`pymssql`](https://pypi.org/project/pymssql/) versions `pymssql >= 2.1.5, < 3` * [`pymongo`](https://pypi.org/project/pymongo/) versions `pymongo >= 3.1, < 5.0` * [`pymysql`](https://pypi.org/project/PyMySQL/) versions `PyMySQL < 2` * [`redis`](https://pypi.org/project/redis/) versions `redis >= 2.6` * [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) versions `sqlalchemy >= 1.0.0, < 2.1.0` * [`sqlite3`](https://docs.python.org/3/library/sqlite3.html) * [`tortoiseorm`](https://pypi.org/project/tortoise-orm/) versions `tortoise-orm >= 0.17.0` * [`cassandra`](https://pypi.org/project/cassandra-driver/) versions `cassandra-driver ~= 3.25`, `scylla-driver ~= 3.25` * [`elasticsearch`](https://pypi.org/project/elasticsearch/) versions `elasticsearch >= 6.0` * [`asyncpg`](https://pypi.org/project/asyncpg/) versions `asyncpg >= 0.12.0` ### HTTP Frameworks * [`asgi`](https://pypi.org/project/asgiref/) versions `asgiref ~= 3.0` * [`django`](https://pypi.org/project/Django/) versions `django >= 1.10` Please note that for Django instrumentation to work, you need to set the [DJANGO\_SETTINGS\_MODULE](https://docs.djangoproject.com/en/5.1/topics/settings/#envvar-DJANGO_SETTINGS_MODULE) environment variable. Make sure to set this in your Kubernetes manifest or Dockerfile to ensure proper configuration. * [`fastapi`](https://pypi.org/project/fastapi/) versions `fastapi ~= 0.58` * [`flask`](https://pypi.org/project/Flask/) versions `flask >= 1.0` * [`pyramid`](https://pypi.org/project/pyramid/) versions `pyramid >= 1.7` * [`starlette`](https://pypi.org/project/starlette/) versions `starlette >= 0.13, < 0.15` * [`falcon`](https://pypi.org/project/falcon/) versions `falcon >= 1.4.1, < 5.0.0` * [`tornado`](https://pypi.org/project/tornado/) versions `tornado >= 5.1.1` ### HTTP Clients * [`aiohttp-client`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0` * [`httpx`](https://pypi.org/project/httpx/) versions `httpx >= 0.18.0` * [`requests`](https://pypi.org/project/requests/) versions `requests ~= 2.0` * [`urllib`](https://docs.python.org/3/library/urllib.html) * [`urllib3`](https://pypi.org/project/urllib3/) versions `urllib3 >= 1.0.0, < 3.0.0` ### Messaging Systems Clients * [`aio-pika`](https://pypi.org/project/aio-pika/) versions `aio_pika >= 7.2.0, < 10.0.0` * [`celery`](https://pypi.org/project/celery/) versions `celery >= 4.0, < 6.0` * [`confluent-kafka`](https://pypi.org/project/confluent-kafka/) versions `confluent-kafka >= 1.8.2, <= 2.7.0` * [`kafka-python`](https://pypi.org/project/kafka-python/) versions `kafka-python >= 2.0, < 3.0`, `kafka-python-ng >= 2.0, < 3.0` * [`pika`](https://pypi.org/project/pika/) versions `pika >= 0.12.0` * [`remoulade`](https://pypi.org/project/remoulade/) versions `remoulade >= 0.50` ### RPC (Remote Procedure Call) * [`grpc`](https://pypi.org/project/grpcio/) versions `grpcio >= 1.42.0` ### Web Servers * [`aiohttp-server`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0` * [`wsgi`](https://docs.python.org/3/library/wsgiref.html) ### Cloud Services and SDKs * [`boto`](https://pypi.org/project/boto/) versions `boto ~= 2.0` * [`boto3sqs`](https://pypi.org/project/boto3/) versions `boto3 ~= 1.0` * [`botocore`](https://pypi.org/project/botocore/) versions `botocore ~= 1.0` ### Framework and Library Utilities * [`jinja2`](https://pypi.org/project/Jinja2/) versions `jinja2 >= 2.7, < 4.0` ### Other * [`asyncio`](https://pypi.org/project/asyncio/) * [`click`](https://pypi.org/project/click/) versions `click >= 8.1.3, < 9.0.0` * [`threading`](https://docs.python.org/3/library/threading.html) ### Gen AI * [`openai`](https://pypi.org/project/openai/) versions `openai >= 1.26.0` ### Loggers Automatic injection of trace context (trace id and span id) into log records for the following loggers: * [`logging`](https://docs.python.org/3/library/logging.html) The Python 3.9+ distro includes the libraries below, including additional Gen AI instrumentations not available on Python 3.8. ### Database Clients, ORMs, and Data Access Libraries * [`aiopg`](https://pypi.org/project/aiopg/) versions `aiopg >= 0.13.0, < 2.0.0` * [`dbapi`](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/dbapi/dbapi.html) * [`mysql`](https://pypi.org/project/mysql-connector-python/) version `mysql-connector-python >= 8.0, < 10.0` * [`mysqlclient`](https://pypi.org/project/mysqlclient/) version `mysqlclient < 3` * [`psycopg`](https://pypi.org/project/psycopg/) versions `psycopg >= 3.1.0` * [`psycopg2`](https://pypi.org/project/psycopg2/) versions `psycopg2 >= 2.7.3.1`, `psycopg2-binary >= 2.7.3.1` * [`pymemcache`](https://pypi.org/project/pymemcache/) versions `pymemcache >= 1.3.5, < 5` * [`pymssql`](https://pypi.org/project/pymssql/) versions `pymssql >= 2.1.5, < 3` * [`pymongo`](https://pypi.org/project/pymongo/) versions `pymongo >= 3.1, < 5.0` * [`pymysql`](https://pypi.org/project/PyMySQL/) versions `PyMySQL < 2` * [`redis`](https://pypi.org/project/redis/) versions `redis >= 2.6` * [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) versions `sqlalchemy >= 1.0.0, < 2.1.0` * [`sqlite3`](https://docs.python.org/3/library/sqlite3.html) * [`tortoiseorm`](https://pypi.org/project/tortoise-orm/) versions `tortoise-orm >= 0.17.0` * [`cassandra`](https://pypi.org/project/cassandra-driver/) versions `cassandra-driver ~= 3.25`, `scylla-driver ~= 3.25` * [`elasticsearch`](https://pypi.org/project/elasticsearch/) versions `elasticsearch >= 6.0` * [`asyncpg`](https://pypi.org/project/asyncpg/) versions `asyncpg >= 0.12.0` ### HTTP Frameworks * [`asgi`](https://pypi.org/project/asgiref/) versions `asgiref ~= 3.0` * [`django`](https://pypi.org/project/Django/) versions `django >= 2.0` Please note that for Django instrumentation to work, you need to set the [DJANGO\_SETTINGS\_MODULE](https://docs.djangoproject.com/en/5.1/topics/settings/#envvar-DJANGO_SETTINGS_MODULE) environment variable. Make sure to set this in your Kubernetes manifest or Dockerfile to ensure proper configuration. * [`fastapi`](https://pypi.org/project/fastapi/) versions `fastapi ~= 0.92` * [`flask`](https://pypi.org/project/Flask/) versions `flask >= 1.0` * [`pyramid`](https://pypi.org/project/pyramid/) versions `pyramid >= 1.7` * [`starlette`](https://pypi.org/project/starlette/) versions `starlette >= 0.13` * [`falcon`](https://pypi.org/project/falcon/) versions `falcon >= 1.4.1, < 5.0.0` * [`tornado`](https://pypi.org/project/tornado/) versions `tornado >= 5.1.1` ### HTTP Clients * [`aiohttp-client`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0` * [`httpx`](https://pypi.org/project/httpx/) versions `httpx >= 0.18.0` * [`requests`](https://pypi.org/project/requests/) versions `requests ~= 2.0` * [`urllib`](https://docs.python.org/3/library/urllib.html) * [`urllib3`](https://pypi.org/project/urllib3/) versions `urllib3 >= 1.0.0, < 3.0.0` ### Messaging Systems Clients * [`aio-pika`](https://pypi.org/project/aio-pika/) versions `aio_pika >= 7.2.0, < 10.0.0` * [`celery`](https://pypi.org/project/celery/) versions `celery >= 4.0, < 6.0` * [`confluent-kafka`](https://pypi.org/project/confluent-kafka/) versions `confluent-kafka >= 1.8.2, < 3.0.0` * [`kafka-python`](https://pypi.org/project/kafka-python/) versions `kafka-python >= 2.0, < 3.0`, `kafka-python-ng >= 2.0, < 3.0` * [`pika`](https://pypi.org/project/pika/) versions `pika >= 0.12.0` * [`remoulade`](https://pypi.org/project/remoulade/) versions `remoulade >= 0.50` ### RPC (Remote Procedure Call) * [`grpc`](https://pypi.org/project/grpcio/) versions `grpcio >= 1.42.0` ### Web Servers * [`aiohttp-server`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0` * [`wsgi`](https://docs.python.org/3/library/wsgiref.html) ### Cloud Services and SDKs * [`boto3sqs`](https://pypi.org/project/boto3/) versions `boto3 ~= 1.0` * [`botocore`](https://pypi.org/project/botocore/) versions `botocore ~= 1.0`, `aiobotocore ~= 2.0` ### Framework and Library Utilities * [`jinja2`](https://pypi.org/project/Jinja2/) versions `jinja2 >= 2.7, < 4.0` ### Other * [`asyncio`](https://pypi.org/project/asyncio/) * [`click`](https://pypi.org/project/click/) versions `click >= 8.1.3, < 9.0.0` * [`threading`](https://docs.python.org/3/library/threading.html) ### Gen AI * [`anthropic`](https://pypi.org/project/anthropic/) versions `anthropic >= 0.3.11` * [`google-genai`](https://pypi.org/project/google-genai/) versions `google-genai >= 1.32.0` * [`langchain`](https://pypi.org/project/langchain-core/) versions `langchain-core > 0.1.0` * [`openai-agents`](https://pypi.org/project/openai-agents/) versions `openai-agents >= 0.3.3` * [`openai`](https://pypi.org/project/openai/) versions `openai >= 1.26.0` * [`vertexai`](https://pypi.org/project/google-cloud-aiplatform/) versions `google-cloud-aiplatform >= 1.64` * [`weaviate`](https://pypi.org/project/weaviate-client/) versions `weaviate-client >= 3.26.0, < 5` ### Loggers Automatic injection of trace context (trace id and span id) into log records for the following loggers: * [`logging`](https://docs.python.org/3/library/logging.html) # Enrich with OpenTelemetry API Source: https://docs.odigos.io/oss/instrumentations/ruby/enrichment Odigos will automatically instrument your services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs. However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code. This is sometimes referred to as `manual instrumentation`. ## Use Cases Examples of custom spans you might want to add to your code include: * Spans for the execution of some potentially heavy or slow computation in you service. * Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation. * Spans to describe some logical operations in your business logic that are meaningful in your domain. ## Required Dependencies Add the following dependencies to your project by running: ```bash theme={null} gem install opentelemetry-sdk ``` ## Creating a Span To acquire a tracer: ```ruby theme={null} # If in a Rails app, this lives in config/initializers/opentelemetry.rb require "opentelemetry/sdk" OpenTelemetry::SDK.configure do |c| c.service_name = '' end # 'MyAppTracer' can be used throughout your code now MyAppTracer = OpenTelemetry.tracer_provider.tracer('') ``` To create a new span: ```ruby theme={null} require "opentelemetry/sdk" def do_work MyAppTracer.in_span("do_work") do |span| # do some work that the 'do_work' span tracks! end end ``` ## Additional Information For more use cases, see the [OpenTelemetry Ruby API documentation](https://opentelemetry.io/docs/languages/ruby). # Automatic Instrumentation Source: https://docs.odigos.io/oss/instrumentations/ruby/native This language instrumentation is using OpenTelemetry native-ruby SDK and instrumentation libraries. ## Supported Versions The minimal supported version of Ruby is **3.1.0**. The maximal supported version of Ruby is **3.5.x**, when Ruby releases a new version we'll add the support for it. ## Instrumentation Libraries The following Ruby gems will be auto instrumented by Odigos: ### HTTP Clients: * [`http`](https://rubygems.org/gems/http) versions `*` * [`httpclient`](https://rubygems.org/gems/httpclient) versions `*` * [`restclient`](https://rubygems.org/gems/restclient) versions `*` * [`net-http`](https://rubygems.org/gems/net-http) versions `*` * [`faraday`](https://rubygems.org/gems/faraday) versions `*` * [`excon`](https://rubygems.org/gems/excon) versions `*` * [`ethon`](https://rubygems.org/gems/ethon) versions `*` ### gRPC Clients: * [`grpc`](https://rubygems.org/gems/grpc) versions `*` ### Web Frameworks & APIs: * [`rails`](https://rubygems.org/gems/rails) versions `*` * [`rack`](https://rubygems.org/gems/rack) versions `*` * [`sinatra`](https://rubygems.org/gems/sinatra) versions `*` * [`grape`](https://rubygems.org/gems/grape) versions `*` * [`gruf` (gRPC framework for Ruby)](https://rubygems.org/gems/gruf) versions `*` * [`racecar` (Kafka consumer framework for Rails)](https://rubygems.org/gems/racecar) versions `*` ### Processing & Task Runners: * [`rake`](https://rubygems.org/gems/rake) versions `>=13.0` * [`concurrent-ruby` (Threading and scheduling utilities)](https://rubygems.org/gems/concurrent-ruby) versions `*` ### Query & Schema APIs: * [`graphql`](https://rubygems.org/gems/graphql) versions `*` * [`koala` (Facebook Graph API client)](https://rubygems.org/gems/koala) versions `*` ### Messaging & Background Jobs: * [`sidekiq`](https://rubygems.org/gems/sidekiq) versions `*` * [`resque`](https://rubygems.org/gems/resque) versions `*` * [`delayed_job`](https://rubygems.org/gems/delayed_job) versions `*` * [`que`](https://rubygems.org/gems/que) versions `*` * [`bunny` (RabbitMQ)](https://rubygems.org/gems/bunny) versions `*` * [`ruby-kafka`](https://rubygems.org/gems/ruby-kafka) versions `*` * [`rdkafka`](https://rubygems.org/gems/rdkafka) versions `*` ### Databases & Caches: * [`pg` (PostgreSQL)](https://rubygems.org/gems/pg) versions `*` * [`mysql2`](https://rubygems.org/gems/mysql2) versions `*` * [`trilogy` (MySQL client by GitHub)](https://rubygems.org/gems/trilogy) versions `*` * [`mongo`](https://rubygems.org/gems/mongo) versions `*` * [`redis`](https://rubygems.org/gems/redis) versions `*` * [`dalli` (Memcached)](https://rubygems.org/gems/dalli) versions `*` * [`lmdb` (Lightning Memory-Mapped Database)](https://rubygems.org/gems/lmdb) versions `*` ### Cloud SDKs: * [`aws-sdk`](https://rubygems.org/gems/aws-sdk) versions `*` * [`aws-sdk-lambda`](https://rubygems.org/gems/aws-sdk-lambda) versions `*` ### Serialization: * [`active_model_serializers`](https://rubygems.org/gems/active_model_serializers) versions `>=0.10.0` # Overview Source: https://docs.odigos.io/oss/overview Odigos is an [open-source](https://github.com/odigos-io/odigos) observability control plane. Odigos leverages the power of [OpenTelemetry](https://opentelemetry.io) and [eBPF](https://ebpf.io) to automatically instrument applications. Odigos automatically deploys and scales collectors according to the traffic of applications. No need to waste time deploying and configuring collectors. The name Odigos originates from Greek, meaning "guide". ## Goals * **Automatic instrumentation** - Odigos automatically instruments your applications and produces distributed traces and metrics **without any code changes**. * **Collector management** - Odigos automatically deploys and scales collectors according to application traffic. No need to spend time deploying and configuring collectors. Odigos achieves its goals by deploying a set of components that work together to collect, process, and export telemetry data. ## Who is Odigos for? * Software Developers * Platform Engineers * IT Professionals * Anyone comfortable with command-line interfaces and basic cloud infrastructure Odigos is designed to be easy to use and deploy. It is a great tool for anyone who wants to get started with observability without the hassle of setting up and managing collectors. ## Adopt OpenTelemetry in minutes Odigos currently supports all the popular managed and open source [destinations](./backends-overview). By producing data in the [OpenTelemetry](https://opentelemetry.io) format, Odigos can be used with any observability tool that supports OTLP and allow you to easily switch to another vendor in the future. Use Odigos to generate distributed traces, metrics and logs with OpenTelemetry. Telemetry types that Odigos supports and produce. Programming languages that Odigos supports and applies auto-instrumentation to. Destinations that Odigos supports and sends telemetry data to. # Kubernetes RBAC Permissions Source: https://docs.odigos.io/oss/permissions This page lists the Kubernetes Roles and ClusterRoles used by Odigos and the Odigos Operator. # Components This section lists the RBAC policies used by the Odigos components. ## ClusterRoles Below are the ClusterRoles used by Odigos components. ### cleanup-clusterrole | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | odigos.io | sources | \* | list
delete | | \* | pods | \* | list | | \* | nodes | \* | list
patch | ### odigos-gateway | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | odigos.io | instrumentationconfigs | \* | get
list
watch | | \* | pods
namespaces | \* | get
list
watch | | apps | replicasets
deployments
statefulsets
daemonsets | \* | get
list
watch | ### odigos-go-offsets-updater | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | \* | list | ### odigos-instrumentor | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | nodes | \* | list
watch
get | | \* | namespaces | \* | list
watch
get | | \* | pods | \* | list
watch
get | | batch | cronjobs | \* | list
watch
get | | apps | daemonsets | \* | get
list
watch
update
patch | | apps | deployments | \* | get
list
watch
update
patch | | apps | statefulsets | \* | get
list
watch
update
patch | | \* | pods/finalizers
configmaps/finalizers | \* | update | | apps | statefulsets/finalizers
deployments/finalizers
daemonsets/finalizers | \* | update | | batch | cronjobs/finalizers
jobs/finalizers | \* | update | | argoproj.io | rollouts/finalizers | \* | update | | apps.openshift.io | deploymentconfigs
deploymentconfigs/finalizers | \* | get
list
watch
update
patch | | argoproj.io | rollouts | \* | get
list
watch
patch | | operator.odigos.io | odigos/finalizers | \* | update | | odigos.io | instrumentationconfigs/status | \* | get
patch
update | | odigos.io | instrumentationconfigs | \* | create
delete
get
list
patch
update
watch | | odigos.io | sources | \* | create
delete
get
list
patch
update
watch | | odigos.io | sources/finalizers | \* | update | | admissionregistration.k8s.io | mutatingwebhookconfigurations | \* | get
list
watch | | admissionregistration.k8s.io | mutatingwebhookconfigurations | odigos-source-mutating-webhook-configuration
odigos-pod-mutating-webhook-configuration | update | | admissionregistration.k8s.io | validatingwebhookconfigurations | \* | get
list
watch | | admissionregistration.k8s.io | validatingwebhookconfigurations | odigos-source-validating-webhook-configuration
odigos-action-validating-webhook-configuration | update | | apiregistration.k8s.io | apiservices | \* | get
list
watch
update
patch
delete | ### odiglet | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | pods
services | \* | get
list
watch | | \* | pods/status | \* | get | | \* | pods/finalizers | \* | update | | \* | nodes | \* | get
list
watch
patch
update | | odigos.io | instrumentationinstances | \* | create
get
list
patch
update
watch
delete | | odigos.io | instrumentationinstances/status | \* | get
patch
update | | odigos.io | instrumentationconfigs | \* | get
list
watch
patch
update | | odigos.io | instrumentationconfigs/status | \* | get
patch
update | | \* | nodes/stats | \* | get
list | | \* | pods
namespaces | \* | get
list
watch | | apps | replicasets
deployments
daemonsets
statefulsets | \* | get
list
watch | | discovery.k8s.io | endpointslices | \* | get
list
watch | ### odigos-ui | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | namespaces | \* | get
list
watch
patch | | apps | deployments
statefulsets
daemonsets | \* | get
list
watch
update
patch | | batch | cronjobs | \* | get
list
watch
update
patch | | apps.openshift.io | deploymentconfigs | \* | get
list
watch
update
patch | | argoproj.io | rollouts | \* | get
list
watch
update
patch | | apps | replicasets | \* | get
list | | \* | services | \* | get
list | | \* | pods | \* | get
list
watch
delete | | odigos.io | \* | \* | get
list
watch | | odigos.io | instrumentationconfigs
instrumentationinstances
sources
samplings | \* | update
patch
create
delete | | operator.odigos.io | odigos | \* | get
list
watch | | actions.odigos.io | \* | \* | get
list
watch | ## Roles Below are the Roles used by Odigos components. These Roles are only scoped to the Namespace in which Odigos is installed. ### cleanup-role | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | odigos-deployment
odigos-configuration | get | | \* | configmaps | \* | list | ### odigos-gateway | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | odigos-gateway | get
list
watch | ### odigos-go-offsets-updater | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | odigos-go-offsets | get
update | | \* | secrets | odigos-pro | get | ### odigos-instrumentor | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | \* | get
list
watch
create
patch
update
delete | | \* | services | \* | get
list
watch
create
patch
update
delete | | \* | pods | \* | get
list
watch | | apps | deployments | \* | create
delete
get
list
patch
update
watch | | apps | deployments/status | \* | get | | autoscaling | horizontalpodautoscalers | \* | create
patch
update
delete | | odigos.io | collectorsgroups | \* | get
list
create
patch
update
watch
delete | | odigos.io | collectorsgroups/status | \* | get
list
watch
patch
update | | odigos.io | destinations | \* | get
list
watch | | odigos.io | destinations/status | \* | get
patch
update | | odigos.io | instrumentationrules
processors | \* | get
list
watch
patch
update
delete
create | | odigos.io | processors/status | \* | get
patch
update | | odigos.io | instrumentationrules/status | \* | get
patch
update | | odigos.io | actions | \* | get
list
watch
patch
update
delete
create | | odigos.io | samplings | \* | get
list
watch | | odigos.io | actions/status | \* | get
patch
update | | odigos.io | recommendations | \* | get
list
watch
create
patch
update
delete | | odigos.io | recommendations/status | \* | get
patch
update | | actions.odigos.io | \* | \* | get
list
watch
update | | actions.odigos.io | \*/status | \* | get
patch
update | | odigos.io | samplings/status | \* | get
patch
update | | \* | secrets | \* | get
list
watch | | \* | secrets | instrumentor-webhooks-cert | update | | \* | secrets | autoscaler-webhook-cert
autoscaler-webhooks-cert | delete | | apps | daemonsets | odiglet | get
list
watch | | apps | deployments/finalizers | \* | update | | odigos.io | collectorsgroups/finalizers | \* | get
patch
update | | batch | cronjobs | odigos-go-offsets-updater | get
list
watch
create
update
patch
delete | | batch | jobs | \* | create | ### odigos-leader-election-role | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | events | \* | create
patch | | coordination.k8s.io | leases | \* | get
list
watch
create
update
patch
delete | ### odiglet | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | \* | list
watch | | \* | configmaps | odigos-data-collection
effective-config | get
list
watch | ### odigos-ui | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps | \* | get
list
watch
create
update
patch | | \* | secrets | \* | get
list
watch
create
patch
update
delete | | apps | replicasets | \* | get
list | | autoscaling | horizontalpodautoscalers | \* | get | | odigos.io | \* | \* | get
list
watch | | odigos.io | instrumentationrules
destinations
actions
samplings | \* | create
patch
update
delete | | odigos.io | recommendations | \* | update
patch | | odigos.io | collectorsgroups | \* | get
list
watch | | actions.odigos.io | \* | \* | get
list
watch
create
patch
update
delete | | operator.odigos.io | odigos | \* | get
list
watch | | apps | deployments/scale | odigos-instrumentor | get
update
patch | | apps | daemonsets | odiglet | get
update
patch | | \* | pods | \* | get
list
delete | | \* | pods/log | \* | get | | \* | pods/proxy | \* | get | # Operator This section lists the RBAC policies used by the Odigos Operator. Many of these permissions are necessary in order to create the RBAC policies for the components listed above. ## ClusterRoles | APIGroups | Resources | Resource Names | Verbs | | - | - | - | - | | \* | configmaps
endpoints
secrets
services | \* | create
delete
deletecollection
get
list
patch
update
watch | | \* | configmaps/finalizers
pods/finalizers | \* | update | | \* | events | \* | create
get
list
patch
watch | | \* | namespaces | \* | get
list
patch
watch | | \* | namespaces/status
nodes/spec
nodes/stats
replicationcontrollers
replicationcontrollers/status
resourcequotas | \* | get
list
watch | | \* | nodes | \* | get
list
patch
update
watch | | \* | pods | \* | delete
get
list
watch | | \* | pods/log
pods/proxy
pods/status | \* | get | | \* | serviceaccounts | \* | create
delete
get
list
patch
watch | | actions.odigos.io | \* | \* | create
delete
deletecollection
get
list
patch
update
watch | | actions.odigos.io | \*/status | \* | get
patch
update | | admissionregistration.k8s.io | mutatingwebhookconfigurations
validatingwebhookconfigurations | \* | create
delete
get
list
patch
update
watch | | apiextensions.k8s.io | customresourcedefinitions | \* | create
delete
deletecollection
get
list
patch
update
watch | | apiregistration.k8s.io | apiservices | \* | create
get
list
patch
update
watch | | apps | daemonsets
deployments
replicasets
statefulsets | \* | create
delete
deletecollection
get
list
patch
update
watch | | apps | daemonsets/finalizers
deployments/finalizers
replicasets/finalizers
statefulsets/finalizers | \* | update | | apps | daemonsets/status
deployments/status
statefulsets/status | \* | get | | apps | deployments/scale | \* | get
patch
update | | apps.openshift.io | deploymentconfigs
deploymentconfigs/finalizers | \* | get
list
patch
update
watch | | argoproj.io | rollouts | \* | get
list
patch
update
watch | | argoproj.io | rollouts/finalizers | \* | update | | autoscaling | horizontalpodautoscalers | \* | create
delete
get
list
patch
update
watch | | batch | cronjobs | \* | create
delete
get
list
patch
update
watch | | batch | cronjobs/finalizers
jobs/finalizers | \* | update | | batch | jobs | \* | create
delete
get
list
watch | | coordination.k8s.io | leases | \* | create
delete
get
list
patch
update
watch | | discovery.k8s.io | endpointslices | \* | get
list
watch | | extensions | daemonsets
deployments
replicasets | \* | get
list
watch | | odigos.io | \* | \* | \* | | odigos.io | collectorsgroups/finalizers
sources/finalizers | \* | update | | odigos.io | collectorsgroups/status
destinations/status
instrumentationconfigs/status
instrumentationinstances/status
processors/status | \* | get
list
patch
update
watch | | odigos.io | instrumentationrules/status
recommendations/status | \* | get
patch
update | | odigos.io | recommendations
sampling | \* | create
delete
get
list
patch
update
watch | | operator.odigos.io | odigos | \* | create
delete
get
list
patch
update
watch | | operator.odigos.io | odigos/finalizers | \* | update | | operator.odigos.io | odigos/status | \* | get
patch
update | | policy | podsecuritypolicies | privileged | use | | rbac.authorization.k8s.io | clusterrolebindings
clusterroles
rolebindings
roles | \* | create
delete
deletecollection
get
list
patch
update
watch | | security.openshift.io | securitycontextconstraints | \* | use | | authentication.k8s.io | tokenreviews | \* | create | | authorization.k8s.io | subjectaccessreviews | \* | create | # Add Cluster Info Source: https://docs.odigos.io/oss/pipeline/actions/attributes/addclusterinfo This action adds static resource attributes to spans, metrics data points and log records. It is useful to add cluster-wide attributes to all telemetry signals. ## Considerations Before enabling **add cluster info**, please note the following: * This action is meant to be used for **static** attributes that are known in advanced. It is not suitable for dynamic attributes that need to be extracted from an API or calculated at runtime. * Some destinations may require specific configuration to index resource attributes and make them available for querying. Please refer to the documentation of the destination you are using for more information. * If the attribute already exists in the telemetry signal, the value will be overwritten by the value provided in the action. * An empty string is a valid value for the `attributeStringValue` field ## Configuration Options The AddClusterInfo action is configured using the `odigos.io/v1alpha1.Action` CRD with the `addClusterInfo` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES`, `METRICS`, `LOGS` **addClusterInfo** `object` : Configuration for the AddClusterInfo action. * This field is *required* for this action type **clusterAttributes** `object[]` : An array of attributes to be added to the telemetry signals. * This field is *required* **attributeName** `string` : The name of the attribute to be added to the telemetry signals. * This field is *required* **attributeStringValue** `string` : The value of the attribute to be added to the telemetry signals. * This field is *required* **overwriteExistingValues** `boolean` : Whether to overwrite existing attribute values. * This field is *optional*, and defaults to `false` You can include any static value attribute that is meaningful to you, like `team.name` etc. We recommend: * `k8s.cluster.name`: The name of the k8s cluster which can be useful to distinguish between different clusters. * `deployment.environment`: The environment of the cluster (e.g. `production`, `staging`, `development`, etc). ## Basic Example The following example demonstrates how to insert the `k8s.cluster.name` attribute to all telemetry signals using the new Action CRD. Create a YAML file with the following content: ```yaml insert-cluster-name.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: insert-cluster-name namespace: odigos-system spec: actionName: insert-cluster-name signals: - TRACES - METRICS - LOGS addClusterInfo: clusterAttributes: - attributeName: k8s.cluster.name attributeStringValue: my-cluster ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f insert-cluster-name.yaml ``` # DB Query Templatization Source: https://docs.odigos.io/oss/pipeline/actions/attributes/dbquerytemplatization This action templatizes database query text by replacing number and string literals with placeholders to reduce cardinality. ## Considerations Before enabling **DB query templatization**, please note the following: * The action rewrites `db.query.text` (or the deprecated `db.statement`) in place when present. * Number and string literals are replaced with `?`. Identifiers, bind/positional parameters, booleans, and `NULL` are left unchanged. * Spans without a `db.system.name` (or the deprecated `db.system`) attribute, or with a known non-SQL system (e.g. MongoDB, Redis), are skipped. ## Use Cases **Cardinality control** * Raw SQL with embedded literals (e.g. `WHERE id = 1` vs `WHERE id = 2`) creates high-cardinality query attributes and span names. Templatizing literals collapses those into a shared template such as `WHERE id = ?`. **Cost and backend indexing** * Lower cardinality on database spans improves aggregations, span metrics, and indexing in your observability backend, and can reduce storage volume. **Sensitive values in queries** * Replacing string and number literals removes many accidental secrets or PII that were inlined into SQL (passwords, emails, IDs) while keeping the query structure intact. ## Configuration Options The DbQueryTemplatization action is configured using the `odigos.io/v1alpha1.Action` CRD with the `dbQueryTemplatization` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES` **dbQueryTemplatization** `object` : Configuration for the DbQueryTemplatization action. * This field is *required* for this action type **scopes** `object` : Limits which sources this templatization config applies to. * This field is *optional* * If unset or empty, the config is applied to all sources. * If multiple source scopes are set, they must all match simultaneously (AND logic). **sources** `object[]` : A list of workloads to apply this action to. * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, etc.). **namespaces** `string[]` : Apply this action to all sources in the listed namespaces. **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages. **templatizeLiterals** `boolean` : When enabled, number and string literals in SQL queries are replaced with placeholders. * This field is *optional*, and defaults to `false` * What is templatized: * number literals (e.g. `1`, `3.14`) * string literals (e.g. `'alice'`) * What is **not** templatized: * identifiers (table names, column names, etc.) * positional parameters (e.g. `$1`) * bind parameters (e.g. `:name`) * boolean literals and `NULL` * keywords, operators, punctuation, whitespace, comments, etc. ## Basic Example Given a span with: ```text theme={null} db.query.text: SELECT * FROM users WHERE id = 1 AND name = 'alice' ``` the action rewrites it to: ```text theme={null} db.query.text: SELECT * FROM users WHERE id = ? AND name = ? ``` Create a YAML file with the following content: ```yaml db-query-templatization.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: db-query-templatization namespace: odigos-system spec: actionName: DB Query Templatization signals: - TRACES dbQueryTemplatization: templatizeLiterals: true ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f db-query-templatization.yaml ``` ## Scoped Example To limit templatization to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources. ```yaml db-query-templatization-scoped.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: db-query-templatization-my-app namespace: odigos-system spec: actionName: DB Query Templatization for my-app signals: - TRACES dbQueryTemplatization: scopes: sources: - name: my-app namespace: default kind: Deployment templatizeLiterals: true ``` Pair this action with [Infer DB Attributes](./inferdbattributes) so database spans get both templatized query text and inferred `db.operation.name` / `db.collection.name` attributes. # Delete Attribute Source: https://docs.odigos.io/oss/pipeline/actions/attributes/deleteattribute This action will delete the specified attributes from all telemetry signals that are specified in the `signals` field, regardless of the source, or any other condition. ## Considerations Before enabling **delete attribute**, please note the following: * This action will delete selected attributes from spans, metric data points and log records, but will not delete the entire span, metric data point or log record. * For matching attributes, the attribute value will be deleted entirely. This action is not suitable for partial deletion or masking of the attribute value. For example, if you collect an `http.request.body` attribute, this action can be used to delete the attribute but not a key from the JSON object. * While it is common for information to be recorded in attributes, it can also be recorded into names or other fields, for which this action will not be effective. ## Use Cases **Security** * By default, OpenTelemetry should not record PII (Personally Identifiable Information) or sensitive data such as passwords, api tokens, creditcard numbers etc. However, if the data is accidentally added to the telemetry signals, or if a manual instrumentation recorded it into an attribute, this action can be used to delete such data. **Cost Reduction** * Some vendors charge based on the amount of data ingested. For self hosted destinations, the cost is correlated to the use of cloud resources which grows with the based on the amount of data you process and store. By deleting unnecessary attributes, you can reduce the amount of data ingested and reduce costs. **Usability** * Remove duplications - sometimes, the same attribute is added to the telemetry signals by multiple sources. This action can be used to delete the duplicates and keep the data clean. For example, you might find the host name recorded both as `host.name` and `k8s.node.name` which are the same value. * Verbosity - not all data is useful to all users. You might wish to delete certain attributes when they give no value to your organization or observability needs. Sometimes, recording too much data can obscure the important information and make it harder to find. ## Configuration Options The DeleteAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `deleteAttribute` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES`, `METRICS`, `LOGS` **deleteAttribute** `object` : Configuration for the DeleteAttribute action. * This field is *required* for this action type **attributeNamesToDelete** `string[]` : An array of strings representing the names of the attributes to be deleted from the telemetry signals. * This field is *required* ## Basic Example The following example demonstrates how to delete the `host.name` attribute from all telemetry signals using the new Action CRD. Create a YAML file with the following content: ```yaml delete-host-name.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: delete-host-name namespace: odigos-system spec: actionName: "delete host.name" signals: - TRACES - METRICS - LOGS deleteAttribute: attributeNamesToDelete: - host.name ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f delete-host-name.yaml ``` # Extract Attribute Source: https://docs.odigos.io/oss/pipeline/actions/attributes/extractattribute This action extracts values from span attributes (payloads, SQL statements, URL paths) into new top-level span attributes. ## Considerations Before enabling **extract attribute**, please note the following: * This action only operates on **trace** signals. * Extraction reads from a fixed set of high-cardinality string attributes and writes the captured value to a **new** span attribute. The source attributes are scanned in order until a match is found, and only the **first** match is used per extraction rule. * The action will scan the following span attributes for matches: * `db.statement` * `db.query.text` * `messaging.message.payload` * `http.request.payload` * `http.response.payload` * If the `targetAttributeName` attribute already exists on the span, it will **not** be overridden. * Each extraction must specify either `lookupKey` + `dataFormat`, **or** a custom `regex` — not both. When using `regex`, the pattern must contain exactly one capture group; the value of the first capture group is written to the attribute named `targetAttributeName`. * Each `targetAttributeName` must be unique within a single action. ## Use Cases **Accessing deeply nested attributes within strings** * Easily extract and access values of deeply nested attributes within payloads of JSONs, SQL statements or URLs. **Indexing high-cardinality identifiers** * Extract domain-specific identifiers (e.g. `order_id`, `user_id`, `study_id`) embedded inside payloads, SQL statements, or URL paths into top-level span attributes so they can be indexed, filtered, and aggregated by your backend. **Correlating spans across services** * Lift correlation IDs that are buried inside structured payloads (JSON envelopes, request/response bodies) onto the span itself so traces can be searched and joined by business identifiers. **Cost-aware enrichment** * Avoid sending the full payload to your backend by extracting only the meaningful fields, then dropping or masking the original payload attribute with another action. ## Configuration Options The ExtractAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `extractAttribute` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES` **extractAttribute** `object` : Configuration for the ExtractAttribute action. * This field is *required* for this action type **extractions** `object[]` : An ordered array of extraction rules to apply to each span. Must contain at least one entry. * This field is *required* **targetAttributeName** `string` : The name of the new span attribute the extracted value will be written to. * This field is *required* * Must be unique across all extractions in the same action. **lookupKey** `string` : The literal key to look up inside the scanned attributes (resolved via `dataFormat` into a regex). * This field is *required* when `dataFormat` is set, and must be omitted when `regex` is set. **dataFormat** `string` : A pre-set format that defines how `lookupKey` is matched inside the scanned attributes. * This field is *required* when `lookupKey` is set, and must be omitted when `regex` is set. * Supported values: * `json` — matches `key: value` pairs in JSON envelopes (e.g. `{"order_id": "42"}` or `order_id: 42`). * `sql` — matches `key = value` pairs in SQL statements (e.g. `WHERE order_id = '42'` or `order_id=42`). * `resource_path` — matches `key/value` segments inside URL paths (e.g. `/projects/test-project`), stopping at the next `/`, whitespace, `?`, `&`, `#`, or quote. **regex** `string` : A custom regular expression with a single capture group whose value is written to the attribute named `targetAttributeName`. * This field is *required* when `lookupKey` and `dataFormat` are not set. * The first capture group of the first match is used. Each extraction rule must use **exactly one** of: * `lookupKey` + `dataFormat` — for the built-in JSON/SQL/URL matchers, or * `regex` — for a custom pattern with a single capture group. ## Basic Example The following example demonstrates the four extraction variations against a single span that has the following attributes: * `messaging.message.payload`: `{"task_id": "task-7f3a9b2c", "status": "Completed"}` * `http.request.payload`: `tenants/acme-corp/regions/us-east/workspaces/main/services/analytics/jobs/task-7f3a9b2c/runs/run-001` * `db.statement`: `SELECT * FROM invoices WHERE invoice_id = '42' -- request_id req-xyz9` After the action runs, the span will be enriched with the following extracted attributes: | New attribute name | Extracted value | | - | - | | `extracted.json.task_id` | `task-7f3a9b2c` | | `extracted.sql.invoice_id` | `42` | | `extracted.url.tenant` | `acme-corp` | | `extracted.url.region` | `us-east` | | `extracted.url.job` | `task-7f3a9b2c` | | `extracted.regex.request_id` | `xyz9` | Create a YAML file with the following content: ```yaml extract-attribute.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: extract-attribute namespace: odigos-system spec: actionName: extract attribute signals: - TRACES extractAttribute: extractions: # JSON variation - matches a JSON envelope inside messaging.message.payload - lookupKey: task_id dataFormat: json targetAttributeName: extracted.json.task_id # SQL variation - matches `key = 'value'` syntax in SQL statements - lookupKey: invoice_id dataFormat: sql targetAttributeName: extracted.sql.invoice_id # URL variation - matches `key/value` segments inside http.request.payload - lookupKey: tenants dataFormat: resource_path targetAttributeName: extracted.url.tenant - lookupKey: regions dataFormat: resource_path targetAttributeName: extracted.url.region - lookupKey: jobs dataFormat: resource_path targetAttributeName: extracted.url.job # Custom regex variation - one capture group, value goes to the new attribute - regex: "req-([a-z0-9]+)" targetAttributeName: extracted.regex.request_id ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f extract-attribute.yaml ``` # Infer DB Attributes Source: https://docs.odigos.io/oss/pipeline/actions/attributes/inferdbattributes This action parses database query text and adds attributes such as db.operation.name and db.collection.name. ## Considerations Before enabling **infer DB attributes**, please note the following: * Currently, only trace signals are supported. * The action parses `db.query.text` when present, otherwise `db.statement`. * Existing `db.operation.name` / `db.collection.name` (or legacy `db.operation` / `db.sql.table`) values are **not** overwritten. * Attributes are added only when inference is unambiguous: a single SQL operation and a single table/collection. * When new operation (and optionally collection) attributes are added, the span name may be updated to `OPERATION collection` (e.g. `SELECT users`) if it does not already contain those values. * Spans without a `db.system` / `db.system.name` attribute, or with a known non-SQL system (e.g. MongoDB, Redis), are skipped. ## Use Cases **Enrich incomplete database spans** * Some instrumentations record only the raw query text. This action derives OpenTelemetry semantic attributes (`db.operation.name`, `db.collection.name`) so backends can filter and group by operation and table. **Better span names and metrics** * Normalized span names such as `SELECT users` make traces easier to read and improve aggregations when span metrics are enabled. **Consistent attributes across languages** * Apply the same inference rules cluster-wide, regardless of which language or library produced the span. ## Configuration Options The InferDbAttributes action is configured using the `odigos.io/v1alpha1.Action` CRD with the `inferDbAttributes` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES` **inferDbAttributes** `object` : Configuration for the InferDbAttributes action. * This field is *required* for this action type * No additional options are required; an empty object enables inference for matching sources. **scopes** `object` : Limits which sources this config applies to. * This field is *optional* * If unset or empty, the config is applied to all sources. * If multiple source scopes are set, they must all match simultaneously (AND logic). **sources** `object[]` : A list of workloads to apply this action to. * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, or `DaemonSet`). **namespaces** `string[]` : Apply this action to all sources in the listed namespaces. **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages. ## Basic Example Given a span with: ```text theme={null} db.query.text: SELECT * FROM users WHERE id = 1 ``` and without `db.operation.name` / `db.collection.name`, the action adds: | Attribute | Value | | - | - | | `db.operation.name` | `SELECT` | | `db.collection.name` | `users` | and updates the span name to `SELECT users` if it hasn't been set to that already. Create a YAML file with the following content: ```yaml infer-db-attributes.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: infer-db-attributes namespace: odigos-system spec: actionName: Infer DB Attributes signals: - TRACES inferDbAttributes: {} ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f infer-db-attributes.yaml ``` ## Scoped Example To limit inference to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources. ```yaml infer-db-attributes-scoped.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: infer-db-attributes-my-app namespace: odigos-system spec: actionName: Infer DB Attributes for my-app signals: - TRACES inferDbAttributes: scopes: sources: - name: my-app namespace: default kind: Deployment ``` Pair this action with [DB Query Templatization](./dbquerytemplatization) to both infer semantic attributes and replace literals in the query text. # Kubernetes Attributes Source: https://docs.odigos.io/oss/pipeline/actions/attributes/k8sattributes This action adds kubernetes related resource attributes to spans, metrics data points and log records. ## Overview By adding Kubernetes-specific attributes to telemetry signals, you can gain better visibility into workloads running in the cluster. Helps in correlating traces, logs and metrics with specific Kubernetes resources, such as pods, deployments, nodes and clusters. Adding resource attributes based on custom kubernetes annotations/labels can enrich the data with user defined aggregations and metadata corresponding to the originating pod that produced the telemetry. ## Configuration Options The K8sAttributes action is configured using the `odigos.io/v1alpha1.Action` CRD with the `k8sAttributes` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES, LOGS, METRICS` **k8sAttributes** `object` : Configuration for the K8sAttributes action. * This field is *required* for this action type **collectContainerAttributes** `boolean` : Collect the following container related attributes: * `k8s.container.name` * `container.id` * `container.image.name` * `container.image.tag` This field is *optional* and defaults to `false` **collectReplicaSetAttributes** `boolean` : Collect the following replica set related attributes: * `k8s.replicaset.name` if CollectWorkloadUID is set to true, the following attribute will be collected as well: * `k8s.replicaset.uid` This field is *optional* and defaults to `false` **CollectWorkloadUID** `boolean` : Collect the workload UID: * `k8s.deployment.uid` * `k8s.daemonset.uid` * `k8s.statefulset.uid` Only one of these attributes can appear for a given span/log/metric - based on the workload it originated from. This field is *optional* and defaults to `false` **CollectClusterUID** `boolean` : Collect the `k8s.cluster.uid` attribute, which is set to the uid of the namespace "kube-system". This allows aggregating observability based on the cluster it originated from. This field is *optional* and defaults to `false` **labelsAttributes** `K8sLabelAttribute[]` : Collect resource attributes based on the specified labels. Each label attribute can specify: * `labelKey`: The Kubernetes label key to extract * `attributeKey`: The OpenTelemetry attribute key to create * `from` (deprecated): Single source - "pod" or "namespace" * `fromSources`: Array of sources to extract from, with precedence (pod > namespace > node) When using `fromSources` with multiple sources, if the same label exists in multiple sources, the most specific source (e.g., pod) takes precedence over less specific sources (e.g., namespace). This field is *optional* and can remain empty. **annotationsAttributes** `K8sAnnotationAttribute[]` : Collect resource attributes based on the specified annotations. Each annotation attribute can specify: * `annotationKey`: The Kubernetes annotation key to extract * `attributeKey`: The OpenTelemetry attribute key to create * `from` (deprecated): Single source - "pod" or "namespace" * `fromSources`: Array of sources to extract from, with precedence (pod > namespace > node) When using `fromSources` with multiple sources, if the same annotation exists in multiple sources, the most specific source (e.g., pod) takes precedence. This field is *optional* and can remain empty. ## Basic Example The following example demonstrates how to collect Kubernetes attributes, including labels and annotations, and attach them to telemetry signals using the new Action CRD. Create a YAML file with the following content: ```yaml k8s-attributes.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: collect-k8s-attributes namespace: odigos-system spec: actionName: "Collect Kubernetes Attributes" signals: - TRACES - METRICS - LOGS k8sAttributes: collectContainerAttributes: true collectWorkloadUID: true collectClusterUID: true labelsAttributes: - labelKey: "app.kubernetes.io/name" attributeKey: "app.kubernetes.name" from: "pod" annotationsAttributes: - annotationKey: "kubectl.kubernetes.io/restartedAt" attributeKey: "kubectl.kubernetes.restartedAt" from: "pod" ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f k8s-attributes.yaml ``` ## Multi-Source Label Extraction Example The following example demonstrates how to extract the same label from multiple sources with proper precedence. This is useful when you want pod-level labels to override namespace-level labels when both exist. Create a YAML file with the following content: ```yaml k8s-attributes-multi-source.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: collect-environment-label namespace: odigos-system spec: actionName: "Collect Environment Label with Precedence" signals: - TRACES - METRICS - LOGS k8sAttributes: labelsAttributes: - labelKey: "environment" attributeKey: "k8s.environment" fromSources: - pod # Highest precedence - pod labels override namespace labels - namespace # Lower precedence - used as fallback ``` In this configuration: * If a pod has an `environment` label, that value will be used * If the pod doesn't have the label, but the namespace does, the namespace value will be used * This allows namespace-wide defaults while permitting pod-level overrides Apply the action to the cluster: ```bash theme={null} kubectl apply -f k8s-attributes-multi-source.yaml ``` # PII Masking Source: https://docs.odigos.io/oss/pipeline/actions/attributes/piimasking This action masks PII from traces using predefined categories and optional custom format or regex rules. ## Considerations Before enabling **pii masking**, please note the following: * Predefined PII categories replace matched values with a category-specific token (e.g. `***EMAIL***`, `***CREDIT_CARD***`). * Custom format and regex maskings replace only the matched capture group with `****`, leaving the rest of the value intact. * Currently, only trace signals are supported. * All span attribute values in every matching span will be examined and masked accordingly. ## Use Cases **Security** * By default, OpenTelemetry should not record PII (Personally Identifiable Information) or sensitive data such as passwords, api tokens, etc. However, if the data is accidentally added to the telemetry signals, or if a manual instrumentation recorded it into an attribute, this action can be used to mask such data. **Legal and Compliance** * Ensure compliance with legal and privacy. * Payment Card Industry (PCI) Data Security Standards prohibit logging certain things or storing them unencrypted. **Domain-specific secrets** * Mask application-specific fields (e.g. SSN in JSON payloads, passwords in SQL statements, or identifiers in URL paths) using `customFormatMaskings` or `customRegexMaskings` when they are not covered by a predefined PII category. ## Configuration Options The PiiMasking action is configured using the `odigos.io/v1alpha1.Action` CRD with the `piiMasking` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES` **piiMasking** `object` : Configuration for the PiiMasking action. * This field is *required* for this action type **scopes** `object` : Limits which sources this masking config applies to. * This field is *optional* * If unset or empty, the config is applied to all sources. **sources** `object[]` : A list of workloads to apply this action to. * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, or `DaemonSet`). **namespaces** `string[]` : Apply this action to all sources in the listed namespaces. **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages. **piiCategories** `string[]` : An array of strings representing the PII categories you want to mask. * This field is *optional* * The available PII categories are: * `CREDIT_CARD` - mask Visa and MasterCard credit card numbers * `EMAIL` - mask email addresses * `JWT` - mask JSON Web Tokens * `UUID` - mask UUIDs Can't find the PII category you need? Use `customFormatMaskings` or `customRegexMaskings`, or contact us and we will add it for you. **customFormatMaskings** `object[]` : Format-based masking rules applied in order. Each rule looks up a key inside structured attribute values and masks the matched value. * This field is *optional* **lookupKey** `string` : The field or path segment whose value should be masked (e.g. a JSON key, SQL column name, or URL path segment). * This field is *required* **dataFormat** `string` : The format of the data to search in. * This field is *required* * Supported values: * `json` — matches `key: value` pairs in JSON envelopes (e.g. `{"ssn": "123-45-6789"}` or `ssn: 123-45-6789`). * `sql` — matches `key = value` pairs in SQL statements (e.g. `WHERE password = 'hunter2'`). * `resource_path` — matches `key/value` segments inside URL paths (e.g. `/orders/abc-123`), stopping at the next `/`, whitespace, `?`, `&`, `#`, or quote. **customRegexMaskings** `object[]` : Regex-based masking rules applied in order. * This field is *optional* **regex** `string` : A custom regular expression with a single capture group whose value is masked with `****`. * This field is *required* * The pattern must contain at least one capture group; only the first capture group is replaced. ## Basic Example The following example demonstrates how to configure the Pii Masking action to mask Visa and MasterCard credit card numbers from span attribute values using the Action CRD. Create a YAML file with the following content: ```yaml pii-masking.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: pii-masking namespace: odigos-system spec: actionName: PII Masking signals: - TRACES piiMasking: piiCategories: - CREDIT_CARD ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f pii-masking.yaml ``` ## Custom Masking Example The following example combines a predefined category with custom format and regex rules. Given span attributes such as: * `{"ssn": "123-45-6789", "name": "alice"}` * `WHERE password = 'hunter2' AND status = 'ok'` * `/api/v1/orders/abc-123/items` * `auth api_key=super-secret-value next` the action produces: | Input fragment | Masked result | | - | - | | `user@example.com` | `***EMAIL***` | | `"ssn": "123-45-6789"` | `"ssn": "****"` | | `password = 'hunter2'` | `password = '****'` | | `/orders/abc-123/` | `/orders/****/` | | `api_key=super-secret-value` | `api_key=****` | Create a YAML file with the following content: ```yaml pii-masking-custom.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: pii-masking-custom namespace: odigos-system spec: actionName: Custom PII Masking signals: - TRACES piiMasking: piiCategories: - EMAIL customFormatMaskings: - lookupKey: ssn dataFormat: json - lookupKey: password dataFormat: sql - lookupKey: orders dataFormat: resource_path customRegexMaskings: - regex: 'api[_-]?key=([^\s&]+)' ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f pii-masking-custom.yaml ``` ## Scoped Example To limit masking to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources. ```yaml pii-masking-scoped.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: pii-masking-my-app namespace: odigos-system spec: actionName: PII Masking for my-app signals: - TRACES piiMasking: scopes: sources: - name: my-app namespace: default kind: Deployment piiCategories: - CREDIT_CARD - EMAIL ``` # Rename Attribute Source: https://docs.odigos.io/oss/pipeline/actions/attributes/renameattribute This action will rename the specified attributes from all telemetry signals that are specified in the `signals` field, regardless of the source, or any other condition. ## Configuration Options The RenameAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `renameAttribute` configuration section. **actionName** `string` : Allows you to attach a meaningful name to the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the action for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the action will operate on. * This field is *required* * Supported values: `TRACES`, `METRICS`, `LOGS` **renameAttribute** `object` : Configuration for the RenameAttribute action. * This field is *required* for this action type **renames** `object` : A map of strings, key representing the name of the attribute to rename and values the desired name. * This field is *required* ## Basic Example The following example demonstrates how to rename the deprecated `net.peer.address` attribute to stable `network.peer.address` in all telemetry signals using the new Action CRD. Create a YAML file with the following content: ```yaml rename-net-to-network.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Action metadata: name: rename-net-to-network namespace: odigos-system spec: actionName: Rename network attributes signals: - LOGS - METRICS - TRACES renameAttribute: renames: net.peer.address: network.peer.address net.local.address: network.local.address net.peer.ip: network.peer.address net.host.ip: network.local.address ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f rename-net-to-network.yaml ``` # Customized Source: https://docs.odigos.io/oss/pipeline/actions/crd You can define custom actions using OpenTelemetry Collector Processors in Odigos. This option is for advanced users who are familiar with the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), its [configuration](https://opentelemetry.io/docs/collector/configuration/), and with [Kubernetes CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). You can browse the full list of processors that Odigos includes [here](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61). To view the configuration for each processor, it is recommended to visit the README of this component in [OpenTelemetry Collector Contrib codebase](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor). Adding a processor to your Odigos pipeline can be done by creating a `Processor` custom resource in your Kubernetes cluster under the `odigos-system` namespace: ## Configuration Options **processorName** `string` : Allows you to attach a meaningful name to the processor for convenience. * This field is *optional* * **notes** `string` : Allows you to attach notes regarding the processor for convenience. * This field is *optional* * **disabled** `boolean` : Allows you to temporarily disable the processor, but keep it saved for future use. * This field is *optional*, and defaults to `false` **signals** `string[]` : An array with the signals that the processor will act on. * This field is *required* * Supported values: `TRACES`, `METRICS`, `LOGS` **collectorRoles** `string[]` : An array with the collector roles that the processor will act on. * This field is *required* * Supported values: `CLUSTER_GATEWAY`, `NODE_COLLECTOR` **type** `string` : The type of the processor. * This field is *required* * The name of the processor you want to use, as defined in the OpenTelemetry Collector (batch, attributes, etc). **orderHint** `number` : If your processors need to run in a specific order relatively to other processors, you can hint the order by setting an integer value here. * This field is *optional* * The lower the value, the earlier the processor will run in the collector pipeline. If the value is missing or 0, the processor will run in an arbitrary order in the collector pipeline. **processorConfig** `object` : A field to pass configuration to the processor. * This field is *optional* * The structure of this field is specific to each processor, and you can find the configuration options for each processor in the [OpenTelemetry Collector Contrib codebase](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor). ## Basic Example The following example demonstrates how to add a resource attribute `deployment.environment` with the value `production` to all spans in the cluster. Create a YAML file with the following content: ```yaml example-processor.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Processor metadata: name: example-processor namespace: odigos-system spec: type: resource processorConfig: attributes: - key: deployment.environment value: production action: insert signals: - TRACES collectorRoles: - CLUSTER_GATEWAY ``` Apply the action to the cluster: ```bash theme={null} kubectl apply -f example-processor.yaml ``` # Introduction Source: https://docs.odigos.io/oss/pipeline/actions/introduction Actions are a way to modify the OpenTelemetry data recorded by Odigos Sources, before it is exported to your destinations. ## What are Processors? Under the hood, Odigos uses the [OpenTelemetry Collector Processors](https://opentelemetry.io/docs/collector/configuration/#processors) to implement actions. Processors are a class of OpenTelemetry Collector component. They are used to transform, filter, or enrich your OpenTelemetry data before it is sent to your [destinations](../../backends-overview). ## Benefits * Odigos Actions are language agnostic. They run in the OpenTelemetry collectors and can act on any data type (traces, metrics, logs) no matter the source or the programming language of your service. * Processing the data on the k8s node or the cluster level reduces the load on your services and allows you to apply the same processing rules to all your services and signals. * Applying actions in your cluster can reduce the volume of data sent to your backend, which can reduce your costs. * OpenTelemetry maintains a large collection of processors, which are ready to use and can be configured to your needs. Actions are a powerful tool to help you manage your observability data and ensure that you are sending the right data to your backend. ## Action Types ### Attribute Actions Attribute Actions are a way to modify the OpenTelemetry data recorded by Odigos Sources, before it is exported to your destinations. * [Add Cluster Info](../../pipeline/actions/attributes/addclusterinfo) * [Delete Attribute](../../pipeline/actions/attributes/deleteattribute) * [Rename Attribute](../../pipeline/actions/attributes/renameattribute) * [Pii Masking](../../pipeline/actions/attributes/piimasking) * [Kubernetes Attributes](../../pipeline/actions/attributes/k8sattributes) * [Extract Attribute](../../pipeline/actions/attributes/extractattribute) * [DB Query Templatization](../../pipeline/actions/attributes/dbquerytemplatization) * [Infer DB Attributes](../../pipeline/actions/attributes/inferdbattributes) ### Sampling Actions Sampling Actions decide which traces are kept and which are dropped before they reach your destinations. Rules are organized into three categories, evaluated in order—**Noisy Operation → Highly Relevant → Cost Reduction**—so you can drop noise early, guarantee retention for important traffic, and cap everything else for cost. * [Sampling overview](../../pipeline/sampling/introduction) * [Head and tail sampling](../../pipeline/sampling/head-and-tail-sampling) * [Rules overview](../../pipeline/sampling/rules/overview) * [Configuration reference](../../pipeline/sampling/configuration) * [Examples](../../pipeline/sampling/examples) ### Custom Actions Odigos also supports adding OpenTelemetry processors with [Kubernetes CRDs](../../pipeline/actions/crd) which you can apply manually, or through a GitOps workflow. # Troubleshooting Source: https://docs.odigos.io/oss/pipeline/actions/troubleshooting If you are experiencing issues with your actions and processors, use the checklist below to troubleshoot the issue. ## View Collector Configurations To view the current configurations of the collectors, you can check the ConfigMap objects in the `odigos-system` namespace. You should see the processors configured under `data/conf/processors`, and actions configured under `data/conf/pipelines`. **Node Collector:** ```bash theme={null} kubectl get cm -n odigos-system -o yaml odigos-data-collection ``` **Gateway Collector:** ```bash theme={null} kubectl get cm -n odigos-system -o yaml odigos-gateway ``` ## View Instrumentor Logs The `odigos-instrumentor` controller is the component that watches for changes in the action or processor CRD and generates the collector configuration. If you don't see your processor in the collector configuration, you can check the logs of the instrumentor for errors or clues: ```sh theme={null} kubectl logs -n odigos-system deployments.apps/odigos-instrumentor ``` ## View Collector Logs If you do see your processor in the collector configuration, and there's an issue, the collector should log an error and will not start. To check the status of the Odigos pods, run: ```sh theme={null} kubectl get pods -n odigos-system ``` Look for pods with STATUS `CrashLoopBackOff` or `Error`.
To see the error, you can examine the logs of that pod: ```sh theme={null} kubectl logs -n odigos-system ``` You might find an error message like this: ``` * error decoding 'processors': unknown type: "invalid" for id: "invalid/example-processor" (valid values: [datadog experimental_metricsgeneration metricstransform resource servicegraph spanmetrics sumologic batch tail_sampling cumulativetodelta resourcedetection remotetap memory_limiter filter redaction routing span attributes deltatorate groupbyattrs groupbytrace k8sattributes probabilistic_sampler]) ``` If this is the case, then please follow the instructions in the error message to correct the processor configuration. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Pipeline Benchmarks Source: https://docs.odigos.io/oss/pipeline/benchmarks Choose the right ResourceSizePreset for your cluster based on telemetry volume and scale. The [ResourceSizePreset](./configuration#1-using-sizing-configuration) profiles (`size_s`, `size_m`, `size_l`, and `size_xl`) map collector resources and scaling headroom to telemetry volume. Use this page to assist you in picking a preset. ## Total span throughput Span throughput is the metric we use for benchmarking and estimating the resources needed to operate Odigos. To calculate the total span throughput (bytes/second), you can either: * Estimate your monthly span volume in bytes, which can often be inferred from your existing observability tools. If you only know the number of spans, multiply it by the average span size in bytes to estimate the total volume. Then convert the monthly volume to bytes per second. * View the span throughput in the Odigos UI under "collectors pipeline". Collector pipeline UI requires enabling the [Odigos Own Metrics](./odigos-own-metrics) feature. Note that while **total span throughput** is the primary metric for determining the appropriate Odigos preset size, additional factors can affect the maximum span throughput your Odigos deployment can handle: * **Number of instrumented pods** - each instrumented pod consumes a small amount of additional static resources in the node collector. * **Pipeline features** - actions and destinations consume additional collector CPU and memory resources. ## Cluster preset vs. total span throughput ### Maximal single collector capacity #### Cluster gateway (1 pod) ([see maximum capacity here](#maximum-scaled-cluster)) The cluster gateway is a Deployment that receives telemetry from node collectors, processes it, and exports it to your destinations. | Size | spans/s | span throughput (MB/second) | | - | -: | -: | | `size_s` | **3,000** | **5.8** | | `size_m` | **18,000** | **34.5** | | `size_l` | **23,000** | **44.1** | | `size_xl` | **31,500** | **60.4** | #### Node collector The node collector is a DaemonSet (one pod per node) that receives telemetry from apps on that node and forwards it to the cluster gateway. | Size | spans/s | span throughput (MB/second) | | - | -: | -: | | `size_s` | **23,500** | **45.1** | | `size_m` | **35,000** | **67.1** | | `size_l` | **49,500** | **94.9** | | `size_xl` | **54,500** | **104.5** | ### Maximum scaled cluster gateway capacity | Size | HPA Maximum Replicas | spans/s | span throughput (MB/second) | | - | -: | -: | -: | | `size_s` | **5** | **15,000** | **28.8** | | `size_m` | **8** | **144,000** | **276.2** | | `size_l` | **12** | **276,000** | **529.4** | | `size_xl` | **15** | **472,500** | **906.3** | \* Calculated with a span size of 1.9KB ## Troubleshooting If you experience any of the following symptoms consistently: * Memory limit hits or OOMKills * CPU throttling under steady load * Gateway HPA scaling to max replicas frequently * Export backlogs or dropped spans Consider selecting a larger preset or tune individual `collectorGateway` / `collectorNode` values on the [Configuration](./configuration) page. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Configuring the Pipeline Source: https://docs.odigos.io/oss/pipeline/configuration Configure the Odigos pipeline to optimize the performance and resource usage of the Odigos components. ## Configurable Components * **Cluster Gateway Collector** - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, processes it, and exports it to the configured destinations. * **Node Data Collection Collector** - a collector that runs as a k8s DaemonSet. It collects / scrape the OpenTelemetry data from the applications running on the nodes and forwards it to the Cluster Gateway Collector. ## Configuring the Pipeline Odigos offers two main options for configuring the pipeline: When configuring the pipeline, you must use **only one of the two methods** — either `profiles`, or directly editing the `odigos-configuration` ConfigMap. Using both methods will result in configurations overwriting each other, potentially leading to unexpected behavior. #### 1. Using Sizing Configuration To configure resource sizing at install time, set the `ResourceSizePreset` value in your Helm chart when installing Odigos—either in your `values.yaml` file or by using `--set ResourceSizePreset=` in the `helm install` command. This approach streamlines the setup process and applies optimized defaults for common use cases. The available sizing options—`size_s`, `size_m`, `size_l`, and `size_xl`—are predefined profiles designed to simplify pipeline configuration. Each profile specifies a defined set of parameters, as detailed below: Don't know which size is the right one for your cluster? Read more in [Benchmarks](./benchmarks). **Cluster Gateway Collector**: | Size | Minimum Replicas | HPA Maximum Replicas | Request CPU (m) | Limit CPU (m) | Request Memory (Mi) | Limit Memory (Mi) | | - | - | - | - | - | - | - | | `size_s` | **1** | **5** | **300m** | **300m** | **300Mi** | **300Mi** | | `size_m` | **2** | **8** | **1000m** | **1000m** | **600Mi** | **600Mi** | | `size_l` | **3** | **12** | **1250m** | **1250m** | **850Mi** | **850Mi** | | `size_xl` | **5** | **15** | **1500m** | **1500m** | **2000Mi** | **2000Mi** | **Node Data Collection Collector**: | Size | Request Memory (Mi) | Limit Memory (Mi) | Request CPU (m) | Limit CPU (m) | | - | - | - | - | - | | `size_s` | **150Mi** | **300Mi** | **150m** | **300m** | | `size_m` | **250Mi** | **500Mi** | **250m** | **500m** | | `size_l` | **500Mi** | **750Mi** | **500m** | **750m** | | `size_xl` | **1024Mi** | **1024Mi** | **1000m** | **1000m** | #### 2. Advanced Pipeline Configuration For advanced control over the pipeline, you can override the default sizing and fine-tune resource and scaling parameters for the `collectorGateway` and `collectorNode`. The way you apply these advanced settings depends on how you installed Odigos: * **If you installed Odigos using the CLI:** Edit the `odigos-configuration` ConfigMap directly in your cluster. This allows you to set advanced options for both `collectorGateway` and `collectorNode` (such as custom CPU/memory requests, limits, and replica counts). * **If you installed Odigos using Helm:** Make your changes in the `values.yaml` file or by using the `--set` flag during installation or upgrade. **Do not** edit the `odigos-configuration` ConfigMap directly, as Helm manages this resource and will overwrite manual changes. This method gives you full control over the pipeline configuration, letting you override any default or profile-based settings to match your specific requirements. Below is an example of how to configure the `collectorGateway` and `collectorNode` parameters: ```yaml theme={null} # HYBRID CONFIGURATION APPROACH: # By default, all values below are automatically set based on the ResourceSizePreset values.yaml attribute describe above. # Uncomment and modify any value to override the automatic sizing for that specific parameter. # # IMPORTANT CONSTRAINTS: # 1. minReplicas must be less or equal to maxReplicas # 2. If you set limitMemoryMiB without requestMemoryMiB, request will equal limit # 3. If you set requestMemoryMiB without limitMemoryMiB, limit will equal request # 4. Same logic applies to CPU settings collectorGateway: # the memory request for the cluster gateway collector deployment. # it will be embedded in the deployment as a resource request # of the form "memory: Mi". # default value is 600Mi # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value. # requestMemoryMiB: 600 # the memory limit for the cluster gateway collector deployment. # it will be embedded in the deployment as a resource limit # of the form "memory: Mi". # default value is 600Mi # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value. # limitMemoryMiB: 600 # the CPU request for the cluster gateway collector deployment. # it will be embedded in the deployment as a resource request # of the form "cpu: m". # default value is 1000m # If you set only requestCPUm, the limitCPUm will be set to the same value. # requestCPUm: 1000 # the CPU limit for the cluster gateway collector deployment. # it will be embedded in the deployment as a resource limit # of the form "cpu: m". # default value is 1000m # If you set only limitCPUm, the requestCPUm will be set to the same value. # limitCPUm: 1000 # The number of replicas for the cluster gateway collector deployment. # Also uses in MinReplicas the HPA config. # minReplicas: 2 # The maxReplicas in the HPA config. # maxReplicas: 8 # sets the "limit_mib" parameter in the memory limiter configuration for the collector gateway. # it is the hard limit after which a force garbage collection will be performed. # if not set, it will be 50Mi below the memory limit. # memoryLimiterLimitMiB: 550 # sets the "spike_limit_mib" parameter in the memory limiter configuration for the collector gateway. # note that this is not the processor soft limit, but the diff in MiB between the hard limit and the soft limit. # if not specified, this value will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit). # memoryLimiterSpikeLimitMiB: 110 # the GOMEMLIMIT environment variable value for the collector gateway deployment. # this is when go runtime will start garbage collection. # if not specified, it will be set to 80% of the hard limit of the memory limiter. # goMemLimitMiB: 440 collectorNode: # RequestMemoryMiB is the memory request for the node collector daemonset. # it will be embedded in the daemonset as a resource request of the form "memory: Mi" # default value is 250Mi # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value. # requestMemoryMiB: 250 # LimitMemoryMiB is the memory limit for the node collector daemonset. # it will be embedded in the daemonset as a resource limit of the form "memory: Mi" # default value is 500Mi # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value. # limitMemoryMiB: 500 # the CPU request for the node collector daemonset. # it will be embedded in the daemonset as a resource request # of the form "cpu: m". # default value is 250m # If you set only requestCPUm, the limitCPUm will be set to the same value. # requestCPUm: 250 # the CPU limit for the node collector daemonset. # it will be embedded in the daemonset as a resource limit # of the form "cpu: m". # default value is 500m # If you set only limitCPUm, the requestCPUm will be set to the same value. # limitCPUm: 500 # this parameter sets the "limit_mib" parameter in the memory limiter configuration for the node collector. # it is the hard limit after which a force garbage collection will be performed. # if not set, it will be 50Mi below the memory limit. # memoryLimiterLimitMiB: 450 # this parameter sets the "spike_limit_mib" parameter in the memory limiter configuration for the node collector. # note that this is not the processor soft limit, but the diff in Mib between the hard limit and the soft limit. # if not set, this will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit). # memoryLimiterSpikeLimitMiB: 90 # the GOMEMLIMIT environment variable value for the node collector daemonset. # this is when go runtime will start garbage collection. # if not specified, it will be set to 80% of the hard limit of the memory limiter. # goMemLimitMiB: 360 ``` ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Datastreams Source: https://docs.odigos.io/oss/pipeline/datastreams Datastreams let you split your observability data into multiple, independent pipelines - each routed to its own destination and optionally enhanced with custom actions like sampling, enrichment, or masking. This feature is in beta. It may be subject to changes and improvements based on user feedback. # Introduction Odigos supports Datastreams, a powerful feature that allows you to split telemetry sources into multiple logical streams. Each stream can be independently routed to different destinations and have specific actions applied—such as [sampling](../pipeline/sampling/introduction), attribute enrichment, masking rules, and more. # Motivation There are many use cases for adopting Datastreams, including: 1. **Cost optimization** - Route lower-priority services (like internal or backoffice apps) to more cost-effective destinations, such as Parquet files stored in S3. 2. **Targeted processing** - Apply custom processing (e.g., [sampling](../pipeline/sampling/rules/overview), enrichment) to only a subset of services. 3. **Gradual migration** - Test a new destination on specific services before committing to a full migration. 4. **Organizational segmentation** - Split data based on development teams, product areas, or environments. # Creating Datastreams After [installing Odigos CLI](../setup/installation), run the following command to start the Odigos UI: ```bash theme={null} odigos ui ``` ### In a new Odigos install You'll be taken to the Setup page. This is where your journey with Odigos begins, and you'll be asked to provide a few initial details: 2. Provide a name for the Datastream 3. Select the telemetry [Sources](../pipeline/sources/introduction) to include 4. Choose the [destination(s)](../backends-overview) to send the data to 5. Click **Done** ### In an existing Odigos install You can create a new Datastream from the Overview page. Click the `+ NEW` button at the top-left (under the current Datastream views) and follow the same steps as in the setup phase. This example demonstrates how to configure `example-source` to send telemetry to `jaeger-example` through the `example` Datastream. 1. Create or update a Source custom resource with the Datastream label, as described in the [linked documentation](../../pipeline/sources/create#workload-source). *For example:* ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default labels: odigos.io/data-stream-example: "true" spec: workload: name: frontend namespace: default kind: Deployment ``` 2. Create or Update a [destination](../../backends-overview) custom resource spec with the Datastream source selector. *For example:* ```yaml theme={null} sourceSelector: dataStreams: - example ``` Now let's take [Jaeger](../../backends/jaeger) as complete example here: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Destination metadata: name: jaeger-example namespace: odigos-system spec: data: JAEGER_URL: destinationName: jaeger signals: - TRACES type: jaeger sourceSelector: dataStreams: - example ``` > 💡 Tip: Even if you configure datastreams via manifests, you can still view and manage them through the Odigos UI for better visibility. # Caveats and Considerations 1. **Trace integrity** - Distributed traces often span multiple services that participate in the same operation. If you assign these services to different datastreams, the trace will be broken across destinations. Each destination will only receive part of the trace, making it incomplete and harder to troubleshoot. To preserve trace integrity, keep all services involved in a single transaction within the same datastream. 2. **Action support** - At present, actions defined in Odigos are applied globally to all streams. Stream-specific actions (e.g., custom [sampling rules](../pipeline/sampling/rules/overview) per datastream) are not yet supported but will be in a future release. # Odigos Own Metrics Source: https://docs.odigos.io/oss/pipeline/odigos-own-metrics Enable Odigos self-telemetry and the bundled VictoriaMetrics store for pipeline visibility in the UI. ## Overview Odigos collects **own metrics** about its pipeline — collector throughput, export rates, eBPF instrumentation counters, and other internal telemetry. These metrics describe Odigos itself, not your application workloads. By default, the bundled metrics store is **disabled** (`ownTelemetry.metricsStore.disabled: true`). When you enable it, Odigos deploys a VictoriaMetrics instance (`odigos-victoriametrics`) in the Odigos namespace and routes own metrics there so the UI can query them. ## What enabling the metrics store does When `ownTelemetry.metricsStore.disabled` is set to `false`: 1. **VictoriaMetrics is deployed** — a single-replica `odigos-victoriametrics` Deployment with 1-day retention stores own metrics inside the cluster. 2. **Collectors report own telemetry** — node collectors and the cluster gateway scrape their OpenTelemetry internal `/metrics` endpoints and forward the data through the gateway to VictoriaMetrics. 3. **Odiglet instrumentation metrics are collected** — node collectors scrape eBPF instrumentation counters from odiglet and forward them through the gateway to the store. 4. **Odigos UI representation** — metrics are display under "collectors pipeline". Separately from the metrics store, collectors always export traffic-size metrics directly to the UI server over OTLP. That path powers **per-source and per-destination throughput** (`throughputBytes`, `totalDataSent`) in the UI without requiring the metrics store. ## Enabling The metrics store is disabled by default to keep the Odigos footprint minimal on small clusters. ```yaml theme={null} ownTelemetry: metricsStore: disabled: false ``` ```bash theme={null} helm upgrade --install odigos odigos/odigos \ --namespace odigos-system \ --reuse-values \ --set ownTelemetry.metricsStore.disabled=false ``` ## Configuration options ### Export to external metrics destinations You can also send own metrics to your existing metrics backends by enabling **Collect Odigos own metrics** on a metrics destination. When either the metrics store or at least one such destination is enabled, collectors start the own-metrics collection pipeline. See [Odigos own metrics settings](/api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-OdigosOwnMetricsSettings) in the API reference for the full `OdigosOwnMetricsSettings` schema. ## Resource footprint The bundled VictoriaMetrics instance requests roughly: | Resource | Request | Limit | | - | - | - | | CPU | 100m | 500m | | Memory | 512Mi | 2Gi | Metrics are retained for **1 day** on ephemeral storage (`emptyDir`). ## Using own metrics for sizing Own metrics help you understand how much telemetry Odigos is processing before you change [ResourceSizePreset](./configuration#1-using-sizing-configuration) or collector resources. 1. Enable the metrics store. 2. Instrument representative workloads and let traffic run for a few minutes. 3. Check **source throughput** and **destination throughput** in the Odigos UI. 4. Compare total span throughput (bytes/sec) with the [Benchmarks](./benchmarks) preset tables to pick `size_s` through `size_xl`. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Overview Source: https://docs.odigos.io/oss/pipeline/overview ## High Level Architecture Odigos Architecture

The Odigos Pipeline is composed of the following components: * **Instrumentations** - code that records OpenTelemetry signals from your application. It can be code in your application native programming language or eBPF programs that record OpenTelemetry data from the linux kernel. * **OpenTelemetry SDK** - a library that runs in each instrumented process. It receives the OpenTelemetry data from instrumentations and export it out of the process boundary. * **Node Collector** - also referred to as `data-collection collector`. Each node in the cluster runs a single instance of this OpenTelemetry collector (DaemonSet). It receives the OpenTelemetry data from the OpenTelemetry SDKs, process it, and exports it to the next component in the pipeline. * **Cluster Gateway Collector** - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, process it, and export it to the next component in the pipeline. ## Component types: * **Kubernetes Operators** - reconcile the desired state of cluster resources and deploy / configure the necessary components. * **DaemonSets** - perform node-level tasks, such as interacting with a process running on the node OS, accessing the Kubelet, and reading logs from the node filesystem. * **Deployments** - run cluster-level components like the OpenTelemetry Collector and the Odigos UI. ## Component Segments Odigos is divided into three main segments: * Instrumentation * Inject OpenTelemetry agents to collect the instrumentation data from Odigos sources (instrumented applications) * Record and report runtime information from the process environment that is relevant to instrumentation * Export collected telemetry to the pipeline. * Pipeline * Offload common or heavy tasks from the instrumented application runtime * Apply Odigos Actions on the telemetry data * Batch and send the telemetry data to the configured destinations * Odigos Management * orchestrate the pipeline and the instrumentation based on the user's configuration and cluster state, * provide a UI for the odigos control plane. ## Pipeline ### Instrumentor (Pipeline Manager) **Type**: The instrumentor is a Kubernetes operator
**Role**: The instrumentor is responsible for managing the OpenTelemetry Collectors which implement the OpenTelemetry pipeline.
**K8s Resource**: `kubectl get deployments -n odigos-system odigos-instrumentor` * Start and stop Collectors when needed (e.g. when sources or destinations are added or removed). * Schedule the node collector to start only after the cluster collector is ready. * Generate the configuration for the Collectors based on Odigos sources, destinations, and Actions. ### Cluster Collector (Gateway Collector) **Type**: Cluster collector is a deployment of the OpenTelemetry collector
**Role**: The cluster collector receives and processes all telemetry collected in the cluster, applies Actions, and exports data to the configured destinations.
**K8s Resource**: `kubectl get deployments -n odigos-system odigos-gateway` * Batch telemetry data for better performance and export it to the configured destinations * Modify the telemetry data based on the Odigos Actions. * Apply [tail sampling](./sampling/head-and-tail-sampling) based on the Odigos [sampling rules](./sampling/rules/overview) to reduce the volume of data sent to the destinations. * Collect metrics about the entities and report it to the UI for monitoring. * Offload tasks from the agents, to reduce the overhead on the workload runtime environment * Centralize any common processing that can be shared across multiple agents. ### Node Collector **Type**: Node collector is a daemon set and will run one pod per node in the cluster
**Role**: The Node Collector (in the context of Pipeline component) receives telemetry data from agents running on the node, enriches it with node-level attributes, and forward it efficiently and reliably to the cluster collector.
**K8s Resource**: `kubectl get daemonsets -n odigos-system odigos-data-collection` * Receive telemetry data from agents running on the node (Odiglet for eBPF based agents, and instrumentation processes for native OpenTelemetry agents). * Add node-level attributes to the telemetry data (`k8s.node.name`, cloud attributes, etc). * Batch and send telemetry data to the Cluster Collector, offloading any network overhead from the instrumented application runtime. The Node Collector is also responsible for recording some k8s metrics, and collecting the generated logs from the node fs. Read more Under the [Instrumentation](#node-collector-2) section. ## Instrumentation ### Instrumentor (Auto Instrumentation Agent Manager) **Type**: Instrumentor is a Kubernetes operator
**Role**: The Instrumentor is responsible for managing auto instrumentation agents which collect telemetry data from Odigos sources.
**K8s Resource**: `kubectl get deployments -n odigos-system odigos-instrumentor` * Create and manage `instrumentationconfig.odigos.io` objects * Trigger Odiglet to inspect runtime information of workloads. * Trigger deployment of the Odigos pipeline. * Merge Odigos instrumentation rules and config to a workload scope configuration for the agent. * Inject the instrumentation device into the workload manifest resource as virtual device. * Monitor the `source.odigos.io` CRD and cleanup any instrumentation related resources when a source is removed. * Monitor the Odigos sources and destinations to trigger deployment of the Odigos pipeline. * Reconcile Odigos configuration to run validations, set defaults, and apply Odigos Profiles (configuration presets) to the cluster. ### Odiglet **Type**: Odiglet is a daemon set and will run one pod per node in the cluster
**Role**: Odiglet sets up the node environment to run telemetry collection agents.
**K8s Resource**: `kubectl get daemonsets -n odigos-system odiglet` * Inspect container runtime details of instrumented sources (to determine programming language, relevant environment variables, runtime versions, etc). * Implement the Kubernetes virtual instrumentation devices. In the presence of a supported device on the pod, Odiglet will mount the agent code from file system to the pod and add the relevant environment variables to initiate instrumentation. * For eBPF-based instrumentation, Odiglet will inject the eBPF code into the pod, pick up telemetry from eBPF, and push it downstream to the Node Collector. * Monitor for config changes to instrumentations and update the relevant agents accordingly at runtime. ### Node Collector **Role**: Node collector (in the context of instrumentation) is responsible for scraping Kubernetes metrics, and collect logs from the node filesystem. * When a metrics destination exists, the node collector will use the OpenTelemetry Collector ["kubeletstats receiver"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver) to scrape k8s related metrics from the Kubelet and export them as OpenTelemetry metrics to the cluster collector. * When a logs destination exists, the node collector will use the OpenTelemetry Collector ["filelog receiver"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) to read the relevant pod logs from the node filesystem and export them as OpenTelemetry logs to the cluster collector. ## Odigos ### Odigos UI **Type**: Odigos UI is a web application backed by a deployment in the cluster and accessible by the user's browser.
**Role**: The Odigos UI is a web application that allows users to configure various aspects of the Odigos installation.
**K8s Resource**: `kubectl get deployments -n odigos-system odigos-ui` * Serve as web server for the Odigos UI * Serve static files to the frontend * Provide a GraphQL API for the frontend, which in turn interacts with Kubernetes API to manage the Odigos installation. * Provide a user interface for monitoring and managing the Odigos installation, including the sources, destinations, actions, and instrumentation rules. * Examine the current state of Odigos and show any errors. # Code Attributes Source: https://docs.odigos.io/oss/pipeline/rules/codeattributes The "Code Attributes" rule can be used to add semantic Code attributes to spans. Code attributes can help trace back an operation in a trace to the code that generated it. This Instrumentation Rule is currently only available with the Odigos **Enterprise** plan.
[Contact us](https://odigos.io/) for more information. # Custom Instrumentation Source: https://docs.odigos.io/oss/pipeline/rules/custominstrumentation The "Custom Instrumentation" rule can be used to instrument specific functions in PHP applications, or (with Odigos Enterprise) arbitrary Go/Java functions via eBPF. ## Considerations In Odigos open source, Custom Instrumentation is supported for **PHP** applications using the community native OpenTelemetry distro. Custom eBPF-based instrumentations for Go and Java require [Odigos Enterprise](https://odigos.io/pricing). PHP probes are delivered as environment variables and require a pod restart to take effect. ## Configuration Options **className** `string` - The fully qualified class name to instrument. Leave empty for a global function. - This field is *optional* **functionName** `string` - The name of the function or method to instrument. - This field is *required* ## Basic Example The following example instruments the `processOrder` method on `App\Service\OrderService`. Create a YAML file named `custom-instrumentation.yaml` with the following content: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: InstrumentationRule metadata: name: custom-instrumentation-rule namespace: odigos-system spec: ruleName: "Custom Instrumentation for processOrder" customInstrumentations: php: - className: "App\\Service\\OrderService" functionName: "processOrder" ``` Apply the rule to the cluster: ```shell theme={null} kubectl apply -f custom-instrumentation.yaml ``` Restart the PHP workload so the new `ODIGOS_PHP_AGENT_CUSTOM_INSTRUMENTATIONS` environment variable is picked up. # Headers Collection Source: https://docs.odigos.io/oss/pipeline/rules/headerscollection The "Headers Collection" rule can be used to add span attributes by collecting HTTP Headers data. This Instrumentation Rule is currently only available with the Odigos **Enterprise** plan.
[Contact us](https://odigos.io/) for more information. # Introduction Source: https://docs.odigos.io/oss/pipeline/rules/introduction Instrumentation Rules control how telemetry is recorded from your application. A rule can be applied to a set of Kubernetes workloads and instrumentation libraries. ## Rule Types: * [Code Attributes](./codeattributes) * [Headers Collection](./headerscollection) * [Payload Collection](./payloadcollection) * [Custom Instrumentations](./custominstrumentation) * [Network Metrics](./networkmetrics) ## Configuration Options **ruleName** `string` : Allows you to attach a meaningful name to the rule for convenience and documentation. * This field is *optional* * Odigos does not use or assume any meaning from this field **notes** `string` : Allows you to attach notes regarding the rule for convenience. * This field is *optional* * Odigos does not use or assume any meaning from this field **disabled** `boolean` : Allows you to temporarily disable the rule, but keep it saved for future use. * This field is *optional*, and defaults to `false` **workloads** `[namespace, name, kind]` : Allows you to apply/limit the rule to specific workloads. If not specified, the rule will be applied to all workloads. * This field is *optional*, and defaults to `nil` (all workloads) * Empty array will make the rule ineffective **namespace** `string` - The name of the Kubernetes namespace. * This field is *required* **name** `string` - The name of the Kubernetes workload. * This field is *required* **kind** `string` - The kind of the Kubernetes workload. * This field is *required* * Supported values: `Deployment`, `DaemonSet`, `StatefulSet` **instrumentationLibraries** `[name, language, spanKind]` : Allows you to apply/limit the rule to specific instrumentation libraries. If not specified, the rule will be applied to all instrumentation libraries. * This field is *optional*, and defaults to `nil` (all libraries) * Empty array will make the rule ineffective **name** `string` - The unique name of the instrumentation library. * This field is *required* **language** `string` - The programming language of the instrumentation library. * This field is *required* * Alphabetical, lowercase, and no spaces **spanKind** `string` - The span kind of the instrumentation library. * This field is *required* * Supported values: `client`, `server`, `producer`, `consumer`, `internal` ## Apply to Sources Odigos allows you to apply/limit rules to specific sources. * To apply a rule to all sources, omit the `workloads` field. * To apply a rule to specific sources, provide an array of workload objects (identified by its namespace, name, and kind). Odigos allows you to define multiple rules, and will merge them together to create a single rule per source. Conflicting options will be resolved by choosing to record the attribute, if it's specified by one rule (at least). ## Apply to Instrumentation Libraries Odigos allows you to apply/limit rules to specific instrumentation libraries. * To apply a rule to all instrumentation libraries, omit the `instrumentationLibraries` field. * To apply a rule to specific instrumentation libraries, provide an array of instrumentation libraries (identified by its unique name, language, and span kind for golang). Any rule for a specific instrumentation library will take precedence over a rule that does not specify a library. This is an advanced feature. It is recommended to set the same rules for all instrumentation libraries, and only use this feature when you need fine-grained control. # Network Metrics Source: https://docs.odigos.io/oss/pipeline/rules/networkmetrics Enable network flow and TCP stats metrics for instrumented workloads via InstrumentationRule The **Network Metrics** rule enables network flow metrics and TCP stats metrics for matching workloads. These metrics are collected in odiglet and exported over OTLP alongside other workload telemetry. Learn more about the underlying collectors and metric families in the [OBI instrumentation guide](/oss/instrumentations/obi). ## Considerations Before enabling **network metrics**, please note the following: * Enable the cluster infrastructure in Helm by setting `metricsSources.networkMetrics.enabled` to `true` (it defaults to disabled). ## Configuration Options **networkMetrics** `object` : Enable network flow and TCP stats metrics for scoped workloads. Setting this field (even as an empty object `{}`) enables collection; omit it to leave metrics disabled. Multiple rules merge with OR semantics: if any matching rule enables network metrics, they are enabled for that source/language. ## Cluster infrastructure for network flow metrics Network flow metrics require odiglet to run with [`hostNetwork`](https://kubernetes.io/docs/reference/kubernetes-api/core/pod-v1/#:~:text=%60hostNetwork%60), which makes the pod share the host's network namespace instead of getting its own. Configure this once at the cluster level in Helm: ```yaml theme={null} metricsSources: networkMetrics: enabled: true ``` The InstrumentationRule alone is not enough for network flow metrics without that Helm setting. ## Basic Example Create `InstrumentationRule` resources in the Odigos installation namespace (typically `odigos-system`). Rules in other namespaces are not picked up by Odigos. The following example enables network metrics cluster-wide for all workloads. Set `metricsSources.networkMetrics.enabled` to `true` in your Odigos Helm values and upgrade the release. Create a YAML file with the following content: ```yaml network-metrics.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: InstrumentationRule metadata: name: network-metrics namespace: odigos-system spec: ruleName: "Network metrics" networkMetrics: {} ``` ```shell theme={null} kubectl apply -f network-metrics.yaml ``` Network metrics collection will run on any new pod created after the rule is applied. Existing pods will not be affected and require a restart to initiate metrics collection. ## Scoped Example To limit the rule to a single workload, add `scopes`. Omit `scopes` (or leave it empty) to apply the rule cluster-wide. ```yaml network-metrics-scoped.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: InstrumentationRule metadata: name: network-metrics-my-app namespace: odigos-system spec: ruleName: "Network metrics for my-app" scopes: sources: - name: my-app namespace: default kind: Deployment networkMetrics: {} ``` Apply with: ```shell theme={null} kubectl apply -f network-metrics-scoped.yaml ``` ## Exported Metrics When this rule is enabled, the following metrics are exported for matching workloads: | Name (OpenTelemetry) | Name (Prometheus) | Type | Unit | Description | | - | - | - | - | - | | `odigos.network.flow.bytes` | `odigos_network_flow_bytes_total` | Counter | bytes | Bytes sent from a source network endpoint to a destination network endpoint | | `odigos.network.flow.packets` | `odigos_network_flow_packets_total` | Counter | Packets sent from a source network endpoint to a destination network endpoint | | | `odigos.network.inter.zone.bytes` | `odigos_network_inter_zone_bytes_total` | Counter | bytes | Bytes flowing between cloud availability zones in your cluster (Kubernetes only) | | `odigos.stat.tcp.rtt` | `odigos_stat_tcp_rtt_seconds` | Histogram | seconds | TCP round-trip time (RTT) latency observed between network endpoints | | `odigos.stat.tcp.failed.connections` | `odigos_stat_tcp_failed_connections_total` | Counter | 1 | Failed TCP connection attempts between endpoints, labeled by failure reason | These metrics are collected using [OBI](https://opentelemetry.io/docs/zero-code/obi). See the [OBI instrumentation guide](/oss/instrumentations/obi) to learn more about the collectors, metric families, and available attributes. # Payload Collection Source: https://docs.odigos.io/oss/pipeline/rules/payloadcollection The "Payload Collection" rule can be used to add span attributes by collecting payload data from various sources like HTTP requests, HTTP responses, database queries, and messaging systems. This Instrumentation Rule is currently only available with the Odigos **Enterprise** plan.
[Contact us](https://odigos.io/) for more information. # Configuration Source: https://docs.odigos.io/oss/pipeline/sampling/configuration Configure sampling actions in Odigos Open Source. You can configure sampling rules in the **UI** or as **YAML** (Kubernetes **CRDs**). The schema matches the same concepts as the UI, so GitOps workflows can express the same behavior as in the product. Every rule has three parts: a **Source scope** that defines which workloads the rule applies to, an **Operation** that picks the specific operations within that scope, and a **Keep Percentage** that controls how much matching traffic is retained. The Source scope options are identical across all three categories; the Operation and Keep Percentage options differ by category. Each rule can also carry an optional **Name** and **Notes** for easier identification. This page is the **field reference**. For end-to-end walkthroughs—UI steps and the equivalent YAML side by side—see [Examples](./examples). ## Source scope A rule can be scoped to one of: * **Entire cluster** — the rule applies to every source in the cluster. * **Specific Source** — one or more named sources (services). * **Specific Namespace** — one or more Kubernetes namespaces. * **Specific Programming Language** — one or more runtimes: **Python**, **Java**, **NodeJS**, **.Net**, **PHP**, **Ruby**, or **Go**. You can select multiple values within a scope type—for example several namespaces, or several languages. When you populate more than one scope type on the same rule, the selections are combined with **AND**, not **OR**. A rule scoped to "Specific Namespace = `prod`" plus "Specific Programming Language = Java" matches **only** Java workloads in the `prod` namespace; it does not match every workload in `prod` or every Java workload across the cluster. ## Operation The Operation options depend on the rule's category. * **All Operations** — every operation in scope. * **HTTP Server** — match HTTP requests received by a server. Configure: * An **HTTP route** (exact path) or **HTTP route prefix**. * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**. * **HTTP Client** — match outbound HTTP calls made by the source. Configure: * A **server address** (the remote host being called). * A **templated path** (exact) or **templated path prefix**. * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**. * **gRPC Server** — match incoming gRPC calls received by a server. Configure: * A **service** — the fully-qualified gRPC service name (e.g. `acme.inventory.v1.InventoryService`). Set only the service to match every method on that service. * A **method** — the bare method name (e.g. `ListItems`). Combine with **service** for a single specific RPC. * **gRPC Client** — match outbound gRPC calls made by the source. Configure: * A **service** — the fully-qualified gRPC service name (e.g. `acme.inventory.v1.InventoryService`). Set only the service to match every method on that service. * A **method** — the bare method name (e.g. `ListItems`). Combine with **service** for a single specific RPC. * A **server address** (the remote host being called). * **All Operations** — every operation in scope. * **HTTP Server** — match HTTP requests received by a server. Configure: * An **HTTP route** (exact path) or **HTTP route prefix**. * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**. * **Kafka Consumer** — match traces from a Kafka consumer. Configure the **Kafka topic**. * **Kafka Producer** — match traces from a Kafka producer. Configure the **Kafka topic**. HTTP Client is **not** available in this category. * **All Operations** — every operation in scope. * **HTTP Server** — match HTTP requests received by a server. Configure: * An **HTTP route** (exact path) or **HTTP route prefix**. * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**. * **Kafka Consumer** — match traces from a Kafka consumer. Configure the **Kafka topic**. * **Kafka Producer** — match traces from a Kafka producer. Configure the **Kafka topic**. HTTP Client is **not** available in this category. ## Keep Percentage Each rule defines what fraction of matching traffic is retained. The available options differ by category: | Category | Options | | - | - | | Noisy Operation | **Drop All**, or **Keep at most** *N*% | | Highly Relevant | **Keep All**, or **Keep at least** *N*% | | Cost Reduction | **Keep at most** *N*% (set *N* = 0 to drop everything) | **Keep at most** is a cap; **Keep at least** is a floor. See [Rule evaluation](./rules/overview#rule-evaluation) for how these percentages interact within a category and between Highly Relevant and Cost Reduction. ## Enabled Every rule has an **Enabled** toggle that controls whether it participates in sampling decisions. Disabled rules stay in your configuration but are excluded from evaluation, which makes the toggle useful for several patterns: * **Temporary suspension.** Turn a rule off during an incident, debugging session, or evaluation period when you need broader visibility—without losing the rule's configuration. * **Disable vs. delete.** Disabling preserves the matcher, scope, and Keep Percentage so you can re-enable later without re-authoring. Use it when a rule may still be useful later but isn't needed right now. * **"What-if" metrics.** Disabled rules continue to record matching metrics: how many traces *would have* matched, and how many would have been kept or dropped if the rule were active. Useful for sizing a candidate rule's impact before committing to it. See [Disabling a rule](./examples#disabling-a-rule) for UI screenshots and the equivalent YAML. ## Name and Notes Both fields are optional and exist purely to make rules easier to identify and maintain—they have **no effect** on evaluation or matching. * **Name** — a short, free-text label for the rule. The name appears in the rule list in the UI and may also surface on metrics and span attributes, so use something descriptive that makes the rule recognizable in dashboards and queries. * **Notes** — free-text describing **why** the rule exists. Use this to capture context that isn't obvious from the matcher itself: the incident that triggered the rule, the team that owns it, or the cost target it's serving. Especially useful when auditing rules or onboarding new operators. Treat Name and Notes like a commit message for your sampling policy. A good Name + Notes pair turns "what does this rule do?" into a one-line answer at a glance. # Considerations Source: https://docs.odigos.io/oss/pipeline/sampling/considerations Operational and design considerations for sampling in Odigos Open Source. Operational and design considerations to keep in mind when running sampling in production. ## Gateway load, OOM, and scaling When the gateway accepts a trace for tail evaluation, it allocates memory to hold spans until the [aggregation window](./head-and-tail-sampling#aggregation-window) closes. If the gateway isn't sized correctly, bursts of traffic can cause **OOM** restarts. Symptoms include **dropped spans** or collector crashes that aren't caused by your sampling rules. For tail sampling, prefer **scaling up** (bigger collector pods) over **scaling out** (more replicas). All spans of a single trace must land on the same pod for aggregation, so fewer, larger pods simplify routing and load balancing. Validate sizing against your own load—there's no one-size-fits-all. ## When any sampler enables tail mode Any sampling rule that requires tail evaluation activates **tail mode** for the entire cluster, even rules that look inactive or minimal—for example a placeholder service name or a broad `/` route. Once tail mode is on, the gateway buffers each trace for the [aggregation window](./head-and-tail-sampling#aggregation-window), which increases memory use, export latency, and the risk of dropped spans. If you don't need tail behavior, either remove any unused or test sampling rules from the cluster, or disable tail sampling explicitly in Odigos configuration (Helm values, or the UI **Settings** page). # Examples Source: https://docs.odigos.io/oss/pipeline/sampling/examples End-to-end walkthroughs of common sampling rules in Odigos Open Source, with UI steps and equivalent YAML. End-to-end walkthroughs of common sampling rules. Each example shows the rule in the **UI** and as a **YAML** `Sampling` resource (the same CRD applied via `kubectl apply` or GitOps). Use these as starting points and adjust scope, operations, and percentages to match your environment. YAML examples are accurate against the current `Sampling` CRD schema. The screenshots for each scenario reflect the live UI; minor styling may shift between releases. For the field reference behind each example—every Source scope option, every Operation option, every Keep Percentage choice—see [Configuration](./configuration). For the rules engine itself, see [Rules](./rules/overview). **YAML examples on this page** conform to the `Sampling` CRD (`odigos.io/v1alpha1`). For every field, type, validation rule, and default—including options not covered by the examples below—see the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling). **Scope rules to only the workloads they need to evaluate.** Cluster-wide rules are checked against every trace; tightly scoped rules (Specific Source, Namespace, or Language) are skipped entirely for unrelated workloads, which avoids wasted matching work in the gateway. Several examples below illustrate this pattern. ## 1. Drop a noisy custom endpoint **Goal:** Drop traces for a high-volume polling endpoint (`/api/v1/poll`) that produces many traces but is not useful for debugging or SLOs. This is a typical **Noisy** rule for traffic that's neither a Kubernetes probe nor a metrics scrape, but is still pure noise. Create a Noisy rule with: * **Name:** `Drop polling endpoint` * **Notes:** `High-volume poll endpoint with no SLO/debug value.` * **Source scope:** Specific Source = `odimall / Deployment / notification-service` * **Operation:** HTTP Server, route prefix `/api/v1/poll`, method **GET** * **Drop Percentage:** Drop All

UI screenshot of a Noisy rule that drops GET traces for /api/v1/poll, scoped to the notification-service Deployment in the odimall namespace.

```yaml drop-poll-endpoint.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: drop-poll-endpoint namespace: odigos-system spec: name: Drop /api/v1/poll noisyOperations: - name: Drop polling endpoint notes: High-volume poll endpoint with no SLO/debug value. percentageAtMost: 0 sourceScopes: - workloadNamespace: odimall workloadKind: Deployment workloadName: notification-service operation: httpServer: routePrefix: /api/v1/poll method: GET ``` ## 2. Keep slow requests at 100% **Goal:** Always retain traces for `GET /api/products` whenever they take longer than 500 ms. This is a **Highly Relevant** duration rule—a complement to the built-in [Keep All Error Traces](./rules/highly-relevant#built-in-rule-keep-all-error-traces) rule, focused on latency rather than errors. Create a Highly Relevant rule with: * **Name:** `Slow GET /api/products` * **Notes:** `SLO-relevant path; retain anything over 500ms for investigation.` * **Rule Type:** Duration * **Source scope:** Specific Source = `odimall / Deployment / frontend` * **Operation:** HTTP Server, route `/api/products`, method **GET** * **Keep Percentage:** Keep All * **Duration:** 500ms

UI screenshot of a Highly Relevant rule that keeps 100% of slow GET /api/products traces, scoped to the frontend Deployment in the odimall namespace.

```yaml keep-slow-products.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: keep-slow-products namespace: odigos-system spec: name: Keep slow product reads highlyRelevantOperations: - name: Slow GET /api/products notes: SLO-relevant path; retain anything over 500ms for investigation. durationAtLeastMs: 500 percentageAtLeast: 100 sourceScopes: - workloadNamespace: odimall workloadKind: Deployment workloadName: frontend operation: httpServer: route: /api/products method: GET ``` ## 3. Apply a cluster-wide 5% baseline **Goal:** Keep 5% of all cluster traffic as a statistical baseline, regardless of source or operation. This is the same shape as the built-in [Drop Traces Cluster-Wide](./rules/cost-reduction#built-in-rule-drop-traces-cluster-wide) rule and is the simplest way to cut volume after Noisy and Highly Relevant rules have run. Create a Cost Reduction rule with: * **Name:** `5% baseline` * **Notes:** `Statistical baseline for non-noisy, non-important traffic.` * **Source scope:** Entire cluster * **Operation:** All Operations * **Drop Percentage:** 5

UI screenshot of a Cost Reduction rule that caps cluster-wide retention at 5%.

```yaml baseline-cluster-sample.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: baseline-cluster-sample namespace: odigos-system spec: name: Cluster baseline 5% costReductionRules: - name: 5% baseline notes: Statistical baseline for non-noisy, non-important traffic. percentageAtMost: 5 ``` Cost Reduction rules with **no** Operation block apply to every operation in the scope. Combine with Highly Relevant rules to make sure error and latency traces survive the cap—Highly Relevant always wins over Cost Reduction. ## 4. Scope a rule to namespace and language **Goal:** Drop scrape traffic to `/internal/metrics` for **Java** workloads in the `prod` namespace only—leave other namespaces and other languages untouched. This example demonstrates the **AND** semantics across scope types: a single source scope item with both `workloadNamespace` and `workloadLanguage` set. Create a Noisy rule with: * **Name:** `Java metrics scrape in prod` * **Notes:** `Internal Prometheus scrape endpoint; the scrape itself isn't useful for tracing.` * **Source scope:** Specific Namespace = `prod` **and** Specific Programming Language = **Java** * **Operation:** HTTP Server, route `/internal/metrics`, method **GET** * **Drop Percentage:** Drop All

UI screenshot of a Noisy rule scoped with namespace prod AND language Java.

```yaml drop-metrics-java-prod.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: drop-metrics-java-prod namespace: odigos-system spec: name: Drop /internal/metrics (Java in prod) noisyOperations: - name: Java metrics scrape in prod notes: Internal Prometheus scrape endpoint; the scrape itself isn't useful for tracing. percentageAtMost: 0 sourceScopes: - workloadNamespace: prod workloadLanguage: java operation: httpServer: route: /internal/metrics method: GET ``` Source scope combines in two layers: * **Different categories must all match.** This rule sets Specific Namespace = `prod` together with Specific Programming Language = `java`, so a workload has to be in `prod` **and** be a Java workload. In YAML that maps to a single `sourceScopes` item with both fields set on it. * **Multiple values inside one category are alternatives.** Setting the namespace category to `prod` and `staging` would match a Java workload in either one. In YAML that maps to multiple `sourceScopes` items, each carrying just the namespace field. ## 5. Combine Highly Relevant and Cost Reduction **Goal:** A `payment-service` where **every error trace** must be kept (Highly Relevant) and **all other traffic** is capped at 1% (Cost Reduction). This is the canonical "keep what matters, sample the rest" pattern, expressed in a single `Sampling` object that bundles both categories together. Create two rules: **1. Highly Relevant** * **Name:** `Keep payment-service errors` * **Notes:** `Errors from payment-service are always investigated; never sample them away.` * **Rule Type:** Error * **Source scope:** Specific Source = `odimall / Deployment / payment-service` * **Operation:** All Operations * **Keep Percentage:** Keep All

UI screenshot of a Highly Relevant rule with Rule Type Error scoped to payment-service, Keep at least 100%.

**2. Cost Reduction** * **Name:** `payment-service 1% cap` * **Notes:** `Cap healthy payment-service traffic to keep cost in check; HR errors above always win.` * **Source scope:** Specific Source = `odimall / Deployment / payment-service` * **Operation:** All Operations * **Drop Percentage:** 1%

UI screenshot of a Cost Reduction rule on payment-service with Keep at most 1%.

```yaml payment-service-policy.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: payment-service-policy namespace: odigos-system spec: name: payment-service policy notes: Always keep errors; cap healthy traffic at 1%. highlyRelevantOperations: - name: Keep payment-service errors notes: Errors from payment-service are always investigated; never sample them away. error: true percentageAtLeast: 100 sourceScopes: - workloadNamespace: odimall workloadKind: Deployment workloadName: payment-service costReductionRules: - name: payment-service 1% cap notes: Cap healthy payment-service traffic to keep cost in check; HR errors above always win. percentageAtMost: 1 sourceScopes: - workloadNamespace: odimall workloadKind: Deployment workloadName: payment-service ``` A trace from `payment-service` that contains any error span is kept at 100% by the Highly Relevant rule. A trace with no errors falls through to the Cost Reduction cap of 1%. Both rules are evaluated for every matching trace; Highly Relevant's keep percentage wins when both apply. See [Evaluation between Highly Relevant and Cost Reduction](./rules/overview#evaluation-between-highly-relevant-and-cost-reduction-categories). ## 6. Drop noisy gRPC health checks **Goal:** Drop standard gRPC liveness/readiness checks (`grpc.health.v1.Health.Check`, `Watch`, ...) on the `inventory` Deployment. This is the gRPC counterpart of [Example 1](#1-drop-a-noisy-custom-endpoint), and shows how setting `service` alone scopes a rule to every method of a whole gRPC service. ```yaml drop-grpc-health-checks.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: drop-grpc-health-checks namespace: odigos-system spec: name: Drop gRPC health checks noisyOperations: - name: Drop gRPC health checks notes: Standard liveness/readiness pings; not useful for tracing. percentageAtMost: 0 sourceScopes: - workloadNamespace: odimall workloadKind: Deployment workloadName: inventory operation: grpcServer: service: grpc.health.v1.Health ``` Because `service` and `method` are independent, a single `service:` line filters every method on a service. Add `method: ListItems` to scope to one specific RPC instead, or combine with `grpcClient.serverAddress` to drop outgoing calls to a particular backend. ## YAML reference notes A few details that aren't obvious from YAML alone but matter when authoring rules. The full schema—every field, type, and validation rule—is in the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling). The YAML enum for `workloadLanguage` uses lowercase identifiers: `java`, `python`, `go`, `dotnet`, `javascript`, `php`, `ruby`. The UI displays the friendly names (Java, Python, Go, .Net, NodeJS, PHP, Ruby). `workloadKind` must match the Kubernetes resource kind exactly (case-sensitive PascalCase): `Deployment`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob`. Lowercase or pluralized values will silently fail to match. Empty or missing fields don't mean "match nothing"—each one has a default that often biases toward broader matching: * **`sourceScopes` empty or absent** — the rule matches **every** source. * **`operation` omitted** — the rule matches **every** operation in the scope. * **Noisy `percentageAtMost` omitted** — defaults to **0%** (drops everything). * **Highly Relevant `percentageAtLeast` omitted** — defaults to **100%** (keeps everything). * **Cost Reduction `percentageAtMost`** — **required**. There is no default; the resource will fail validation if the field is absent. * On HTTP Server matchers, set **either** `route` (exact match) **or** `routePrefix` (prefix match)—not both. Both match the framework's **templated path** (e.g., `/users/{id}`, `/users/:id`, `/users/*`), not the resolved URL. * `method` is a free-form string in YAML. The UI restricts it to GET, POST, PATCH, DELETE, and PUT, but YAML accepts any method (e.g., HEAD, OPTIONS). Leave it empty to match any method. * HTTP Client matchers (Noisy only) use `templatedPath` / `templatedPathPrefix`—the framework's parameterized path (e.g., `/users/{id}`), **not** the resolved URL (e.g., `/users/42`). Match against the route as your HTTP library reports it. * `service` matches the fully-qualified gRPC service name (e.g., `acme.inventory.v1.InventoryService`, `grpc.health.v1.Health`); `method` matches the bare method name (e.g., `ListItems`, `Check`). Both fields are **exact match** and **case-sensitive**, mirroring proto naming. * The canonical "drop every call to this service" rule is one line: set `service` and leave `method` empty. To target a single RPC, set both. To match every gRPC call regardless of service, leave both empty. * Both OpenTelemetry semconv conventions are supported transparently: the older split form (`rpc.service` + bare `rpc.method`) and the newer fully-qualified form (`rpc.method` = `"Service/method"`, with `rpc.service` deprecated). For the latter the matcher splits `rpc.method` on the first `/` to derive the service/method, so the same rule matches spans from agents on either version. * The `rpc.*` attribute namespace is shared with other RPC frameworks (Apache Dubbo, Connect RPC, JSON-RPC, .NET WCF, Java RMI, ONC RPC). The matcher reads `rpc.system` / `rpc.system.name` and, when present, only matches spans whose system equals `"grpc"`; non-gRPC RPC spans never accidentally fire a gRPC rule. Spans that omit `rpc.system` entirely are still considered (permissive default for older instrumentations). * A span must carry at least `rpc.service` **or** `rpc.method` to be considered a gRPC span; rules on an attribute that can't be derived from the span will not match (e.g. a `service:` rule against a span whose only RPC attribute is the bare-method `rpc.method` or the `_OTHER` sentinel). * gRPC Client matchers (Noisy only) additionally support `serverAddress` for matching the remote host (the gRPC channel target). Leave it empty to match any address. Leaving `kafkaTopic` empty on a `kafkaConsumer` or `kafkaProducer` matcher matches **every** topic for that direction. Set the topic explicitly to scope a Kafka rule to a specific stream. Setting both `error: true` and `durationAtLeastMs` on a single `highlyRelevantOperations` item is an **AND**—the span must be an error **and** longer than the threshold to match. For "errors **or** slow," use two separate items in the `highlyRelevantOperations` array. The rule `name` is written as a **metric attribute** and stored as a **span attribute** on participating spans, so use a value that's safe for dashboards and queries—avoid PII or characters your backend doesn't index well. `disabled: true` on a rule removes it from sampling decisions but **still emits metrics** for it. That's useful for tuning a rule and observing its would-be impact before re-enabling it, rather than deleting it outright. ## Applying YAML rules You can apply any of the YAML examples above with `kubectl`, or commit them to your GitOps repository alongside the rest of your Odigos configuration: ```bash theme={null} kubectl apply -f drop-poll-endpoint.yaml ``` Each `Sampling` resource is a self-contained group of rules. You can split rules across multiple `Sampling` objects—by team, environment, or use case—and they will be **joined together** to form the global sampling policy. ```bash theme={null} kubectl get samplings.odigos.io -n odigos-system ``` ### Disabling a rule To take a rule out of sampling decisions without deleting it, you can either flip the toggle in the UI or set `disabled: true` in YAML. Disabled rules **still emit sampling metrics**, so you can observe what the rule would have done before re-enabling it—useful as a "soft delete" while tuning, rather than removing the rule outright. See [Enabled](./configuration#enabled) for when to reach for this versus deleting the rule. Each rule's detail view has a toggle in the top-right corner that flips its label between **Rule Enabled** and **Rule Disabled**. Slide it off to take the rule out of sampling decisions immediately; slide it back on to re-enable. The toggle is **per rule**—other rules in the same `Sampling` object continue to apply.

UI screenshot of the Rule Enabled toggle in the on (green) position.

UI screenshot of the Rule Disabled toggle in the off (red) position.

Set `disabled: true` on an individual entry in `noisyOperations`, `highlyRelevantOperations`, or `costReductionRules`. The other rules in the same `Sampling` object continue to apply. This is what the UI toggle writes. ```yaml disable-a-rule.yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Sampling metadata: name: payment-service-policy namespace: odigos-system spec: name: payment-service policy costReductionRules: - name: payment-service 1% cap notes: Cap healthy payment-service traffic to keep cost in check; HR errors above always win. disabled: true percentageAtMost: 1 sourceScopes: - workloadNamespace: odimall workloadKind: Deployment workloadName: payment-service ``` The examples on this page cover the most common shapes; the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling) is the source of truth for every field, type, validation rule, and default on the CRD. Use it when you're extending an example or building a rule that doesn't fit one of the patterns above. # FAQs Source: https://docs.odigos.io/oss/pipeline/sampling/faqs Frequently asked questions about sampling in Odigos Open Source. Frequently asked questions about sampling in Odigos. ## Concepts and terminology Both, depending on who you ask. Some teams describe sampling as **what gets dropped** (e.g. "drop 90%"), others as **what gets kept** (e.g. "keep 10%"). Odigos uses the **keep** framing throughout the UI and docs because it aligns with how rules are written (each rule has a **Keep Percentage**). The two phrasings describe the same outcome; we standardize on "keep" to avoid confusion. See [Introduction to sampling](./introduction#ambiguity-of-terminology). "Sampling" is the name of the **pipeline mechanism** that makes a per-trace keep/drop decision—not a description of "always keep some, always drop some." Dropping all health probes is just sampling at 0%. The same engine that drops 100% of probe traffic also retains 25% of `/api/products` traffic; both are sampling decisions, just at different keep percentages. Odigos applies a single keep/drop decision per trace, not per span. With **head sampling**, the decision is made at the root span and propagated to every child span (parent-based sampling). With **tail sampling**, the Odigos Gateway buffers all spans for a trace ID and makes one decision after the [aggregation window](./head-and-tail-sampling#aggregation-window). In both cases, splitting a trace—keeping some spans and dropping others—would break parent-child links, end-to-end timing, and any analysis that depends on the full request path, so a kept trace is always complete from root through children. See [Sampling level](./introduction#sampling-level). ## How rules are evaluated Categories are evaluated in a fixed sequence: **Noisy → Highly Relevant → Cost Reduction**. A **Noisy** match short-circuits the rest—the trace is dropped (or sampled down) and the other categories are skipped. A **Highly Relevant** match does **not** short-circuit—Cost Reduction rules are still evaluated, but the Highly Relevant Keep Percentage wins. Cost Reduction caps only take effect on traces with no Highly Relevant match. See [Order of operations between categories](./rules/overview#order-of-operations-between-categories). No. Within a single category, rules have **no hierarchy**—they're all equal. When a trace matches multiple rules in the same category, the engine satisfies all matches at once: the **highest** Keep Percentage wins for **Highly Relevant** (floors, "keep at least"), and the **lowest** Keep Percentage wins for **Noisy Operation** and **Cost Reduction** (caps, "keep at most"). This is a deliberate design choice: ordered rule lists (like firewall rules) become unmaintainable when you have dozens of rules and have to mentally simulate the order to predict behavior. Cost Reduction percentages are **caps**, not exact rates, because the engine considers **all** matching rules together. If two Cost Reduction rules match the same trace—one capped at 25% and another at 1%—the lower cap (1%) wins. This makes it safe to add new rules without worrying that they'll silently override existing ones. The same logic runs in reverse for Highly Relevant: percentages there are **floors** ("keep at least"), and the highest floor wins. Almost nothing—and that's usually not what you want. Highly Relevant rules mark important traces, but they don't reduce volume on their own; non-matching traces still flow through the pipeline. Worse, the gateway still buffers traces for tail evaluation, which **adds load without cutting volume**. To control volume, pair Highly Relevant rules with at least one Cost Reduction rule (commonly the built-in Drop Traces Cluster-Wide rule). See [Highly Relevant](./rules/highly-relevant#interaction-with-cost-reduction). Yes. Within a single scope type, multiple selections are combined with **OR** (e.g. namespaces `prod` OR `staging`). Across different scope types, selections are combined with **AND** (e.g. namespace `prod` AND language Java matches **only** Java workloads in `prod`—not every workload in `prod` and not every Java workload across the cluster). See [Source scope](./configuration#source-scope). ## Operational impact It depends on the rule. **Noisy** rules use head sampling and run in the instrumentation agent where supported—they reduce gateway load. **Highly Relevant**, **Cost Reduction** and **Noisy** rules in agents with no head-sampling support, require tail evaluation, which means the gateway buffers each affected trace for the [aggregation window](./head-and-tail-sampling#aggregation-window). Even a single tail-mode rule activates buffering for all traffic, raising memory and CPU on the gateway. Once the gateway accepts a trace for tail evaluation, it allocates memory to hold spans until the aggregation window closes. During traffic bursts, many traces can be accepted concurrently before backpressure kicks in—each one keeps allocating memory until the pod hits its limit and OOMs. This is why bursty workloads can produce dropped spans or collector restarts even when your sampling rules look correct. Sizing the gateway for peak burst capacity (not steady-state) is the most reliable fix. See [Gateway load, OOM, and scaling](./considerations#gateway-load-oom-and-scaling). Both arithmetically yield similar capacity, but **scaling up is usually preferable for tail sampling**. All spans of a single trace must land on the same gateway pod for aggregation, so each scale event re-shuffles the trace-ID hash routing. Fewer, larger pods scale less often and produce less routing churn. There's no one-size-fits-all answer—validate against your own load. They are lost. Odigos does not currently use persistent queues or full backpressure between the data collector and the gateway, so a crash drops everything the gateway was buffering. **Graceful** shutdowns (rolling updates, scale-downs) flush their buffers before exit; only ungraceful crashes lose data. The presence of **any** sampling rule that requires tail evaluation activates tail mode for matching traffic—even rules that look minimal, such as a placeholder service name or a broad `/` route. Once tail mode is on, the gateway buffers traces for the aggregation window, which raises memory and latency to export. If you don't need tail behavior, either remove unused or test sampling rules from the cluster, or disable tail sampling explicitly in Odigos configuration (Helm values, or the UI **Settings** page). See [When any sampler enables tail mode](./considerations#when-any-sampler-enables-tail-mode). ## Tail sampling specifics There is no reliable signal for "this trace is finished," so the engine uses a fixed timeout instead. **30 seconds** is the Odigos default, matching the OpenTelemetry community default. You can change it through the Helm value `sampling.tailSampling.traceAggregationWaitDuration` (for example `'45s'` or `'15s'`); the same value applies to every trace in the cluster. Larger windows hold more memory and delay traces appearing in your backend; smaller windows risk evaluating partial traces and fragmenting a single trace across multiple sampling decisions. See [Aggregation window](./head-and-tail-sampling#aggregation-window). The **first** span the gateway sees for a trace ID, following the OpenTelemetry tail-sampling standard. Practical consequence: requests longer than 30 seconds may be evaluated on a partial trace. Tail evaluation requires every span of a single trace to land on the same gateway pod—otherwise the gateway can't aggregate the trace before deciding. Odigos's data collectors use a **trace-based load balancing exporter** that hashes the trace ID and routes all spans for that trace to the same gateway. Plain round-robin (the default without tail sampling) would scatter spans across pods and break aggregation. ## Built-in rules It drops traces for the **liveness**, **readiness**, and **startup** probe paths discovered on each workload's Kubernetes manifest. The rule's Keep Percentage defaults to **0%** (drop everything) and is configurable. It only matches **HTTPGet** probes—`exec` and `tcpSocket` probes, or probes routed through ingress rewrites or service-mesh wrappers, are not auto-handled and need an explicit Noisy rule. See [Built-in rule: Kubernetes Health Probes](./rules/noisy-operation#built-in-rule-kubernetes-health-probes). The built-in rule itself is fixed at 100% retention and is not configurable. If you need a different keep percentage for error traces—for example because your error volume is too high to retain all of it—author your own Highly Relevant rule on errors with the Keep Percentage you want. Custom rules are how you express any intent other than the built-in 100%. See [Built-in rule: Keep All Error Traces](./rules/highly-relevant#built-in-rule-keep-all-error-traces). Latency thresholds depend on the application: 200 ms is fast for one service and unacceptable for another, so there's no sensible default the product can ship. Errors are different—an error span is universally a signal worth keeping—which is why the error rule is built in but a duration rule is not. Define your own Highly Relevant duration rule with the threshold and Keep Percentage that fit your service. Today's Operation options are **HTTP Server**, **HTTP Client** (Noisy only), and **Kafka Consumer / Kafka Producer** (Highly Relevant and Cost Reduction). Odigos will add support for additional protocols as users needs demand it. # Head and tail sampling Source: https://docs.odigos.io/oss/pipeline/sampling/head-and-tail-sampling How head-based and tail-based trace sampling differ in Odigos and when each approach is appropriate. ## Types of sampling implementation This page contrasts head sampling (decisions in the **agent** when a trace starts) and tail sampling (decisions at the **Odigos Gateway** after spans are brought together). For how Odigos treats whole traces and root-span decisions, see [Introduction to Sampling](./introduction). ### Head sampling (Agent) Head sampling is the most basic and efficient form of sampling. The decision is made at the root span before child spans are processed; the trace in its entirety is either kept or dropped to match that decision. * **Decision point** — At the root span as the trace starts, inside the supported instrumentation agents. The same keep or drop applies to the entire trace (parent-based). * **Context** — Limited to root-level fields: service name, entry endpoint, and similar. Typical rule inputs are things like service, root span details, and specific routes (e.g. `foo/about`). You cannot base the decision on errors or latency that appear only on child spans, or at span end time (http response codes, etc) * **Efficiency** — Spans in a dropped trace are skipped at span creation time, so almost no work is done on them: no agent processing, no serialization into eBPF maps, no handling at the Odiglet or data collection stage, and no network or I/O cost. Volume is cut before any downstream pipeline stage sees it. * **Accuracy** — The choice is early: you do not yet know how the trace will finish, so you cannot guarantee keeping traces based on downstream failures or slowness. Percentage-based cuts behave statistically—rare one-offs may be missed, while frequent behavior still tends to show up in the retained sample over time. * **Best for** — Noise you can recognize at the start (health checks, scrapes, probes, low-value traffic); percentage-based sampling when Odigos Gateway or destination capacity is tight; cost and volume control in environments where root-only policy is enough. ### Tail sampling (Odigos Gateway) Tail sampling defers the keep/drop decision until spans for a trace are buffered and aggregated into a single unit the evaluator can read. The decision uses the full span set, not only the root span, so it can reflect the whole request path before anything is exported downstream. Timing, buffering, and delay are covered in [Aggregation window](#aggregation-window). * **Decision point** — After spans for a trace are buffered and aggregated, in the Odigos Gateway. * **Context** — Full trace: errors, duration, services and endpoints anywhere in the tree, including off the critical path. Rules can target behavior that only shows up on nested spans. * **Efficiency** — CPU and memory requirements on the Odigos Gateway are increased compared with head sampling: buffering and per-trace evaluation need more resources. Most volume is still dropped before destinations, which can ease storage and downstream cost. * **Accuracy** — Full-trace context lets retention match policy on errors, latency, or any span—but evaluation runs only after the [aggregation window](#aggregation-window), so traces show up later in UIs, and very long requests may be judged on a partial trace. * **Best for** — Debugging errors and slow requests end-to-end; full-trace sampling goals; percentage-style or volume cuts after you have full-path context; policies that need services or routes off the entry span. #### Aggregation window The pipeline does not wait for a definitive “end of trace” event, because that would require an indefinite wait. Instead, spans are held up to a bounded window to bound memory and latency. The implementation uses a **default 30-second** interval from the first observed span for a trace ID, consistent with OpenTelemetry-style tail processing, then forwards the trace to the sampler with the spans received by then. Evaluation can therefore run on partial traces for very long requests. The aggregation window is configurable through the Helm value `sampling.tailSampling.traceAggregationWaitDuration` (for example `'45s'` or `'2m'`). The same value applies to every trace in the cluster. Larger windows hold more memory and delay traces appearing in your backend; smaller windows risk evaluating partial traces and fragmenting a single trace across multiple sampling decisions. Observability UIs typically show tail-sampled data delayed by the configured aggregation window (about 30 seconds by default). Export, queueing, storage, and indexing add further delay on top of that gap. ## Head vs. tail: quick comparison | | Head | Tail (Odigos Gateway) | | - | - | - | | **Decision point** | Start of trace (root span) | After aggregation | | **Context** | Root only (service, endpoint, …) | Full trace (errors, latency, any span) | | **Efficiency** | High; low gateway load for the decision | Higher CPU and memory on the gateway | | **Accuracy** | Early decision; percentage cuts are statistical | Full-trace policy; aggregation window tradeoffs | | **Best for** | Noise, cost control, root-level rules | Errors, latency, nested routes, full-path rules | ## Choosing the Right Sampling Strategy: Head vs. Tail Sampling is primarily a tool for cost and volume management. While both head and tail sampling aim to reduce the amount of data sent to your backend, they differ significantly in when the decision is made and what information is used to make it. ### Head-Based Sampling Head-based sampling makes a decision at the very beginning of a trace—usually at the root span. This decision is then propagated to all downstream services (often called "parent-based sampling"), ensuring that either the entire trace is kept or the entire trace is dropped. **When to use it:** * **High-Volume Environments:** If your system generates an overwhelming number of traces, head sampling is the most efficient way to drop data early, saving CPU and memory on your application agents and reducing network traffic immediately. * **Simple Logic:** Use it when you only need to sample based on known "head" attributes, such as the service name, the initial endpoint (e.g., /health vs. /api/orders), or a simple percentage (e.g., "keep 5% of all traffic"). * **Resource Constraints on Collectors:** Since decisions happen at the source, your collectors/gateways don't need to buffer or process data that is already marked to be dropped. **The Trade-off:**
It is a "blind" decision. Because the decision is made at the start, you cannot know if that specific trace will eventually encounter an error or experience high latency. You risk losing the exact "needle in the haystack" traces you might need for debugging. OpenTelemetry formalizes this as **representativeness**: a smaller sample can mathematically represent the larger population, so percentage-based head sampling preserves *aggregate* signal even when individual traces are dropped. See [OpenTelemetry: Why sampling?](https://opentelemetry.io/docs/concepts/sampling/#why-sampling). ### Tail-Based Sampling Tail-based sampling waits until the trace is complete (or a timeout is reached) before deciding whether to keep it. This is handled at the gateway collector level. **When to use it:** * **Ensuring Visibility of Errors:** If your goal is to "keep 100% of traces with errors," tail sampling is essential. You can inspect every span in the trace for an error status before making the final call. * **Latency-Based Sampling:** Use this when you want to capture traces that exceed a specific duration (e.g., "keep any trace longer than 500ms"). * **Complex Business Rules:** When the importance of a trace depends on something that happens deep in the call stack—such as hitting a specific database or a legacy third-party API—tail sampling provides the necessary full context. **The Trade-off:**
It is resource-intensive. The gateway must buffer all spans for a trace (often for 30 seconds or more) to ensure it has the "full picture" before deciding. This increases memory usage at the gateway and introduces a delay before traces appear in your UI. Furthermore, it requires "trace-based load balancing" to ensure all spans for a single trace arrive at the same collector. # Introduction to Sampling Source: https://docs.odigos.io/oss/pipeline/sampling/introduction Start here for trace sampling in Odigos: why sampling is used, how cost and visibility are balanced, and the core terminology behind the product. ## What is sampling **Sampling** is how you choose which traces are **kept** in the pipeline and which are **dropped** before they reach backends and UIs. It trims volume so ingest, storage, and search stay manageable, while policies still aim to leave enough signal for debugging and reliability work. In Odigos, sampling is **per trace**: the entire trace is either retained end to end or not exported, so parent–child structure and timing stay coherent when something is kept. How and when that choice is made—early in the agent at the time of creation (**head** sampling), or at the **Odigos Gateway** after spans are assembled (**tail** sampling)—is covered on [Head and tail sampling](./head-and-tail-sampling). ## Core motivations and definitions ### Primary driver: cost reduction The main reason for sampling is **cost reduction**. Unsampled traces can flood collectors, backends, and analysis tools with volume that drives up ingest fees, storage, and compute. Sampling also helps prevent the observability stack from being overwhelmed by **redundant or low-value (“uninteresting”) data**, so capacity stays available for signals that support debugging and understanding production behavior. ### Reducing noise from uninteresting traces Sampling is also commonly used as a **sanitation layer** to drop traces that add little debugging value. Typical examples include: * **Health-check probes** and readiness/liveness endpoints * **Metrics scraping** requests (for example, `/metrics` endpoints polled by Prometheus) * **Infrastructure traffic** such as service-mesh sidecar chatter, internal control-plane calls, and routine background jobs Removing this noise keeps the observability stack focused on application behavior and prevents redundant data from drowning out meaningful signals. ### Ambiguity of terminology In many systems, “sampling” is described either as **what is dropped** or **what is kept**. Those views are equivalent mathematically, but they express different policy intent—retention versus reduction. Odigos is oriented around **prioritized signal**: defining and keeping **important** traces so that critical evidence remains available for investigation and problem-solving. Phrasings such as “drop 90%” and “keep 10%” describe the same outcome. Odigos emphasizes **prioritization** and **clear intent** rather than opaque drop semantics alone. ## Introduction to sampling ### Why use sampling? Cost versus visibility Visibility is maximized—every trace is available for analysis—but that comes with cost, infrastructure load, and noise. At high volume, the stack and ingest budget become the limiting factors. Not every trace is stored; the aim is still to preserve enough signal to detect errors, latency regressions, and dependency issues. The balance is deliberate: invest ingest and storage in traces that matter instead of exhaustive duplication of routine, successful traffic. ### The Odigos philosophy Traditional framing often starts from dropping data to save money. Odigos shifts the emphasis to **prioritizing what matters**: configuration expresses what should be treated as important (errors, slow paths, specific services or attributes, and so on), and the pipeline retains those traces preferentially. Cost control follows from that prioritization rather than from arbitrary bulk deletion alone. When designing sampling policies, **define “important” first** (failures, SLO risks, critical services), then set fallbacks or rates for everything else. That mirrors how Odigos surfaces configuration and avoids tuning only by a global drop percentage. ### Sampling level Sampling in Odigos is per trace: a single keep-or-drop decision is applied to every span. That decision is made at the root span for head sampling, or at the Odigos Gateway for tail sampling. Partial retention would break parent–child links, timing, and attributes that span the whole request, so a kept trace is always **complete** from the root span through its children. ### Key terminology These terms describe how traces are classified in the sampling model: Traces that match configured rules for high value—for example, failures, high latency, or explicitly selected attributes. The system is designed to **keep** these preferentially so they remain available for investigation. Typical traffic that does not match those priority rules. These may be **sampled at a lower rate** (or dropped according to configuration) while still allowing a statistical view of normal behavior where policies allow it. Low-value traffic such as **health checks, readiness/liveness probes, metrics scraping, and routine infrastructure calls**. These are typically **dropped on sight** as a sanitation step, since they add volume without aiding investigation. Traces that are **not retained** for export to destinations after sampling decisions—whether classified as noisy, deprioritized as regular, or excluded by rate limits. They are not available in the backend for that retention window; this is how volume and cost are reduced. Together, this vocabulary ties configuration to outcomes: policies define what is important, drop what is noisy, treat everything else as regular unless additional rules apply, and dropped traces represent the tradeoff for sustainable cost and signal-to-noise ratio. # Cost Reduction Source: https://docs.odigos.io/oss/pipeline/sampling/rules/cost-reduction Cap how much of the remaining traffic is exported to control trace volume and ingest cost. **Cost Reduction** rules cap how much traffic is retained for export, after Noisy and Highly Relevant rules have run. They are the **third and final** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories) and are how you control **volume and ingest cost** for traffic that is neither obviously noisy nor specifically marked as important. ## How they're evaluated * **Applied to traces that fall through earlier categories.** Traces that matched a Noisy drop never reach this stage; traces that matched a Highly Relevant rule are governed by that rule instead (see [Interaction with Highly Relevant](#interaction-with-highly-relevant)). * **Percentage-based.** Each rule has a **Keep Percentage** that defines the fraction of matching traffic to retain. * **Scoped.** A Cost Reduction rule can apply to the whole cluster, one or more namespaces, services, or programming languages, and can target all operations or a specific HTTP route or Kafka topic. See [Configuration](../configuration) for the full authoring options. ## Built-in rule: Drop Traces Cluster-Wide Odigos ships a single built-in Cost Reduction rule, **Drop Traces Cluster-Wide**, which applies one **Keep Percentage** uniformly across every trace in the cluster. You configure the percentage; the engine then makes an independent, probabilistic keep/drop decision per trace. At high trace volume, the **law of large numbers** applies and sample proportions—error rates, latency percentiles, per-service and per-route traffic shares—converge on the underlying population. The retained traces preserve the **shape** of your traffic, so common scenarios remain visible at their original relative frequencies. Uniform sampling preserves **statistics**, not individual events. At low keep percentages, or for rare scenarios such as low-frequency error paths or occasional regressions, specific occurrences can be missed entirely. To **guarantee** retention for traces you care about, use a [Highly Relevant](./highly-relevant) rule—which always wins over Cost Reduction—rather than relying on this rule alone. ## Interaction with Highly Relevant Highly Relevant **always wins** over Cost Reduction on traces that match both, regardless of which keep percentage is higher. That means a Cost Reduction cap of 10% on `/api/orders` does **not** suppress a Highly Relevant rule that keeps 100% of error traces on the same route. See [Evaluation between Highly Relevant and Cost Reduction](./overview#evaluation-between-highly-relevant-and-cost-reduction-categories) for worked examples. ## Typical use cases * Keep 5% of all cluster traffic as a baseline statistical sample * Keep 50% of `/api/products` traffic to control a high-volume endpoint * Keep 1% of traces from a chatty internal service # Highly Relevant Source: https://docs.odigos.io/oss/pipeline/sampling/rules/highly-relevant Mark and retain important traces using RED-style signals like errors and high duration, with a per-rule keep percentage. **Highly Relevant** rules mark traces you want to treat as **important**, aligned with **RED**-style signals (request, **error**, **duration**). They are the **second** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories), evaluated only on traces that did **not** match any Noisy rule. ## How they're evaluated * **Tail-aware.** Importance often depends on something that only appears mid-trace—an error on a child span, or a long downstream call—so the evaluator typically needs the assembled trace. See [Head vs. tail sampling](../head-and-tail-sampling). * **One error span marks the entire trace.** When a Highly Relevant error rule applies, a single error span anywhere in the tree is enough to classify the whole trace as important. * **100% retention by default.** Highly Relevant rules keep every matching trace by default—as the built-in error rule does—which works for most workloads. If a specific operation produces too much volume, lower its Keep Percentage on a custom rule to retain only a fractional sample, while still benefiting from Highly Relevant priority. ## Built-in rule: Keep All Error Traces Odigos ships a single built-in Highly Relevant rule, **Keep All Error Traces**, which retains **100%** of traces that contain at least one span with **error** status. The built-in rule's keep percentage is fixed at **100%** and is **not configurable**. Because the evaluator inspects every span in the trace, a single error anywhere in the call tree is enough to keep the whole trace—even when the root span itself succeeds. If 100% retention of error traces is more than you need—for example because your error volume is high—author your own Highly Relevant rule on errors with the **Keep Percentage** you want. Custom rules are how you express any intent other than the built-in 100%. ## Interaction with Cost Reduction When a trace matches rules in **both** Highly Relevant and Cost Reduction, the Highly Relevant Keep Percentage **always wins**, whether higher or lower than the Cost Reduction cap. See [Evaluation between Highly Relevant and Cost Reduction](./overview#evaluation-between-highly-relevant-and-cost-reduction-categories) for worked examples. Highly Relevant rules without any Cost Reduction rules **do not aggressively reduce volume** on their own. Unimportant traffic still flows through the gateway and may be evaluated as part of full-trace policy. Pair Highly Relevant with Cost Reduction when you need both signal and volume control. ## Typical use cases * Keep all (or most) traces with at least one error span * Keep a higher percentage of traces over a duration threshold (for example > 500 ms) * Always keep a sample from a critical service or route # Noisy Operation Source: https://docs.odigos.io/oss/pipeline/sampling/rules/noisy-operation Drop or down-sample low-value traffic — health probes, scrapes, heartbeats — early at the root span with head sampling. **Noisy Operation** rules drop or down-sample traffic that you treat as having little or no value—for example Kubernetes liveness and readiness probes, periodic health checks, or scraper traffic. They are the **first** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories): a trace that matches a Noisy rule is **not** evaluated against Highly Relevant or Cost Reduction rules. ## How they're evaluated * **Head sampling at the root span.** The keep/drop decision is made on the entry span, before child spans are processed. * **No trace buffering.** Because the decision happens at the start of the trace, the gateway does not need to wait for an [aggregation window](../head-and-tail-sampling#aggregation-window). * **Lowest overhead.** Head sampling is the **most efficient** sampling type, with the **least CPU and memory** impact on collectors and gateway. Head-sampled Noisy rules require an instrumentation agent that supports head sampling. Today this includes **Node.js**, **Python**, **Java**, and **Go**. For other languages or runtimes, Noisy rules are **evaluated at tail-sampling time** instead—the trace is buffered for the full [aggregation window](../head-and-tail-sampling#aggregation-window) before the drop happens, so the efficiency advantages above (no buffering, lowest overhead) **do not apply**. ## Possible outcomes A matching trace is either: * **Dropped** — never exported and never evaluated against Highly Relevant or Cost Reduction rules. * **Sampled down** — kept at a configured percentage. The kept fraction also skips later categories. Dropped traces do not continue to other categories. If you need a noisy route to remain visible when something goes wrong (for example a failing health check), replace the Noisy rule with a **Highly Relevant** rule on errors and use a **Cost Reduction** rule to cap the successful traffic. ## Built-in rule: Kubernetes Health Probes Odigos ships a single built-in Noisy rule, **Kubernetes Health Probes**, which drops traces for the **liveness**, **readiness**, and **startup** probe paths discovered on each workload's Kubernetes manifest. The rule's **Keep Percentage** controls how much probe traffic is retained. Common configurations: * **0% — drop everything (default).** Probes carry no signal about application behavior, so dropping them keeps the pipeline focused on real traffic and minimizes cost. * **\~2% — sparse liveness signal.** For services with little or no real traffic, a small permanent value keeps a handful of probe traces flowing. The retained samples act as a "service is up" indicator on dashboards that would otherwise look empty, helping distinguish a *quiet* service from a *silently broken* one. * **100% — debug probe failures.** Temporarily raise while investigating failing or flaky probes, then reset to your normal value once resolved. Only **HTTPGet** probes declared on the manifest are matched. Probes defined as `exec` or `tcpSocket`, or probes routed through different paths (custom configuration, ingress rewrites, service-mesh wrappers), are not auto-handled; add an explicit Noisy rule for those paths if needed. ## Typical use cases * Kubernetes `liveness`, `readiness`, and `startup` probes * Internal `/health` and `/healthz` endpoints * Prometheus `/metrics` scrape traffic * Background heartbeats and keep-alive pings * High-volume, low-value endpoints — for example a static asset route, a polling endpoint, or an autocomplete API that produces many traces without contributing to debugging or SLOs # Overview Source: https://docs.odigos.io/oss/pipeline/sampling/rules/overview How Odigos evaluates sampling rules: categories, order between Noisy, Highly Relevant, and Cost Reduction, conflicts inside a category, and head vs. tail sampling. This page describes how **sampling rules** are evaluated in Odigos: the three **rule categories** (Noisy Operation, Highly Relevant, Cost Reduction), the **order between categories**, and how **conflicts inside a category** resolve. Each category has its own page; this overview focuses on how the categories interact. ## Rule categories Traffic with little or no value (for example Kubernetes health probes). Decided early at the **root span** with **head sampling**. **Important** traffic aligned with **RED**-style signals (errors, duration). Each rule's **Keep Percentage** controls how much is retained. Caps applied to remaining traffic to control volume and cost (for example a percentage across the cluster or a specific route). See [Head vs. tail sampling](../head-and-tail-sampling) for how head and tail sampling differ in Odigos (root-span decisions versus aggregation at the gateway). If you only define **Highly Relevant** rules and **no Cost Reduction** rules, unimportant traffic is not aggressively reduced by those rules alone—policy may still require full-trace context to evaluate importance, which increases gateway work without cutting volume. Use **Highly Relevant** and **Cost Reduction** together when you need both signal and volume control. ## Order of operations between categories Categories are **not** all evaluated for every trace in parallel. Odigos applies them in this fixed **sequence**:

↓

▶

↓

▶

For traces that match rules in this category after the earlier stages, this stage applies caps to control volume and cost.

The pipeline runs the categories in this order. At each stage, every rule in that category is considered; how multiple matches resolve is covered in [Rule evaluation](#rule-evaluation). ## Rule evaluation ### Evaluation inside a category Within a single category, rule order does not matter. When multiple rules match the same trace, the engine combines them so all of them are satisfied at once. For Highly Relevant, percentages are floors — the highest keep percentage wins, so any retention guarantee is honored. For Noisy Operation and Cost Reduction, percentages are caps — the lowest keep percentage wins, so the strictest cap is honored. This makes it safe to add rules without fear of one silently overriding another. ### Evaluation between Highly Relevant and Cost Reduction categories When a trace matches rules in **both** the **Highly Relevant** and **Cost Reduction** categories, the **Highly Relevant** keep percentage **always wins** — regardless of whether it is higher or lower than the **Cost Reduction** cap. **Trace:** `POST /api/orders` that returned a `500` error. **Matching rules:** * **Highly Relevant** — keep **100%** of error traces. * **Cost Reduction** — keep at most **10%** of `/api/orders` traffic. **Outcome:** the trace is **kept at 100%**. Highly Relevant overrides the Cost Reduction cap. **Trace:** `GET /api/products` that took **750 ms**. **Matching rules:** * **Highly Relevant** — keep **25%** of traces longer than **500 ms**. * **Cost Reduction** — keep at most **50%** of `/api/products` traffic. **Outcome:** the trace is **kept at 25%**. Highly Relevant still wins, even though its keep rate is lower than the Cost Reduction cap. ## Why full traces still matter For traces that are evaluated on the **tail** path, the evaluator needs the assembled trace (within the aggregation window) so it can apply **Highly Relevant** conditions that depend on any span. That is why tail sampling and aggregation are part of the pipeline for those policies, even when the category order is Noisy → Highly Relevant → Cost Reduction. # Service Graph Source: https://docs.odigos.io/oss/pipeline/service-graph Visualize how your services connect to each other and to uninstrumented dependencies like Redis or Kafka. ## Overview The Service Graph is a live topology map that shows which services call each other and how those calls are performing. It is built entirely from your distributed traces — no additional instrumentation or agents are required. Odigos generates the service graph automatically when traces are enabled and at least one tracing destination is configured. ## How It Works Odigos uses the OpenTelemetry [Service Graph Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector) inside the Cluster Gateway Collector. The pipeline looks like this: ``` Instrumented services │ (OTLP traces) ▼ Node Collector (DaemonSet) │ (OTLP traces) ▼ Cluster Gateway Collector ├──► servicegraph connector ──► Odigos UI (topology map) ``` 1. The **servicegraph connector** receives every trace that passes through the gateway. It matches CLIENT spans with their corresponding SERVER spans to detect service-to-service calls. 2. For each matched pair it emits a metric labelled with the `client` and `server` service names plus any configured extra dimensions, which the Odigos UI uses to render the interactive topology map. ### Virtual Nodes (Uninstrumented Dependencies) When a service calls an uninstrumented dependency (Redis, Kafka, a third-party API, …) there is no SERVER span. The connector handles this with **virtual nodes**: it reads certain attributes from the CLIENT span to infer the name of the remote endpoint and creates a synthetic node in the graph. The attributes checked — in priority order — are configured via `serviceGraphVirtualNodePeerAttributes`. The connector picks the **first attribute that is present** on the CLIENT span. Only **span-level attributes** are read when resolving virtual node identities (e.g. `server.address`, `db.system`, `peer.service`). ## Enabling and Disabling The service graph is **enabled by default** when traces are active. Disabling it removes the `servicegraph` connector from the gateway pipeline and stops emitting the related metrics. Disabling the service graph also removes the load balancing behavior between the node collectors and the gateway, because the gateway no longer needs to aggregate spans from the same trace. ```yaml theme={null} # values.yaml collectorGateway: serviceGraphDisabled: true # set to true to disable ``` ## Configuration Options All options live under `collectorGateway` in your `values.yaml` (Helm) or by manually editing the `odigos-configuration` ConfigMap in your cluster. ### Extra Dimensions By default the graph distinguishes services by `service.name` only. `serviceGraphExtraDimensions` adds more labels to the edge metrics, creating a separate node for each unique combination of values. **When to use it:** multiple deployments share the same `service.name` (e.g. the same service deployed in `staging` and `production` namespaces). ```yaml theme={null} collectorGateway: serviceGraphExtraDimensions: - k8s.namespace.name # separate nodes per namespace ``` Both span-level and resource-level attributes are supported for extra dimensions — the connector reads from the full span resource context. Avoid high-cardinality attributes such as `k8s.pod.name`; each unique value creates a separate time series in Prometheus. ### Virtual Node Peer Attributes `serviceGraphVirtualNodePeerAttributes` controls which span attribute is used to name uninstrumented dependencies. The connector evaluates the list in order and uses the first attribute that has a non-empty value. **When to use it:** multiple instances of the same technology (e.g. a Redis Cluster with several shards, or two separate Redis caches) are collapsing into a single node because they all resolve to the same value (e.g. `db.system=redis`). ```yaml theme={null} collectorGateway: serviceGraphVirtualNodePeerAttributes: - peer.service # explicit logical name — highest priority - server.address # hostname of the specific instance (e.g. redis-cache.default.svc.cluster.local) - net.peer.name # older convention, wide SDK support - db.name # database index (rarely useful for Redis — most apps use index 0) - db.system # technology name (e.g. "redis") — last resort, not instance-specific ``` ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Add Sources Source: https://docs.odigos.io/oss/pipeline/sources/create Add a Source to enable telemetry collection for specific Kubernetes workloads, or a whole namespace. ## With Odigos UI The easiest way to add a source to Odigos is to use the Odigos UI.
After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI: ```shell theme={null} odigos ui ``` And then, in the setup page or the sources page, select the workloads / namespaces for which you want telemetry collection to be enabled. To include all the workloads in a namespace as sources, and automatically add future-new workloads, enable the `Future Apps` toggle in the UI. Otherwise, you can enable the `Select All` toggle to include all the workloads in the namespace as sources, excluding future-new workloads. ## With Odigos CLI See the [CLI documentation on Sources](../../cli/odigos_sources) for detailed command documentation for manipulating Sources. ## Kubernetes Manifests If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can add a source to Odigos by creating a Source object. Only one Source object may exist for a given workload, except in namespace instrumentation. ### Workload Source The following Source object enables instrumentation for a Deployment named `frontend` in the `default` namespace and belong to Datastream `example`. > **Note:** Sources that do not have a Datastream label will be automatically attached to the default Datastream. ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default labels: odigos.io/data-stream-example: "true" spec: workload: name: frontend namespace: default kind: Deployment ``` ### Namespace Source The following Source object enables instrumentation for all the workloads in the `default` namespace: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: namespace-source namespace: default spec: workload: name: default namespace: default kind: Namespace ``` Namespace Sources must have the following fields:
* `spec.workload.name == spec.workload.namespace` * `spec.workload.kind: Namespace` ### Disable Workload Source in Namespace You can have a namespace Source and a workload Source that both enable instrumentation for a workload. In this case, if one is deleted, the other will persist instrumentation for the workload. To exclude specific workloads from namespace instrumentation, set `disableInstrumentation: true` on the workload Source. For example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: namespace-source namespace: default spec: workload: name: default namespace: default kind: Namespace --- apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default spec: disableInstrumentation: true workload: name: frontend namespace: default kind: Deployment ``` In this example, all workloads in the `default` namespace will be instrumented except for the `frontend` Deployment. If `example-source` is deleted (or `disableInstrumentation` is set to `false`), the `frontend` Deployment will be instrumented. # Remove Sources Source: https://docs.odigos.io/oss/pipeline/sources/delete Remove a Source to disable telemetry collection for specific Kubernetes workloads, or a whole namespace. Sources are persistent and decoupled from the workloads they instrument.
If a workload is deleted without deleting the Source, the Source will resume its effects if the workload is re-created later. ## With Odigos CLI The Odigos CLI supports instrumenting and uninstrumenting workloads with the [`odigos sources enable`](../../cli/odigos_sources_enable) and [`odigos sources disable`](../../cli/odigos_sources_disable) commands. Each of these commands accepts a source type, name, and namespace. For example, to remove a source from odigos, run: ```shell theme={null} odigos sources disable deployment -n ``` This command will rollout existing pods to restart without any odigos additions. It can be undone (re-instrumented) by running: " ```shell theme={null} odigos sources enable deployment -n ``` ## With Odigos UI The easiest way to remove a source from Odigos is to use the Odigos UI.
After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI: ```shell theme={null} odigos ui ``` And then, in the sources page, unselect the workloads / namespace for which you want telemetry collection to be disabled. ## Kubernetes Manifests If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can remove a source from Odigos by deleting the Source object, for example: Run: ```shell theme={null} kubectl get sources -n default ``` Output: ```shell theme={null} NAME WORKLOAD KIND NAMESPACE source-7sxsb frontend Deployment default source-b42rn inventory Deployment default source-b7zn9 coupon Deployment default source-v6sxs membership Deployment default source-v9jjw pricing Deployment default ``` Run: ```shell theme={null} kubectl delete source source-7sxsb -n default ``` Output: ```shell theme={null} source.odigos.io "source-7sxsb" deleted ``` # Introduction Source: https://docs.odigos.io/oss/pipeline/sources/introduction Sources are a Custom Resource that define which Kubernetes workloads should be auto-instrumented by Odigos. ## How Sources work Source custom resource objects are created as a reference to a workload or namespace ("entities"). Odigos reacts to changes in these custom resource objects to configure instrumentation for the referenced entity. On this page, "Sources" refers to the custom resource object, while "workload", "namespace", or "entity" refers to the application being instrumented. See the [API Reference](../../api-reference/odigos.io.v1alpha1#source) for the full structure of `Source` objects and their fields. ## Supported Entities Odigos supports the following Kubernetes entities: * Kubernetes **Deployment** * Kubernetes **DaemonSet** * Kubernetes **StatefulSet** * Kubernetes **Namespace** ## Types of Sources A Source object can be either: * A **Workload Source**, which controls instrumentation for a single workload (Deployment, DaemonSet, or StatefulSet) * A **Namespace Source**, which controls instrumentation for every workload in a namespace. ### Workload Sources Workload Sources specify a single workload, and the Source represents the instrumentation configuration for that workload. Use Workload Sources to control instrumentation and configuration for individual Deployments, StatefulSets, or DaemonSets with unique requirements. Example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-workload namespace: default spec: workload: kind: DaemonSet name: sample-app namespace: default ``` ### Namespace Sources Namespace Sources represent instrumentation settings for an entire namespace. Namespace Sources are created by setting `spec.workload.kind: Namespace` along with `spec.workload.name` and `spec.workload.namespace` equal to the name of the namespace to control. Use Namespace Sources to control instrumentation for multiple Deployments, StatefulSets, and DaemonSets from a single spot. Example: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-namespace namespace: default spec: workload: kind: Namespace name: default namespace: default ``` ## Using multiple Sources Workload Sources are intended to be combined with Namespace Sources for flexible, fine-grained control. For example, you may choose to create a Namespace Source to enable instrumentation for namespace `foo`, but want to exclude deployment `bar`. In this case, you can create a Workload Source for deployment `bar` with `disableInstrumentation: true`: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: my-namespace namespace: foo spec: workload: kind: Namespace name: foo namespace: foo --- apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: exclude-bar namespace: foo spec: disableInstrumentation: true workload: kind: Deployment name: bar namespace: foo ``` The opposite is also possible: you could disable instrumentation for namespace `foo` with a Namespace Source but choose to explicitly enable deployment `bar` via a Workload Source that sets `disableInstrumentation: false` (the default). This is because Workload Sources take priority over Namespace Sources, and any settings in a Workload Source will override those in a parent Namespace Source. ## Source Workflows Sources are intended to control and configure instrumentation as objects that are completely decoupled from their target workloads. When working with Sources, they should be treated as such. The only requirements for a Source to identify a workload are: * The Source must set `spec.workload.name`, `spec.workload.namespace`, and `spec.workload.kind` to match the workload (or namespace). * The Source must exist in the same namespace as the target workload (or namespace). Besides these requirements, you are encouraged to manage your Sources as their own objects. For example, when creating Sources from YAML manifests, consider storing them together in their own file such as `sources.yaml`, separate from your workload manifests. This separation of concerns allows you to declaratively control your application's observability without tying it to your application manifests. You should also define RBAC policies that restrict access to only what is needed to work with Source objects. In other words, your observability team may not require the ability to view deployments in a namespace to manage the instrumentation for those deployments. For example, the following Role grants full editor permissions for Sources: ```yaml theme={null} apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: default name: source-editor rules: - apiGroups: ["odigos.io"] resources: ["sources"] verbs: ["*"] ``` ## Source Labels Odigos uses mutating webhooks to automatically copy the values of `spec.workload` to matching labels. It is encouraged to use these labels to manage Sources in a cluster: * `odigos.io/workload-name` * `odigos.io/workload-namespace` * `odigos.io/workload-kind` The value of these labels will be validated to always match the corresponding fields in `spec.workload`. They can be used to group and list Sources, for example: ```shell theme={null} kubectl get sources -l odigos.io/workload-kind=Deployment ``` This will list all `Sources` for `Deployments`. ## Automatically created Sources In some scenarios, Odigos may create Sources for you automatically. * When instrumenting workloads in the UI, Odigos will create a Workload Source for each workload. * When choosing **Include future workloads** in the UI, Odigos will create a Namespace Source for the namespace. * When creating a Source grouping in the UI, Odigos will create a Workload Source for each Workload in the group. In all of these cases, if a Source already exists, Odigos will either patch the existing Source or return an error if it can't. # Troubleshooting Source: https://docs.odigos.io/oss/pipeline/sources/troubleshooting If you selected a workload to auto-instrument and you don't see any data, use the checklist below to troubleshoot the issue. It is recommended to use the [`odigos describe`](../../cli/odigos_describe) command to get detailed information in one command. ## Check is Source Created? A `Source` object is required for Odigos to instrument the workload or namespace, such as: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example namespace: foo spec: workload: name: myapp namespace: foo kind: StatefulSet ``` If you expect to see data from a workload but don't, check if the `Source` object exists: ```shell get all in namespace theme={null} kubectl get sources -n ``` ```shell get by deployment name theme={null} kubectl get sources -n -l odigos.io/workload-name= ``` ```shell get by deployment kind theme={null} kubectl get sources -n -l odigos.io/workload-kind= ``` ## Check is Language Detected? After a `Source` object is created, Odigos will attempt to detect the programming language of the containers in the workload pods, and write the detected language to an `InstrumentationConfig` object in the same namespace. The resulting language is used to determine which instrumentation to apply. If a `Source` object exists, verify that the `InstrumentationConfig` also exists: ```shell get all in namespace theme={null} kubectl get instrumentationconfigs -n ``` ```shell get by crd name theme={null} kubectl get instrumentationconfigs -n

-o yaml ```

It should look something like this: ```yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: example namespace: foo spec: template: spec: containers: - resources: limits: instrumentation.odigos.io/go-ebpf-community: "1" ``` The `instrumentation.odigos.io/go-ebpf-community` resource-limit instructs Odigos on how to mount the relevant OpenTelemetry agents and environment variables into the container, and which processes should be instrumented with eBPF. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Configuration Source: https://docs.odigos.io/oss/pipeline/sources/update Beyond just enabling or disabling instrumentation, Source objects can be used to configure instrumentation settings as well. All Source settings can be configured with the Odigos CLI (see [CLI documentation on updating Sources](../../cli/odigos_sources_update)) or Kubernetes-native tools like `kubectl`. ## service.name The `service.name` attribute is a fundamental resource attribute in [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/resource/#service).
It is required, and is reported in all telemetry data.
It is used to identify the service that emitted a span, metric or log record. Odigos will automatically populate this attribute based on the name of the workload. If there are multiple containers in the workload, the `service.name` will be set to the container name. If you want to override the default value, you can also set the `service.name` attribute by adding the `otelServiceName` attribute to an exiting Source, or by creating a `Source` with that attribute: ```yaml theme={null} apiVersion: odigos.io/v1alpha1 kind: Source metadata: name: example-source namespace: default spec: workload: name: frontend namespace: default kind: Deployment otelServiceName: my-custom-service-name ``` Namespace Sources do not support the `otelServiceName` attribute. If a namespace is instrumented, and a Source for a specific workload is not present, a specific source for that workload will need to be created including the desired `otelServiceName`. ### From Odigos UI The easiest way to update a source from Odigos is to use the Odigos UI.
After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI: ```shell theme={null} odigos ui ``` And then, in the sources page, select the source you want to edit and fill in the value for the `service.name` field. To undo the override, simply erase the value from the `service.name` field. ### Odigos CLI To set the `service.name` using the Odigos CLI, use the `odigos sources update` command ([see documentation](../../cli/odigos_sources_update)). If you know the name of the Source object to update, pass that to `odigos sources update` along with the `--otel-service` flag: ```shell theme={null} odigos sources update --otel-service=my-service-name -n ``` If you don't know the name of the Source object to update, you can provide flags to `odigos sources update` to identify the Source by workload: ```shell theme={null} odigos sources update --workload-kind= \ --workload-name= \ --workload-namespace= \ --otel-service=my-service-name ``` ### Kubernetes Manifest If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can can set the `service.name` attribute by updating the corresponding Source. For example, to set the `service.name` attribute in telemetry for an existing source: To get the relevant Source for your workload you can either use: ```shell theme={null} kubectl get sources -n \ -l odigos.io/workload-kind= \ -l odigos.io/workload-name= \ -l odigos.io/workload-namespace= ``` or list the sources in the relevant namespace: ```shell theme={null} kubectl get sources -n ``` After we have the desired Source we can patch it: ```shell theme={null} kubectl patch source/ -n --type=merge -p '{"spec":{"otelServiceName":""}}' ``` If you don't know the name of a Source object (such as when a Source is automatically created by instrumentation done via the UI), you can combine the use of Source labels with the `kubectl patch` command. For example: ```shell theme={null} kubectl patch source \ $(kubectl get source -l 'odigos.io/workload-name=,odigos.io/workload-kind=' --no-headers -o custom-columns=":metadata.name") \ --type=merge \ -p '{"spec":{"otelServiceName":""}}' ``` # Building a pipeline Source: https://docs.odigos.io/oss/quickstart/building-a-pipeline ## Choosing target applications You should now see the following page: Select target applications

# Generating Data Source: https://docs.odigos.io/oss/quickstart/generation-data ## Wait for collectors to be ready Before generating data, make sure that the collectors and applications are ready. To check this run the following command: ```bash theme={null} kubectl get pods -n default ``` ```bash theme={null} kubectl get pods -n odigos-system ```

# Installation Source: https://docs.odigos.io/oss/quickstart/installation ## Installing Odigos CLI Odigos CLI is the easiest way to install Odigos on your Kubernetes cluster. It is a single binary that can be installed on any Linux, MacOS or Windows machine. If you are on MacOS, you can install Odigos CLI using Homebrew: ```bash theme={null} brew install odigos-io/odigos-cli/odigos ``` 1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux). 2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system. 3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH". * For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt. * For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`. 4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH. 1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows). 2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files. 3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how: * **Option 1**: Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder. * **Option 2**: Add the folder containing the extracted binary to your PATH manually: 1. Right-click the **Start** button and select **System**. 2. Click **Advanced system settings** on the left. 3. In the **System Properties** window, click **Environment Variables** at the bottom. 4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**. 5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file. 6. Click **OK** to close all windows. 4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located. ## Installing Odigos on Kubernetes To install Odigos on your Kubernetes cluster, run the following command: ```bash theme={null} odigos install ``` ## Accessing Odigos UI After Odigos is installed, open the Odigos UI by running the following command: ```bash theme={null} odigos ui ``` By default, Odigos UI is available at [http://localhost:3000](http://localhost:3000). # Introduction Source: https://docs.odigos.io/oss/quickstart/introduction In this tutorial we are going to use Odigos for generating distributed traces for a microservices-based application written in Go, Java, Python, .NET and Node.js. We are going to deploy the application in a Kubernetes cluster and use [Jaeger](https://www.jaegertracing.io/) as the backend for storing and visualizing the traces. This guide is designed for: * IT professionals with experience in system administration or DevOps * Software engineers familiar with containerization and Kubernetes concepts * Anyone comfortable with command-line interfaces and basic cloud infrastructure If you're new to these concepts or prefer a more detailed, step-by-step approach, we recommend starting with our [Beginner's Guide to Odigos Installation](../quickstart/installation-guide-for-beginners). ## Preparing the environment ### Creating a Kubernetes cluster This first step is to create a Kubernetes cluster. We recommend using [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/docs/) for trying Odigos out in a local environment. **Mac users**: please avoid using Docker Desktop built-in Kubernetes cluster as it [does not](https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation) support bind propagation. Create a new local Kubernetes cluster, by running the following command: ```bash theme={null} kind create cluster ``` ```bash theme={null} minikube start ``` ### Deploying the target application We are going to install a [demo application](https://github.com/odigos-io/simple-demo) that consists of 6 microservices written in Java, Go, Python, Node.js, .NET and PHP. Architecture of demo application

# Next Steps Source: https://docs.odigos.io/oss/quickstart/next-steps You just finished the Odigos quickstart guide! 🎉 ## Not just distributed tracing **Odigos supports metrics and logs too!** You can send metrics and logs from your application to Odigos, and then visualize them in your favorite backend. In order to send metrics and logs, just select a destination that supports them. Odigos supports all the major observability backends, both open-source and commercial. Select the relevant backend for your use case below to connect it to Odigos. # Viewing traces Source: https://docs.odigos.io/oss/quickstart/viewing-traces Now that we have some traces, let's view them! Access the Jaeger UI by running the following command: ```bash theme={null} kubectl port-forward -n tracing svc/jaeger 16686:16686 ``` Then, open [http://localhost:16686](http://localhost:16686) in your browser and explore the traces.

# Install Odigos Images from a Custom Docker Registry Source: https://docs.odigos.io/oss/setup/docker-registry ## Tagging and Pushing Images ### Step 1: Set Environment Variables First, export the `VERSION` and `CUSTOM-DOCKER-REGISTRY` environment variables in your shell: ```bash theme={null} export VERSION= export CUSTOM_DOCKER_REGISTRY= export PLATFORM= ``` Replace `` with the version of the Odigos images you want to use, `` with the URL of your custom Docker registry, and `` with the platform architecture of the environment where you are deploying Odigos (linux/amd64, linux/arm64). *** ### Step 2: Pull the Images Start by pulling the required images from the default community registry (`docker.io/keyval`) onto your local machine or CI environment: ```bash theme={null} docker pull docker.io/keyval/odigos-instrumentor:$VERSION docker pull docker.io/keyval/odigos-ui:$VERSION docker pull docker.io/keyval/odigos-odiglet:$VERSION docker pull docker.io/keyval/odigos-collector:$VERSION docker pull docker.io/keyval/odigos-victoria-metrics:$VERSION docker pull docker.io/keyval/odigos-cache:$VERSION ``` If you are using the [k8s-init-container](../instrumentations/configuration/mount-method#3-InitContainer) mount method make sure the odigos-agents image is also pulled, as it's required by the injected InitContainer. On Kubernetes 1.35+, that init container also needs the registry pull secret in each instrumented namespace. See [Init container image pull secrets](../instrumentations/configuration/mount-method#init-container-image-pull-secrets). **This is not the default configuration**, if you're unsure what this means, you can safely ignore this step and continue with the standard setup. ```bash theme={null} docker pull docker.io/keyval/odigos-agents:$VERSION ``` If you are using [image mirrored Go offsets updates](../instrumentations/golang/ebpf#about-go-offsets), also mirror the offsets image: ``` docker pull docker.io/keyval/cli-offsets:latest ``` Keeping this image regularly up-to-date is necessary to pull the latest offsets into your cluster. Odigos component images are published in multi-arch for ARM64 and AMD64 architectures. When pulling these images, be sure to specify the correct architecture *for the environment they will be deployed*. ```bash theme={null} docker pull --platform $PLATFORM docker.io/keyval/odigos-instrumentor:$VERSION docker pull --platform $PLATFORM docker.io/keyval/odigos-ui:$VERSION docker pull --platform $PLATFORM docker.io/keyval/odigos-odiglet:$VERSION docker pull --platform $PLATFORM docker.io/keyval/odigos-collector:$VERSION docker pull --platform $PLATFORM docker.io/keyval/odigos-victoria-metrics:$VERSION docker pull --platform $PLATFORM docker.io/keyval/odigos-cache:$VERSION ``` ### Step 3: Tag the Images Next, Tag each image with your custom Docker registry prefix: ```bash theme={null} docker tag docker.io/keyval/odigos-instrumentor:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-instrumentor:$VERSION docker tag docker.io/keyval/odigos-ui:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION docker tag docker.io/keyval/odigos-odiglet:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-odiglet:$VERSION docker tag docker.io/keyval/odigos-collector:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-collector:$VERSION docker tag docker.io/keyval/odigos-victoria-metrics:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION docker tag docker.io/keyval/odigos-cache:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION ``` > **Note:** Community images use the registry prefix `docker.io/keyval` (for example `docker.io/keyval/odigos-instrumentor`). Image names themselves are `odigos-` without an embedded `keyval/` path segment. If you previously mirrored images under an older layout, re-tag them to match this naming when pushing to your custom registry. ### Step 4: Push the Images Now, push the tagged images to your custom Docker registry: ```bash theme={null} docker push $CUSTOM_DOCKER_REGISTRY/odigos-instrumentor:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-odiglet:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-collector:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION docker push $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION ``` ### Step 5: Configure Access for Private Registries If your Docker registry is private, configure your Kubernetes cluster to pull images from it. Refer to the official Kubernetes documentation for guidance: [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/). Pass the secret name as `imagePullSecrets` at install time so Odigos components can pull mirrored images. For `k8s-init-container` on Kubernetes 1.35+, also copy that secret into each instrumented namespace. See [Init container image pull secrets](../instrumentations/configuration/mount-method#init-container-image-pull-secrets). > **Note: To ensure your Odigos deployments and daemonsets can pull images from a private registry, patch them to add the image pull secret.** > > **Use the following commands:** > > ``` > export ODIGOS_NAMESPACE= > export IMAGE_PULL_SECRET= > kubectl patch deployment odigos-ui -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-instrumentor -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-gateway -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-victoriametrics -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch deployment odigos-cache -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch daemonset odiglet -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > kubectl patch daemonset odigos-data-collection -n $ODIGOS_NAMESPACE \ > --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]' > ``` ### Step 6: Install Odigos Using Custom Images Finally, install Odigos using the images from your custom Docker registry: ```bash theme={null} odigos install --image-prefix=$CUSTOM_DOCKER_REGISTRY ``` ```bash theme={null} helm repo update helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace --set imagePrefix=$CUSTOM_DOCKER_REGISTRY kubectl label namespace odigos-system odigos.io/system-object="true" ``` # Installation Source: https://docs.odigos.io/oss/setup/installation Installation issues? Need help? Message us on [Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) Checkout the [installation options](../setup/installation-options) to customize your installation. There are 3 ways to install odigos in your kubernetes cluster: * [Odigos CLI](#odigos-cli) * [Helm Chart](#helm-chart) * [OpenShift Operator](#openshift-operator) All install methods will deploy and configure resources in the active Kubernetes cluster as per the current context. Before proceeding with the installation, ensure that you are targeting the correct cluster with `kubectl config current-context`. ## Odigos CLI ### Install the CLI If you are on MacOS, you can install Odigos CLI using Homebrew: ```bash theme={null} brew install odigos-io/odigos-cli/odigos ``` 1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux). 2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system. 3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH". * For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt. * For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`. 4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH. 1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows). 2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files. 3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how: * **Option 1**: Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder. * **Option 2**: Add the folder containing the extracted binary to your PATH manually: 1. Right-click the **Start** button and select **System**. 2. Click **Advanced system settings** on the left. 3. In the **System Properties** window, click **Environment Variables** at the bottom. 4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**. 5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file. 6. Click **OK** to close all windows. 4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located. ### Install Odigos ```shell theme={null} odigos install ``` This will install odigos in the `odigos-system` namespace in your kubernetes cluster. foo bar At this point, odigos is installed in your cluster. You can now add sources and destinations to instrument your cluster with OpenTelemetry. ## Helm Chart ### Add Helm Repository ```shell theme={null} helm repo add odigos https://odigos-io.github.io/odigos/ ``` ### Install Odigos ```shell theme={null} helm repo update helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace ``` Once the Odigos pods are online, simply run the following command to launch the Odigos UI. ```shell theme={null} odigos ui ``` ## OpenShift Operator For Operator installation steps, see the docs on: [OpenShift Installation](../setup/installation-options#openshift-installation) # Installation Options Source: https://docs.odigos.io/oss/setup/installation-options You can configure odigos installation using the following options: ## Ignore Namespaces Don't show specific uninteresting namespaces in the Odigos UI. Default values: `kube-system`, `local-path-storage`, `istio-system`, `linkerd`, `kube-node-lease` Add additional namespaces to ignore with **cli**: ```shell theme={null} odigos install --ignore-namespace --ignore-namespace ``` List all namespaces to be ignored with **helm chart**: ```shell theme={null} helm install odigos odigos/odigos --set ignoreNamespace=, ``` Any Source objects for the namespace and workloads in the namespace will still be honored. On GKE, Google may warn that a custom admission controller can affect `kube-system`. The Odigos **pod mutating webhook** is cluster-wide so it can inject agents into application namespaces. It uses `failurePolicy: Ignore`, so webhook errors do not block pod creates in `kube-system`. Ignored namespaces (including `kube-system` by default) are not shown for instrumentation and are not instrumented unless you create a Source for them. ## Ignore Containers This option is useful for excluding specific sidecar containers from being instrumentated by odigos. Default values: `istio-proxy` ```shell theme={null} odigos install --ignore-container --ignore-container ``` ```shell theme={null} helm install odigos odigos/odigos --set ignoreContainer=, ``` ## Custom Container Runtime Socket Path Odigos uses the container runtime socket to query for application environment variables info, and distinguish between those injected by the runtime (via Dockerfile for example) and those defined in Kubernetes manifests. To perform this analysis, Odiglet requires access to the container runtime’s Unix socket (e.g., `containerd.sock`). However, in certain environments—such as Rancher—the socket may not be located in a default path. To address this, Odigos provides a configuration option to mount a custom container runtime socket path into the Odiglet. This path can be specified either during installation or afterwards. ### Usage 1. During and Post installation using set flag for the install command: ```shell theme={null} odigos install --set odiglet.customContainerRuntimeSocketPath= ``` 1. During installation: ```shell theme={null} helm install odigos odigos/odigos --set odiglet.customContainerRuntimeSocketPath= ``` ## OpenShift Installation Installing on OpenShift requires additional settings that can be configured based on your installation mode: Run the Odigos CLI with the `--openshift` flag: ```shell theme={null} odigos install --openshift ``` Set `openshift.enabled=true` in `Values.yaml`: ```yaml theme={null} openshift enabled: true ``` In the OpenShift console, navigate to **Operators** > **OperatorHub** and search for **Odigos**: Click **Install**: Choose a namespace to install the Operator in (this will be the same namespace that Odigos is installed in) and click **Install**: When installation is complete, click **View Operator**: Click **Create Instance** to create an `Odigos` custom resource: If you have an Enterprise Token, fill in the **On Prem Token** field with your Enterprise Token. Otherwise leave it blank to use Odigos OpenSource. Scroll down and click **Create**: Confirm that the Custom Resource has reached state `OdigosInstalled`: OpenShift installation does the following: * Uses [Red Hat certified component container images](https://odigos.io/blog/openshift-certified) * Configures required SELinux permissions for pods to access auto-instrumentation agents on RHEL nodes * Configures RBAC permissions for Odigos components required by default in OpenShift clusters By default, the `--openshift` flag will pull Odigos component images from `registry.connect.redhat.com/odigos`. These images are based on [Red Hat's Universal Base Image 9](https://catalog.redhat.com/software/base-images) and denoted with the `-rhel-certified` suffix. These images are also hosted on the Odigos DockerHub registry, which you can deploy by setting `--openshift` with `--image-prefix docker.io/keyval`. ### Exposing the Odigos UI via OpenShift Routes To access the Odigos UI in OpenShift, use either the standard `odigos ui` command or create a `Route` that exposes the `ui` service on your cluster's external address. For example: ``` oc expose service ui -n odigos-system ``` Then get the external address for the Route with `oc get routes ui -n odigos-system`. **Note:** If you're deploying Odigos on OpenShift, you may need to set the route's TLS termination to Edge to load it in your browser. you will need to modify the YAML of the oc route with `spec.tls.termination: edge` Check the OpenShift documentation or run `oc expose -h` for more information on configuring OpenShift routes according to your cluster's requirements. ## GKE Installation Installing on Google Kubernetes Engine (GKE) requires additional settings that can be configured based on your cluster type: For GKE Standard clusters: ```shell theme={null} odigos install --set gke.enabled=true ``` For GKE Autopilot clusters: ```shell theme={null} odigos install --set gke.enabled=true --set gke.autopilot=true ``` For GKE Standard clusters, set `gke.enabled=true` in `values.yaml`: ```yaml theme={null} gke: enabled: true ``` Or pass it as a flag during installation: ```shell theme={null} helm install odigos odigos/odigos --set gke.enabled=true ``` For GKE Autopilot clusters, also set `gke.autopilot=true`: ```yaml theme={null} gke: enabled: true autopilot: true ``` Or pass both flags during installation: ```shell theme={null} helm install odigos odigos/odigos --set gke.enabled=true --set gke.autopilot=true ``` GKE installation with `gke.enabled=true` does the following: * Configures a `ResourceQuota` for `system-node-critical` pods, which is required for odiglet on GKE ### GKE Autopilot GKE Autopilot restricts privileged workloads by default. Odiglet requires elevated permissions for eBPF-based instrumentation, so it must be allowlisted before it can run on Autopilot clusters. Allowlist synchronization requires GKE version `1.32.2-gke.1652000` or later. When `gke.autopilot=true`, the Helm chart: * Creates an `AllowlistSynchronizer` that syncs the Odigos odiglet allowlist from the [Google-managed partner repository](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads) * Skips node label setup in the odiglet init phase, where Autopilot nodes cannot be modified. Autopilot is auto-detected by the `gk3-` node name prefix; set `gke.autopilot=true` to force this behavior #### Node label limitations On standard clusters, odiglet sets `odigos.io/odiglet-oss-installed=true` (or `odigos.io/odiglet-enterprise-installed=true` for enterprise) on each node after preparing it for instrumentation. Autopilot does not allow modifying node labels, so odiglet skips this step. Without the label: * **Instrumented pod scheduling with hostPath or CSI mount methods** — For `k8s-host-path` and `k8s-csi-driver` mount methods, the instrumentor webhook adds required node affinity matching the odiglet installed label so pods only schedule on nodes where odiglet has prepared `/var/odigos`. Without the label on Autopilot nodes, newly instrumented pods may remain Pending and fail to schedule. The default `k8s-virtual-device` mount method does not rely on this label and is recommended on Autopilot. * **Karpenter integration** — Odiglet normally removes the `odigos.io/needs-init` startup taint when it sets the node label. This step is also skipped on Autopilot, so the Karpenter integration workflow is not supported. #### Manually create the AllowlistSynchronizer If you do not set `gke.autopilot=true`, create the `AllowlistSynchronizer` yourself before odiglet pods are scheduled. See the [GKE documentation for running Autopilot partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads) for more details. Apply the following manifest to install all Odigos allowlists from the partner repository: ```yaml theme={null} apiVersion: auto.gke.io/v1 kind: AllowlistSynchronizer metadata: name: odigos-allowlist-synchronizer spec: allowlistPaths: - "Odigos/Odigos/*" ``` ```shell theme={null} kubectl apply -f allowlist-synchronizer.yaml ``` After the allowlist is installed, verify that it is ready before odiglet pods start: ```shell theme={null} kubectl wait --for=condition=Ready allowlistsynchronizer/odigos-allowlist-synchronizer --timeout=60s kubectl get workloadallowlist ``` For more information on privileged workload allowlisting in Autopilot, see the [GKE documentation on running Autopilot partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads). # Configure Odigos with Argo CD Source: https://docs.odigos.io/oss/setup/odigos-with-argocd This guide shows you how to deploy Odigos in your Kubernetes cluster using Argo CD with the official Odigos Helm chart. This allows for GitOps-based management and observability instrumentation across your applications. ## Prerequisites Before you begin, make sure you have: * A running Kubernetes cluster * Argo CD installed and configured in the cluster * Helm CLI installed (optional but useful for inspection and verification) ## Step 1: Add the Odigos Helm Repository Add the Odigos Helm repository to your Helm configuration: ```bash theme={null} helm repo add odigos https://odigos-io.github.io/odigos --force-update helm repo update ``` ## Step 2: Identify the Helm Chart Version Check available versions of the Odigos chart to determine the desired `targetRevision`: ```bash theme={null} helm search repo odigos --versions ``` ## Step 3: Define the Argo CD Application Manifest Create an Argo CD `Application` custom resource to deploy Odigos using Helm. Save the following YAML as `odigos-application.yaml`: ```yaml theme={null} apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: odigos namespace: # <- Replace with your Argo CD namespace spec: project: default source: repoURL: https://odigos-io.github.io/odigos chart: odigos targetRevision: # REQUIRED: set the version from 'helm search repo odigos --versions' helm: releaseName: odigos values: | # OPTIONAL: add your custom Helm values here if needed. # Argo CD expects this field to be a YAML-formatted string. # To use chart defaults, delete this field entirely or set it to an empty string (""). destination: server: https://kubernetes.default.svc namespace: # <- Replace with the desired namespace where you want Odigos to be installed syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` ## Step 4: Apply the Application Use `kubectl` to create the application in Argo CD: ```bash theme={null} kubectl apply -f odigos-application.yaml ``` This will instruct Argo CD to deploy Odigos using the Helm chart. ## Step 5: Monitor the Deployment You can monitor the status of the deployment through the Argo CD UI: * Navigate to your Argo CD web interface. * Find the application named `odigos`. * Verify that it reaches a **Healthy** and **Synced** state. Once complete, Odigos will be installed in your cluster and ready to instrument workloads. # Configure Odigos with Karpenter Source: https://docs.odigos.io/oss/setup/odigos-with-karpenter ## What is Karpenter? [Karpenter](https://karpenter.sh/) is an open-source cluster autoscaler built for Kubernetes. It dynamically launches just the right compute resources to handle your cluster’s workloads and efficiently scales them based on real-time demand. Unlike traditional autoscalers, Karpenter can provision nodes in seconds and makes scheduling decisions at the pod level, enabling highly dynamic infrastructure scaling. *** ## Mount Method Compatibility Odigos support for Karpenter depends on the configured [mount method](../instrumentations/configuration/mount-method): | Mount method | Karpenter support | What you need | | - | - | - | | [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer) | Works out of the box | No additional Karpenter configuration | | [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) | Supported with Odigos Karpenter integration | Enable `karpenter.enabled` and configure NodePool `startupTaints` (see [HostPath and CSI](#hostpath-and-csi-startup-taints)) | | [`k8s-virtual-device`](../instrumentations/configuration/mount-method#1-VirtualDevice) (default) | Supported only when your Karpenter installation can apply [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) capacity | Enable the NodeOverlay feature (when required) and apply a NodeOverlay CR (see [Virtual Device](#virtual-device-nodeoverlay)) | Follow the section for your mount method below. *** ## Init Container If you use `k8s-init-container`, no Karpenter-specific configuration is required. You can stop here. *** ## Virtual Device (NodeOverlay) With `k8s-virtual-device`, instrumented pods request the extended resource `instrumentation.odigos.io/generic`. That capacity is advertised by the Odigos device plugin on nodes where `odiglet` is already running. Karpenter simulates new nodes from instance-type information. Because that simulation does not include the Odigos device resource by default, Karpenter can fail provisioning with errors such as: ```text theme={null} no instance type has enough resources ... instrumentation.odigos.io/generic: "1" ``` ### Provider and version caveats Support depends on your **Karpenter version, cloud provider, and how the controller is packaged**: * [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is an alpha Karpenter feature (available since Karpenter v1.7.x). Exact enablement steps and API readiness vary by release. * On many installs (including official AWS Karpenter Helm charts), NodeOverlay is **disabled by default** and must be turned on with a feature gate, for example: ```bash theme={null} helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \ --namespace \ --reuse-values \ --set settings.featureGates.nodeOverlay=true ``` Equivalently, set `--feature-gates NodeOverlay=true` / `FEATURE_GATES=...,NodeOverlay=true` on the Karpenter controller. * Not every Karpenter distribution applies NodeOverlays during provisioning, even if the CRD and feature gate exist. Validate in your environment that creating the overlay below allows Karpenter to launch nodes for pods that request `instrumentation.odigos.io/generic`. If NodeOverlay is unavailable or ineffective in your Karpenter install, use [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer), or [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) with the [startup taint integration](#hostpath-and-csi-startup-taints). ### Apply a NodeOverlay After NodeOverlay support is enabled in your Karpenter controller, apply a [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) that adds the Odigos device capacity for the NodePool that should run instrumented workloads. Replace `default` with your NodePool name if needed: ```bash theme={null} cat <<'EOF' | kubectl apply -f - apiVersion: karpenter.sh/v1alpha1 kind: NodeOverlay metadata: name: odigos-instrumentation spec: requirements: - key: karpenter.sh/nodepool operator: In values: ["default"] capacity: instrumentation.odigos.io/generic: "1024" EOF ``` Confirm the overlay is ready: ```bash theme={null} kubectl get nodeoverlay odigos-instrumentation ``` Karpenter should then be able to provision nodes for pods that request `instrumentation.odigos.io/generic`. After the node registers, `odiglet` advertises the real device capacity so pods can schedule. *** ## HostPath and CSI (Startup Taints) For the [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) and [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) mount methods, Odigos adds a **node affinity** rule to instrumented pods so they only run on nodes where `odiglet` is installed. This works fine in static clusters — but in Karpenter-managed clusters, this causes a problem: * `odiglet` is installed by a DaemonSet **after** the node is created. * The label `odigos.io/odiglet-installed=true` is only added to the node at runtime. * **Karpenter does not know in advance** that this label will appear. * So when the scheduler tries to place a pod with this affinity, Karpenter concludes: "no nodes exist (or can be created) that match this label" — and **no node is provisioned**. Once you enable the **Karpenter integration option** (explained below), Odigos will no longer apply node affinity to new instrumented pods. Instead, it relies on a **startup taint mechanism**: * Nodes launched by Karpenter should be initialized with a special taint (`odigos.io/needs-init=NoSchedule`). The exact method for configuring this taint in Karpenter will be explained in the following section. * The `odiglet` removes this taint after it prepares the node for instrumentation. * This allows instrumented pods to schedule only **after** the node is ready — achieving the same safety as affinity, without blocking provisioning. ### 1. Configure the Karpenter NodePool CRD You must configure your Karpenter [NodePool](https://karpenter.sh/docs/concepts/nodepools/) to include a startup taint. This ensures Odigos can prepare each node before instrumented pods are scheduled onto it. Add the following under `spec.template.spec.startupTaints`: ```yaml theme={null} spec: template: spec: startupTaints: - key: odigos.io/needs-init effect: NoSchedule ``` To safely append this taint without overwriting any existing `startupTaints`, use the following command: ```bash theme={null} export NODEPOOL_NAME= kubectl patch nodepool $NODEPOOL_NAME --type merge --patch "$(kubectl get nodepool $NODEPOOL_NAME -o json | jq -c '.spec.template.spec.startupTaints = (.spec.template.spec.startupTaints // []) + [{"key":"odigos.io/needs-init","effect":"NoSchedule"}] | {spec: {template: {spec: .spec.template.spec}}}')" ``` Replace `` with the actual name of your Karpenter NodePool. ### 2. Add the Startup Taint to Existing Nodes To manually add the startup taint to all current nodes: ```bash theme={null} kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} kubectl taint node {} odigos.io/needs-init=:NoSchedule --overwrite ``` This ensures existing nodes behave consistently until they're replaced by Karpenter-managed ones. ### 3. Enable the Karpenter Integration in Odigos Once the tainting setup is complete, inform Odigos to disable node affinity and rely on the taint mechanism. Run the following using the Odigos CLI: ```bash theme={null} odigos set config karpenter-enabled=true ``` If installing or upgrading using helm: ```bash theme={null} helm upgrade --install odigos odigos/odigos \ --namespace odigos-system \ --set karpenter.enabled=true ``` ### 4. Restart the Odiglet DaemonSet To ensure that taint-removal logic is triggered on all nodes, perform a rolling restart of the `odiglet` DaemonSet: ```bash theme={null} kubectl rollout restart daemonset/odiglet -n odigos-system ``` # System Requirements Source: https://docs.odigos.io/oss/setup/system-requirements To ensure optimal performance and compatibility when deploying Odigos, your environment must meet the following system requirements: ## Kubernetes Version Odigos requires Kubernetes version 1.21.12 or higher. ## Kubernetes Cluster Types * [Kind](https://kind.sigs.k8s.io) * [Minikube](https://minikube.sigs.k8s.io/docs/start) * [K3s](https://k3s.io) * [Red Hat OpenShift](https://catalog.redhat.com/software/container-stacks/detail/675201b3ade18c062e2efc0b) (see [how to install on OpenShift](../setup/installation-options#openshift-installation)) * [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/) * [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) (see [how to install on GKE](../setup/installation-options#gke-installation)) * [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service) ## Resource Requirements To achieve its functionality Odigos uses a set of Deployments and DaemonSets. ### Cluster The following components run as Deployment and request the following resources **(per pod)**: | Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit | | - | - | - | - | - | | odigos-autoscaler | 64Mi | 512Mi | 10m | 500m | | odigos-instrumentor (2 replicas) | 64Mi | 512Mi | 10m | 500m | | odigos-ui | 64Mi | 512Mi | 10m | 500m | In addition, Odigos deploys OpenTelemetry Collector Deployment, with auto scaling (multiple replicas) and configurable resources requests and limits. The default values (unless overridden) are: | Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit | | - | - | - | - | - | | odigos-gateway | 600Mi | 600Mi | 1000m | 1000m | Read more [here](../pipeline/configuration). ### Per Node A DaemonSet running on every node in the cluster, consuming resources as outlined below: | Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit | | - | - | - | - | - | | odiglet | 500Mi | 500Mi | 500m | 500m | | data-collection | 250Mi | 500Mi | 250m | 500m | | deviceplugin | 200Mi | 300Mi | 40m | 100m | You can override these values by providing a custom configuration (read more [here](../pipeline/configuration)). ## Network Requirements Ensure that any network-layer components are properly configured to allow the required connectivity between Odigos components. Improper configuration may impact the correct operation of Odigos. To ensure proper operation of Odigos components and instrumented applications, the following network connectivity must be allowed inside the Kubernetes cluster: ### Core Network Connectivity | Source | Destination Service | Ports | Purpose | | - | - | - | - | | Instrumented Application Pods | `odiglet-local` | `4320` | Retrieve instrumentation configuration from Odiglet | | Instrumented Application Pods | `odigos-data-collection-local-traffic` | `4317` (gRPC), `4318` (HTTP) | Export application telemetry data (OTLP protocol) | | Odiglet Pods | `odigos-data-collection-local-traffic` | `4318` | Export eBPF-collected telemetry data | ### Network Layer Considerations In some environments, the following network components may affect traffic between these services: * Service meshes (e.g. Istio, Linkerd, Cilium, Consul) * HTTP proxies or egress gateways * Transparent proxies (corporate or cluster-level) * Network security appliances (WAFs, L7 firewalls, DPI systems) * Kubernetes NetworkPolicies # Uninstall Source: https://docs.odigos.io/oss/setup/uninstall ## With CLI ```shell theme={null} odigos uninstall ``` ## Uninstall Existing Odigos Installation with Helm If you installed Odigos using Helm, you can uninstall it by running the following commands: ```shell theme={null} helm delete odigos -n odigos-system kubectl delete ns odigos-system ``` # Upgrade Source: https://docs.odigos.io/oss/setup/upgrade ## Finding the Current Version ```shell theme={null} odigos version ``` Will print the version of the CLI tool and the version of odigos that is currently installed in the active cluster. ## Upgrading Existing Odigos Installation with CLI First, upgrade the CLI tool to latest version: If you are on MacOS, you can install Odigos CLI using Homebrew: ```bash theme={null} brew install odigos-io/odigos-cli/odigos ``` 1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux). 2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system. 3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH". * For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt. * For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`. 4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH. 1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows). 2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files. 3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how: * **Option 1**: Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder. * **Option 2**: Add the folder containing the extracted binary to your PATH manually: 1. Right-click the **Start** button and select **System**. 2. Click **Advanced system settings** on the left. 3. In the **System Properties** window, click **Environment Variables** at the bottom. 4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**. 5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file. 6. Click **OK** to close all windows. 4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located. And upgrade Odigos installation in the active kubernetes cluster with CLI: ```shell theme={null} odigos upgrade ``` ## Upgrade Existing Odigos Installation with Helm ```shell theme={null} helm repo update helm upgrade odigos odigos/odigos --namespace odigos-system ``` ```shell theme={null} helm repo update helm search repo odigos --versions ## list all available versions helm upgrade --install odigos odigos/odigos \ --namespace odigos-test \ --create-namespace \ --version ``` # Verifying Odigos Images Source: https://docs.odigos.io/oss/setup/verifying-images All Odigos container images are signed at build time with [Sigstore cosign](https://docs.sigstore.dev/) (keyless), and each carries a CycloneDX SBOM attestation. Signatures are stored in the registry next to the image and recorded in the public [Rekor transparency log](https://search.sigstore.dev/), so you can verify that any image you pull was published by the Odigos release pipeline — no keys involved. ## Verify an image Verification needs [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) v3+. Always verify by digest: Verify against the canonical registry, `us-central1-docker.pkg.dev/odigos-cloud/components`. `registry.odigos.io` is a pull alias: images resolve through it, but signature discovery for retroactively signed versions does not currently surface there. ```bash theme={null} IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-odiglet DIGEST=$(docker buildx imagetools inspect "${IMG}:" --format '{{.Manifest.Digest}}') cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \ "${IMG}@${DIGEST}" ``` The identity accepts two workflows on purpose: new releases are signed by the release pipeline, while versions published before signing went live were signed retroactively by the audited backfill workflow. A single-identity pattern would fail on one era or the other. To verify and inspect the attached SBOM (same identity rules): ```bash theme={null} cosign verify-attestation --type cyclonedx \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \ "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components' ``` The SBOM attestation is attached to the **multi-arch index digest** — always pass the digest from `imagetools inspect` as above, not a per-platform child digest (children carry signatures but no attestation). Run as-is — no placeholders to fill in: ```bash theme={null} IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor TAG=v1.35.0 DIGEST=$(docker buildx imagetools inspect "${IMG}:${TAG}" --format '{{.Manifest.Digest}}') IDENTITY='^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' cosign verify \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp "$IDENTITY" \ "${IMG}@${DIGEST}" cosign verify-attestation --type cyclonedx \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --certificate-identity-regexp "$IDENTITY" \ "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components | length' ``` Expected output (trimmed): ```text theme={null} Verification for us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor@sha256:f46fa614… -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates [{"critical":{"identity":{"docker-reference":"…/odigos-instrumentor"}, "image":{"docker-manifest-digest":"sha256:f46fa614…"},"type":"cosign container image signature"}, …}] 1022 ``` The 1022 components are the SBOM contents — inspect them with `jq '.predicate.components'` instead of `length`, e.g. the first entries here are `base-files 13.8+deb13u6` and `github.com/argoproj/argo-rollouts v1.9.1`. Because v1.35.0 predates release-time signing, the certificate for this version names the audited backfill workflow (`ci-core/.github/workflows/backfill-sign.yml`) — that is the expected identity for retroactively signed history, and the reason the identity regexp accepts both workflows. ## Signing identities Each artifact is signed by its own release workflow — never verify with a broad pattern like `.*odigos-io.*`. The canonical list of identities and the version each artifact is signed from lives in [`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md). | Artifact | Release identity (regexp) | | - | - | | Component images (`odigos-*`) | `^https://github\.com/odigos-io/odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*$` | | CLI image + `odigos-victoria-metrics` + `odigos-cache` | `^https://github\.com/odigos-io/odigos/\.github/workflows/release\.yml@refs/heads/main$` | | Enterprise images (`odigos-enterprise-*`) | `^https://github\.com/odigos-io/odigos-enterprise/\.github/workflows/release-images\.yml@refs/(tags/.*\|heads/.*)$` | | `odigos-cli-offsets` | `^https://github\.com/odigos-io/enterprise-go-instrumentation/\.github/workflows/offsets-push\.yml@refs/heads/.*$` | | `odigos-vmagent-instrumentations` | `^https://github\.com/odigos-io/vm-agent/\.github/workflows/publish-agents\.yml@refs/(heads/.*\|tags/agents/.*)$` | To also cover retroactively signed history, combine the release identity with the backfill identity in one pattern, as the commands above do: `^https://github\.com/odigos-io/(@|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$` Two more certificate details are expected and legitimate: * **Backfill**: historical releases were signed retroactively by `odigos-io/ci-core/.github/workflows/backfill-sign.yml` — the certificate honestly records the backfill identity and date, and it is auditable in Rekor. * **Stable `x.y.0` tags** verify against the release-candidate tag identity: the stable tag is a retag of the exact RC digest, so the certificate names the RC that built it. ## Attestable images Every image below is signed and carries a CycloneDX SBOM attestation, from the version listed in [`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md): * **Odigos platform**: `odigos-instrumentor`, `odigos-collector`, `odigos-odiglet`, `odigos-ui`, `odigos-operator`, `odigos-agents`, `odigos-cli`, `odigos-victoria-metrics`, `odigos-cache` * **Odigos Enterprise**: `odigos-enterprise-odiglet`, `odigos-enterprise-instrumentor`, `odigos-enterprise-ui`, `odigos-enterprise-collector`, `odigos-enterprise-central-proxy`, `odigos-enterprise-central-backend`, `odigos-enterprise-central-ui`, `odigos-enterprise-agents`, `odigos-enterprise-connector-runtime`, `odigos-enterprise-connector-aws`, `odigos-enterprise-connector-azure`, `odigos-enterprise-connector-gcp` — `odiglet` and `agents` also publish `-extended` variants (separate digests, same identity) * **Supporting images**: `odigos-cli-offsets`, and `odigos-vmagent-instrumentations` (signed for publishes after 2026-08-12) ## Air-gapped verification The Rekor proof travels inside the signature bundle, so verification works offline: supply `--trusted-root` with a Sigstore trusted-root file instead of relying on `--offline` (removed in cosign 3.x). # Telemetry Types Source: https://docs.odigos.io/oss/telemetry-types Odigos supports the producing and correlation of the following telemetry types: * Traces * Metrics * Logs For a telemetry type to be collected, a [destination](./backends-overview) that accepts it must be configured. In addition, users can enable or disable the collection of a telemetry type on a per-application basis. If none of the applications are configured to collect a telemetry type, the telemetry type will not be collected. ## Traces Distributed traces collected by Odigos will automatically include spans for popular open source projects such as HTTP clients and servers, gRPC clients and servers, Database clients, and many more. In addition, users can enrich their distributed traces with manually created spans by using the relevant [OpenTelemetry APIs](https://opentelemetry.io/docs/instrumentation/). The collection of traces is achieved by combining two open source technologies: * OpenTelemetry for languages with JIT compilation such as Python, Java,.NET and Javascript. * eBPF for compiled languages such as Go. Being based on popular open-source standards allows Odigos to automatically support a huge number of libraries and frameworks. Below is a list of supported libraries for every language: ## Metrics There are three kinds of metrics that Odigos supports: * Metrics related to the running of the application (number of HTTP requests, latency, DB connections, etc.) * Metrics related to the language runtime (GC, threads, heap, etc.) * Metrics related to the host environment (CPU, memory, disk, etc.) ### Applicative Metrics Application-related metrics are derived from the distributed traces described above. Many observability vendors automatically compute application metrics according to the distributed traces. For example, the number of requests served by the application is computed by counting the number of spans with the `http.server` label. In case a destination that does not automatically compute application metrics (like Prometheus + Tempo) is configured, Odigos will compute them on its own by invoking the [spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md) connector. ### Runtime Metrics Those metrics describe the behavior of the language runtime, such as garbage collection, threads, heap, etc. Runtime metrics are collected by the same mechanism that collects distributed traces. ### Host Metrics Host-related metrics are collected via the [host metrics receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) ## Logs Currently, Odigos will ship logs written by the application to the stdout or stderr of the process. Correlation to other telemetry types will be done according to timestamp and resource. We are in the process of shipping logs using the same mechanism that collects distributed traces, this would allow users to correlate logs to distributed traces and metrics in a more accurate way. # Usage report Source: https://docs.odigos.io/oss/usage-reports Odigos sends non-sensitive, non-personally identifiable information to our system. This information helps Odigos maintainers to understand how the open-source community runs Odigos and improve the project. ### Disabling the report #### Using helm chart When installing Odigos using helm chart, you can disable the report by setting `telemetry.enabled` to `false`. #### Using CLI When installing Odigos using CLI, you can disable the report by setting `--telemetry` to `false`. ### What is being reported Odigos collects the following anonymous information: * Are traces/metrics/logs enabled * Selected destinations type (e.g. Jaeger, Prometheus, etc.) * Number of instrumented applications by programming language * Number of unrecognized applications * Number of nodes # ECS Configuration Reference Source: https://docs.odigos.io/vmagent/ecs/configuration Every environment variable of the Odigos ECS Agent and ECS Proxy, how platform names are derived, and where configuration is persisted. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). On ECS the **task definition is the configuration surface**. On a fresh container instance it is the only input the agent has, so the Central endpoint and any other defaults travel inside it — nothing depends on service discovery or a pre-provisioned file. Everything beyond bootstrap — [sources](/vmagent/setup/configuration/add-sources), [destinations](/vmagent/setup/configuration/add-destinations), [actions](/vmagent/setup/configuration/actions/overview), and [instrumentation rules](/vmagent/setup/configuration/instrumentation-rules/overview) — is managed from Odigos Central or with [`odictl`](/vmagent/setup/configuration/odictl), exactly as on any VM Agent. ## Agent environment variables | Variable | Required | Effect | | - | - | - | | `ODIGOS_DEFAULT_CONFIG` | Recommended | A full config document (YAML **or** JSON, same shape as `config.yaml`) that seeds the agent on a fresh node. This is where you set the Central endpoint: `{"central":{"enabled":true,"endpoint":"central.example.com:443"}}`. In Agent + Proxy mode, point it at the proxy instead. | | `ODIGOS_ONPREM_TOKEN` | Yes, unless `license.jwt` is pre-provisioned | Written to `/etc/odigos-ecs-agent/license.jwt` at startup. Deliver it from AWS Secrets Manager via the task definition's `secrets` block, not as a plain `environment` value. | | `ODIGOS_CENTRAL_ENDPOINT` | No | Shorthand for just the endpoint (`host:port` or URL) when you don't need a full document; equivalent to setting `central.enabled` + `central.endpoint`. `ODIGOS_DEFAULT_CONFIG` wins if both are set. | | `ODIGOS_ECS_BEHIND_PROXY` | Agent + Proxy mode | `true` tells the agent it connects to an `odigos-ecs-proxy` rather than to Central. Set it in [Agent + Proxy](/vmagent/ecs/installation-with-proxy) mode; omit it in [Agent only](/vmagent/ecs/installation-agent-only) mode. An endpoint alone cannot distinguish the two, and the reported platform name differs per mode. | | `ODIGOS_AGENT_NAME` | No | Platform name reported upstream, overriding the per-mode default below. | | `ODIGOS_ECS_CLUSTER_NAME` | No | The ECS cluster name. Normally unnecessary — the agent reads the cluster from the ECS task metadata endpoint. Set it when metadata is unavailable, or to report a different name. | `:443` in an endpoint implies `wss`. The connection is **outbound-only**, so a public Central endpoint or one in a different AWS account works with no inbound rules on the cluster. ### Platform naming per mode The agent's name depends on the topology so each mode displays naturally in the Central UI: | Mode | Platform name | Why | | - | - | - | | **Agent only** (no proxy) | the **ECS cluster name**, auto-detected | Each node registers with Central directly; naming the platform after the cluster is what makes a single-node cluster appear as the cluster rather than as an EC2 hostname. | | **Agent + Proxy** | the **node hostname** | The proxy is what reports the cluster name upstream; it identifies its agents per node, and its coverage check matches agent names against the instances' private DNS. | `ODIGOS_AGENT_NAME` overrides both. In Agent-only mode, if the cluster cannot be determined (task metadata unavailable), the agent falls back to the hostname and logs a warning. Both modes register with platform type **`aws.ecs`**, so a cluster presents consistently in Central either way. ### Configuration precedence and persistence `ODIGOS_DEFAULT_CONFIG` is a **default**, not an override. Precedence, lowest to highest: The baseline shipped in the image. `ODIGOS_CENTRAL_ENDPOINT`, `ODIGOS_AGENT_NAME`, and so on. The inline document from the task definition. Whatever Central or `odictl` has written. **This wins on every boot after the first.** The config tree persists on the container **instance** at `/etc/odigos-ecs-agent` (a host bind mount), so the agent keeps its deployment identity and configuration across task restarts and image upgrades. Because the persisted `config.yaml` takes precedence, **editing `ODIGOS_DEFAULT_CONFIG` does not change a value that has already been persisted**. To change a seeded value after first boot, update it through Central or `odictl` — or clear the persisted config on the instance so the new default is applied again. ## Proxy environment variables | Variable | Required | Effect | | - | - | - | | `ODIGOS_CLUSTER_NAME` | Yes | The ECS cluster this proxy represents. Becomes Central's platform name and derives the proxy ID (`ecs-cluster/`). | | `ODIGOS_CENTRAL_ENDPOINT` | No | Central `host:port`. Empty runs the proxy downstream-only (agents still aggregate against it). | | `ODIGOS_CENTRAL_SECURE_WS` | No | `true` forces `wss` upstream (also implied by `:443`). | | `ODIGOS_PLATFORM_TYPE` | No | Platform type reported upstream. Defaults to `aws.ecs`, matching the per-node agent. | | `ODIGOS_PROXY_ID` | No | Overrides the derived Central connection id. | | `ODIGOS_PROXY_LISTEN` | No | Agent listen address. Default `:4321`. | | `ODIGOS_PROXY_JOURNAL` | No | Config-journal **file** path. Default `/var/lib/odigos-ecs-proxy/journal.jsonl`. Point it at a file on an [EFS mount](/vmagent/ecs/installation-with-proxy#persist-the-config-journal-on-efs) if the journal must survive task replacement. | | `ODIGOS_COVERAGE_CHECK` | No | `true` enables the expected-vs-connected check: the proxy lists the cluster's container instances through the ECS/EC2 APIs and reports ACTIVE instances that have no connected agent. Requires `ecs:ListContainerInstances`, `ecs:DescribeContainerInstances`, and `ec2:DescribeInstances` on the task role. | | `ODIGOS_COVERAGE_INTERVAL` | No | Coverage check period (Go duration). Default `60s`. | The proxy is unprivileged — no host PID, no Docker socket, no host mounts. ### What the proxy does with requests | Request | Routing | | - | - | | Config queries (`sources`, `destinations`, `actions`, `instrumentationRules`, `config`, …) | Any one healthy agent — the config is identical on every node. | | Live-state queries (`workloadReports`, counters) | Fan out to all agents; lists merged. | | Mixed queries (config + live fields in one operation) | Fan out; live lists concatenate, config fields keep a single agent's value so they are never multiplied by node count. | | Config mutations (`createSource`, `updateDestination`, …) | Broadcast to every agent **and journaled**. | | Live mutations (`restartSource`, …) | Broadcast, not journaled. | | Profiling jobs, diagnose bundles | One agent. | The journal replays config mutations to agents that connect later, so a node that joins the cluster — or a task that migrates to a fresh instance — converges to the cluster configuration automatically. Each agent's replay position is tracked by its stable deployment id, so a reconnecting agent replays only what it missed. ### Membership and lifecycle events * Agent connects and disconnects are logged and forwarded to Central as `ecs-proxy.agent-lifecycle` events; departed agents keep a tombstone (id, node, last seen). * Coverage changes are logged and forwarded as `ecs-proxy.coverage` events. * `GET /healthz` on port 4321 returns the whole view: connected agents, departed tombstones, and the last coverage result. See [checking the healthz endpoint](/vmagent/ecs/troubleshooting#check-the-proxy-healthz-endpoint). ## What gets instrumented | | Behavior on ECS | | - | - | | ECS-managed containers | **Instrumented.** Discovered through the instance's Docker daemon. | | Host processes and `systemd` services on the container instance | **Ignored.** Unlike the [VM Agent](/vmagent/setup/installation) and [Docker deployment](/vmagent/setup/docker), the ECS distribution only classifies ECS tasks. | | Task restarts | **Never.** All instrumentation is direct attach; ECS owns the container lifecycle, so the agent does not restart, recreate, or modify tasks. | | Java, Go, C++ | Their dynamic eBPF SDKs (Java via dynamic attach). | | Python, Node.js, .NET, PHP, Ruby, others | Covered by [OBI](/oss/instrumentations/obi), which also attaches with no restart. Restart-based environment injection is intentionally not used on ECS. | The agent's own binaries, the `amazon-ecs-agent`, the SSM agent, and `awsvpc` pause containers are excluded from instrumentation automatically. ## How the collector runs As in the [Docker deployment](/vmagent/setup/docker#how-the-collector-runs), there is no `systemd` inside the container: * `odigos-otelcol` runs as a **supervised child process** of the agent, so its lifecycle follows the agent task. * Agent and collector **share the task's stdout**, captured by the `awslogs` driver into the `/odigos/ecs-agent` log group. * With host networking, the collector and OpAMP server listen on the instance's network namespace, reachable by instrumented tasks on localhost or the instance IP. * Collector logs are JSON-formatted on ECS; the `awslogs` driver ships them to CloudWatch Logs as the primary log stream. * [systemd memory limits](/vmagent/setup/configuration/services-mem-limits-configuration) do not apply — cap the agent with the task definition's `cpu`, `memory`, and `memoryReservation` instead. ## Instrumentation file sync The image bakes the Java instrumentation files into a staging directory. On startup the agent rsyncs them onto the container instance's `/var/odigos` (the task definition bind mount), and seeds them into a workload's container namespace when it attaches. Go, C++, and the OBI fallback are compiled into the agent binary and need no on-disk files. This is why `/var/odigos` must be a writable host mount. The full list of elevated task settings is under [required permissions](/vmagent/ecs/requirements#required-permissions). # Install on ECS: Single Node (Agent only) Source: https://docs.odigos.io/vmagent/ecs/installation-agent-only Deploy the Odigos ECS Agent as a daemon service. Every container instance runs one agent that connects directly to Odigos Central — no extra AWS resources. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). This is the default and simplest ECS deployment: **one agent task per EC2 container instance**, each connecting **directly** to Odigos Central. No proxy, no load balancer, no service discovery. It works identically whether the cluster has 1 node or 50 — the only difference is that **each node appears in Central as its own platform**. If you want the whole cluster to appear as a single platform with an aggregated workload list, use [Install: Agent + Proxy](/vmagent/ecs/installation-with-proxy) instead. Complete the [System Requirements](/vmagent/ecs/requirements) first — in particular the **EC2 launch type** and the **kernel 5.10+ with BTF** requirement (use the Amazon Linux 2023 ECS-optimized AMI). ## Step 1: Set your session variables Run every command in this guide in the **same shell**. ```bash theme={null} export AWS_REGION= export CLUSTER= export ACCOUNT=$(aws sts get-caller-identity --query Account --output text) export REG=${ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com export TAG= export CENTRAL_ENDPOINT=':' # e.g. central.example.com:443 (:443 implies wss) echo "ACCOUNT=$ACCOUNT REG=$REG" # neither may be empty before continuing ``` ## Step 2: Store the license token in Secrets Manager ```bash theme={null} export SECRET_ARN=$(aws secretsmanager create-secret --name odigos-onprem-token \ --secret-string "" --query ARN --output text 2>/dev/null \ || aws secretsmanager describe-secret --secret-id odigos-onprem-token --query ARN --output text) echo "SECRET_ARN=$SECRET_ARN" # must be non-empty ``` That create-or-fetch form is idempotent — safe to re-run if the secret already exists. If IAM or an AWS Organizations SCP blocks `secretsmanager:GetSecretValue`, you can put the token directly in the task definition instead: drop the `secrets` block and add `ODIGOS_ONPREM_TOKEN` to `environment`. The token then becomes readable to anyone who can call `ecs:DescribeTaskDefinition`. Acceptable for a throwaway test cluster, not for production. ## Step 3: Create the IAM roles ```bash theme={null} TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ecs-tasks.amazonaws.com"},"Action":"sts:AssumeRole"}]}' # Execution role: image pull, logs, and the license secret. aws iam create-role --role-name odigos-ecs-exec-role --assume-role-policy-document "$TRUST" aws iam attach-role-policy --role-name odigos-ecs-exec-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy aws iam put-role-policy --role-name odigos-ecs-exec-role --policy-name odigos-extras \ --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"logs:CreateLogGroup\",\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":\"secretsmanager:GetSecretValue\",\"Resource\":\"arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT}:secret:odigos-onprem-token-*\"}]}" # Task role: needed for odictl through ECS Exec. aws iam create-role --role-name odigos-ecs-task-role --assume-role-policy-document "$TRUST" aws iam put-role-policy --role-name odigos-ecs-task-role --policy-name ssm-exec \ --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssmmessages:CreateControlChannel","ssmmessages:CreateDataChannel","ssmmessages:OpenControlChannel","ssmmessages:OpenDataChannel"],"Resource":"*"}]}' ``` Note the trailing `-*` on the secret ARN in the policy: Secrets Manager appends a random 6-character suffix to every secret, and that suffix changes if the secret is deleted and recreated. The wildcard means you never have to touch this policy again. `EntityAlreadyExists` on `create-role` means the role exists from a previous attempt — safe to continue with the `attach-role-policy` / `put-role-policy` commands. ## Step 4: Register the task definition Save the task definition below as `odigos-ecs-agent.json`, then substitute your values. ```json theme={null} { "family": "odigos-ecs-agent", "requiresCompatibilities": ["EC2"], "networkMode": "host", "pidMode": "host", "executionRoleArn": "", "taskRoleArn": "", "volumes": [ { "name": "docker-sock", "host": { "sourcePath": "/var/run/docker.sock" } }, { "name": "docker-lib", "host": { "sourcePath": "/var/lib/docker" } }, { "name": "sys-kernel-debug", "host": { "sourcePath": "/sys/kernel/debug" } }, { "name": "odigos-config", "host": { "sourcePath": "/etc/odigos-ecs-agent" } }, { "name": "odigos-instrumentations", "host": { "sourcePath": "/var/odigos" } } ], "containerDefinitions": [ { "name": "odigos-ecs-agent", "image": "", "essential": true, "privileged": true, "cpu": 512, "memoryReservation": 512, "memory": 2048, "stopTimeout": 120, "mountPoints": [ { "sourceVolume": "docker-sock", "containerPath": "/var/run/docker.sock", "readOnly": true }, { "sourceVolume": "docker-lib", "containerPath": "/var/lib/docker", "readOnly": true }, { "sourceVolume": "sys-kernel-debug", "containerPath": "/sys/kernel/debug", "readOnly": false }, { "sourceVolume": "odigos-config", "containerPath": "/etc/odigos-ecs-agent", "readOnly": false }, { "sourceVolume": "odigos-instrumentations", "containerPath": "/var/odigos", "readOnly": false } ], "environment": [ { "name": "ODIGOS_DEFAULT_CONFIG", "value": "{\"central\":{\"enabled\":true,\"endpoint\":\"\"}}" } ], "secrets": [ { "name": "ODIGOS_ONPREM_TOKEN", "valueFrom": "" } ], "ulimits": [ { "name": "memlock", "softLimit": -1, "hardLimit": -1 }, { "name": "nofile", "softLimit": 65536, "hardLimit": 65536 } ], "linuxParameters": { "initProcessEnabled": true }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/odigos/ecs-agent", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "odigos-ecs-agent" } } } ] } ``` All settings are required for specific capabilities. For details, see [required permissions](/vmagent/ecs/requirements#required-permissions). Substitute the placeholders: ```bash theme={null} sed -e "s||${REG}/odigos-ecs-agent:${TAG}|" \ -e "s||${CENTRAL_ENDPOINT}|" \ -e "s||${SECRET_ARN}|" \ -e "s||arn:aws:iam::${ACCOUNT}:role/odigos-ecs-exec-role|" \ -e "s||arn:aws:iam::${ACCOUNT}:role/odigos-ecs-task-role|" \ -e "s||${AWS_REGION}|" \ odigos-ecs-agent.json > /tmp/agent-td.json ``` **Verify before registering.** A leftover `` produces a task that starts fine but can never connect — and the failure only shows up in the logs minutes later. ```bash theme={null} grep -c '<' /tmp/agent-td.json # want 0 jq -r '.containerDefinitions[0].environment[] | select(.name=="ODIGOS_DEFAULT_CONFIG").value' /tmp/agent-td.json | jq . # → {"central": {"enabled": true, "endpoint": "central.example.com:443"}} # confirm this is your REAL endpoint ``` Register it: ```bash theme={null} aws ecs register-task-definition --cli-input-json file:///tmp/agent-td.json >/dev/null && echo "task definition registered" ``` ## Step 5: Create the daemon service The `DAEMON` scheduling strategy runs exactly one agent task per container instance, including on instances that join the cluster later. ```bash theme={null} aws ecs create-service \ --cluster "$CLUSTER" \ --service-name odigos-ecs-agent \ --task-definition odigos-ecs-agent \ --scheduling-strategy DAEMON \ --launch-type EC2 \ --enable-execute-command ``` `create-service` is a **one-time** operation — re-running it fails with `Creation of service was not idempotent`. To roll out a new image or an env-var change later, register a new task definition revision and run [`update-service`](/vmagent/ecs/maintenance#upgrade) instead. `--enable-execute-command` is what lets you later run `odictl` inside the agent container. ## Step 6: Verify ```bash theme={null} aws ecs describe-services --cluster "$CLUSTER" --services odigos-ecs-agent \ --query 'services[0].{running:runningCount,desired:desiredCount,events:events[:3].message}' ``` Wait until `running == desired`. Any placement failures appear in `events` — see [Troubleshooting](/vmagent/ecs/troubleshooting). ```bash theme={null} TASK=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name odigos-ecs-agent \ --desired-status RUNNING --query 'taskArns[0]' --output text) aws ecs execute-command --cluster "$CLUSTER" --task "$TASK" \ --container odigos-ecs-agent --interactive --command "uname -r" ``` Expect `6.1.x` (or any 5.10+). `4.14.x` means the instance is on the Amazon Linux 2 stock AMI and **nothing will be instrumented** — see [System Requirements](/vmagent/ecs/requirements#container-instance-kernel-and-ami). ```bash theme={null} aws logs tail /odigos/ecs-agent --since 5m --follow ``` You want to see the agent connect to Central, and you want **no** `runtime-detector stopped` errors. Open the Central UI. You should see **one platform per container instance**, each named after the ECS cluster. Your ECS tasks written in a supported language appear as discovered workloads, ready to be turned into [sources](/vmagent/setup/configuration/add-sources). ## Step 7: Build your pipeline The ECS Agent is configured exactly like any other VM Agent — from Odigos Central, or with `odictl` inside the agent container: ```bash theme={null} aws ecs execute-command --cluster "$CLUSTER" --task "$TASK" \ --container odigos-ecs-agent --interactive --command odictl ``` Choose which ECS tasks to instrument. Send the telemetry to your observability backend. Transform, filter, and enrich telemetry. ECS-specific environment variables and permissions. In this mode `odictl` targets **single node's** agent — there is no cluster-wide `odictl`. Configuration made through Central applies to the platform you selected, i.e. that node. For cluster-wide configuration, use [Agent + Proxy](/vmagent/ecs/installation-with-proxy). ## Moving to Agent + Proxy later Nothing is thrown away. Deploy the proxy, re-point the agents, and roll the daemon — no reinstall. Follow [Switch to Multi Node](/vmagent/ecs/installation-switch-to-multi-node). # Switch from Single Node to Multi Node Source: https://docs.odigos.io/vmagent/ecs/installation-switch-to-multi-node Move an existing Agent-only ECS deployment to Agent + Proxy so the cluster appears in Odigos Central as one platform — without reinstalling the agents. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). No reinstall. The agents already running from [Install: Single Node](/vmagent/ecs/installation-agent-only) speak the same protocol to the proxy that they speak to Central. You add the proxy, re-point the agents at it, and roll the daemon. After the switch, Central shows **one** platform named after the ECS cluster, replacing the per-node platforms. You need the **`odigos-ecs-proxy`** image in addition to the agent image you already run. Mirror it the same way as in [System Requirements](/vmagent/ecs/requirements#license-and-images). ## Switch Follow [Install: Multi Node](/vmagent/ecs/installation-with-proxy) steps 3–5 (stable address, proxy task definition, proxy service). Prefer the internal NLB; a pinned instance is test/dev only. Do **not** create a second agent service — keep the daemon you already have. In the agent task definition, change `ODIGOS_DEFAULT_CONFIG`'s endpoint to the proxy's `host:4321` **and** add `ODIGOS_ECS_BEHIND_PROXY=true`. Register the new revision. The endpoint alone cannot tell a proxy from Central. Without `ODIGOS_ECS_BEHIND_PROXY=true`, every agent still registers under the cluster name and the coverage check reports all instances as uncovered. See [platform naming](/vmagent/ecs/configuration#platform-naming-per-mode). ```bash theme={null} aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-agent \ --task-definition odigos-ecs-agent --force-new-deployment ``` Use [Install: Multi Node, Step 8](/vmagent/ecs/installation-with-proxy#step-8-verify): agents connected to the proxy, `/healthz` lists every node, and Central shows one platform for the cluster. Values already persisted in each node's `config.yaml` [take precedence](/vmagent/ecs/configuration#configuration-precedence-and-persistence) over `ODIGOS_DEFAULT_CONFIG`. After the first boot behind the proxy, change the endpoint through Central or `odictl` if a node still points at Central. ## Switch back to single node The same flow in reverse: 1. Point `ODIGOS_DEFAULT_CONFIG`'s endpoint back at Central. 2. Remove `ODIGOS_ECS_BEHIND_PROXY` (or set it to `false`). 3. Register a new revision and `update-service --force-new-deployment`. 4. Delete the proxy service. Each node then appears in Central as its own platform again. ## Next steps Full Agent + Proxy install if you are starting from scratch. `ODIGOS_ECS_BEHIND_PROXY` and the proxy environment variables. # Install on ECS: Multi Node (Agent + Proxy) Source: https://docs.odigos.io/vmagent/ecs/installation-with-proxy Deploy the Odigos ECS Agent daemon plus the cluster-scoped ECS Proxy, so a multi-node ECS cluster appears in Odigos Central as a single platform with aggregated workloads. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). This deployment runs the agent daemon on every container instance **plus one `odigos-ecs-proxy` task per cluster**. The agents dial the proxy instead of Central, so the proxy is the cluster's single upstream connection and the whole cluster appears in Odigos Central as **one** platform — with an aggregated workload list, cluster-wide configuration, a config journal for nodes that join later, and an optional coverage check. This guide is a **superset** of [Install: Agent Only](/vmagent/ecs/installation-agent-only): same agent, same task definition, plus one component. If you don't need cluster-wide aggregation, that guide is simpler. Complete the [System Requirements](/vmagent/ecs/requirements) first. You need **both** images here: `odigos-ecs-agent` and `odigos-ecs-proxy`. ## Deployment order Deploy the **proxy first**, then the agents. That way the agents connect on their very first boot, with the correct proxy address baked into their first task definition revision — no retry-loop noise and no redeploy. Internal NLB (preferred), or a pinned instance for test/dev. One task per cluster, listening on TCP 4321. Enable the proxy, with `ODIGOS_ECS_BEHIND_PROXY=true`. ## Step 1: Set your session variables Run every command in this guide in the **same shell**. ```bash theme={null} export AWS_REGION= export CLUSTER= export ACCOUNT=$(aws sts get-caller-identity --query Account --output text) export REG=${ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com export TAG= export CENTRAL_ENDPOINT=':' # e.g. central.example.com:443 echo "ACCOUNT=$ACCOUNT REG=$REG" # neither may be empty before continuing ``` ## Step 2: License token and IAM roles Identical to the Agent-only guide — the **agent** needs the license token; the proxy never touches it. Both task definitions reuse the same two roles. ```bash theme={null} export SECRET_ARN=$(aws secretsmanager create-secret --name odigos-onprem-token \ --secret-string "" --query ARN --output text 2>/dev/null \ || aws secretsmanager describe-secret --secret-id odigos-onprem-token --query ARN --output text) echo "SECRET_ARN=$SECRET_ARN" # must be non-empty TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ecs-tasks.amazonaws.com"},"Action":"sts:AssumeRole"}]}' aws iam create-role --role-name odigos-ecs-exec-role --assume-role-policy-document "$TRUST" aws iam attach-role-policy --role-name odigos-ecs-exec-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy aws iam put-role-policy --role-name odigos-ecs-exec-role --policy-name odigos-extras \ --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"logs:CreateLogGroup\",\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":\"secretsmanager:GetSecretValue\",\"Resource\":\"arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT}:secret:odigos-onprem-token-*\"}]}" aws iam create-role --role-name odigos-ecs-task-role --assume-role-policy-document "$TRUST" aws iam put-role-policy --role-name odigos-ecs-task-role --policy-name ssm-exec \ --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssmmessages:CreateControlChannel","ssmmessages:CreateDataChannel","ssmmessages:OpenControlChannel","ssmmessages:OpenDataChannel"],"Resource":"*"}]}' ``` The proxy's [coverage check](/vmagent/ecs/configuration#proxy-environment-variables) reports ACTIVE container instances that have **no** agent connected — the check that catches a daemon that failed to place, a crash-looping agent, or a license failure. It calls the ECS/EC2 inventory APIs: ```bash theme={null} aws iam put-role-policy --role-name odigos-ecs-task-role --policy-name odigos-coverage \ --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ecs:ListContainerInstances","ecs:DescribeContainerInstances","ec2:DescribeInstances"],"Resource":"*"}]}' ``` Without these permissions, leave `ODIGOS_COVERAGE_CHECK` unset — everything else works. ## Step 3: Give the proxy a stable address The agents need one fixed `host:4321` reachable from the container instances. That is the proxy's **only** networking requirement — no Cloud Map, no service discovery. Pick one of two options. **Option B (pinned instance) is for test and development only.** If that instance is replaced, the proxy's address changes and you must re-point both the placement constraint and the agents. See [the pinned-IP caveat](#caveat-the-pinned-proxy-address). Use Option A (internal NLB) in production. An internal Network Load Balancer is the production method: it survives instance and task replacement. The proxy runs with `awsvpc` networking and registers its ENI IP in the target group. ```bash theme={null} aws elbv2 create-load-balancer --name odigos-ecs-proxy --type network \ --scheme internal --subnets aws elbv2 create-target-group --name odigos-ecs-proxy --protocol TCP \ --port 4321 --target-type ip --vpc-id aws elbv2 create-listener --load-balancer-arn --protocol TCP \ --port 4321 --default-actions Type=forward,TargetGroupArn= ``` The NLB DNS name is the agents' endpoint: ```bash theme={null} export PROXY_ENDPOINT='odigos-ecs-proxy-xxxx.elb..amazonaws.com:4321' ``` Keep `"networkMode": "awsvpc"` and the `portMappings` block in the proxy task definition below. Run the proxy with host networking on one designated container instance; its private IP is the endpoint. No extra AWS resources, but the address is tied to that instance. First confirm the cluster actually has a registered container instance: ```bash theme={null} aws ecs describe-clusters --clusters "$CLUSTER" \ --query 'clusters[0].{status:status,registered:registeredContainerInstancesCount}' ``` If `registered` is `0`, wait for the instances to boot and register: ```bash theme={null} until [ "$(aws ecs describe-clusters --clusters "$CLUSTER" --query 'clusters[0].registeredContainerInstancesCount' --output text)" -ge 1 ]; do echo waiting...; sleep 10 done ``` Then pick the instance to pin to: ```bash theme={null} CI=$(aws ecs list-container-instances --cluster "$CLUSTER" --query 'containerInstanceArns[0]' --output text) if [ -z "$CI" ] || [ "$CI" = "None" ]; then echo "no container instances registered in $CLUSTER yet" >&2 else export EC2_ID=$(aws ecs describe-container-instances --cluster "$CLUSTER" --container-instances "$CI" \ --query 'containerInstances[0].ec2InstanceId' --output text) export PROXY_ENDPOINT="$(aws ec2 describe-instances --instance-ids "$EC2_ID" \ --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text):4321" echo "proxy → $EC2_ID @ $PROXY_ENDPOINT" fi ``` Don't skip the guard. With no registered instances, `list-container-instances` returns the literal string `None`, `EC2_ID` ends up empty — and an **empty** `--instance-ids` makes `describe-instances` query every instance in the region, silently handing you the private IP of an unrelated EC2 instance instead of failing. The container instances' security group must allow TCP **4321** — from the NLB (Option A) or from the other instances (Option B). ## Step 4: Register the proxy task definition Save this as `odigos-ecs-proxy.json`. ```json theme={null} { "family": "odigos-ecs-proxy", "requiresCompatibilities": ["EC2"], "networkMode": "awsvpc", "cpu": "256", "memory": "512", "executionRoleArn": "", "taskRoleArn": "", "containerDefinitions": [ { "name": "odigos-ecs-proxy", "image": "", "essential": true, "portMappings": [ { "containerPort": 4321, "protocol": "tcp" } ], "environment": [ { "name": "ODIGOS_CLUSTER_NAME", "value": "" }, { "name": "ODIGOS_CENTRAL_ENDPOINT", "value": "" }, { "name": "ODIGOS_PLATFORM_TYPE", "value": "aws.ecs" }, { "name": "ODIGOS_COVERAGE_CHECK", "value": "true" } ], "healthCheck": { "command": ["CMD-SHELL", "bash -c '", "awslogs-stream-prefix": "odigos-ecs-proxy" } } } ] } ``` **Option A (NLB) keeps the file as-is.** Option B (pinned instance) requires two edits: change `"networkMode": "awsvpc"` to `"host"`, and remove the `portMappings` block. `ODIGOS_CLUSTER_NAME` becomes the platform name in Central. Drop `ODIGOS_COVERAGE_CHECK` if you did not grant the coverage permissions in Step 2. See the [proxy environment variables](/vmagent/ecs/configuration#proxy-environment-variables) for the full list. To keep the config journal across task replacement, add the EFS volume shown in [persist the config journal on EFS](#persist-the-config-journal-on-efs). ```bash theme={null} sed -e "s||${REG}/odigos-ecs-proxy:${TAG}|" \ -e "s||${CLUSTER}|" \ -e "s||${CENTRAL_ENDPOINT}|" \ -e "s||arn:aws:iam::${ACCOUNT}:role/odigos-ecs-exec-role|" \ -e "s||arn:aws:iam::${ACCOUNT}:role/odigos-ecs-task-role|" \ -e "s||${AWS_REGION}|" \ odigos-ecs-proxy.json > /tmp/proxy-td.json grep -c '<' /tmp/proxy-td.json # want 0 jq -r '.containerDefinitions[0].environment[] | select(.name=="ODIGOS_CENTRAL_ENDPOINT").value' /tmp/proxy-td.json # confirm this is your REAL Central host:port, not a placeholder aws ecs register-task-definition --cli-input-json file:///tmp/proxy-td.json >/dev/null && echo "proxy task definition registered" ``` ## Step 5: Create the proxy service The proxy is a **singleton** — one task per cluster. Use the tab for the address option you chose in [Step 3](#step-3-give-the-proxy-a-stable-address). ```bash theme={null} aws ecs create-service \ --cluster "$CLUSTER" \ --service-name odigos-ecs-proxy \ --task-definition odigos-ecs-proxy \ --desired-count 1 \ --launch-type EC2 \ --enable-execute-command \ --network-configuration "awsvpcConfiguration={subnets=[],securityGroups=[]}" \ --load-balancers "targetGroupArn=,containerName=odigos-ecs-proxy,containerPort=4321" ``` The pinned proxy needs three flags a normal service doesn't: 1. **A JSON placement constraint.** The AWS CLI's shorthand parser cannot handle `in [...]` expressions (it fails with `Expected ',', received: '''`). Pass pure JSON: ```bash theme={null} PLACEMENT="[{\"type\":\"memberOf\",\"expression\":\"ec2InstanceId == '${EC2_ID}'\"}]" ``` 2. **A stop-then-start deployment configuration.** ECS's default rollout starts the new task before stopping the old one. With host networking both tasks want port `4321` on the same instance, so the new task can never be placed and the deployment retries indefinitely. 3. **Availability Zone Rebalancing disabled.** AZ rebalancing defaults to `ENABLED` on new services and explicitly rejects `maximumPercent <= 100`, failing with `Availability Zone Rebalancing does not support maximumPercent <= 100%`. ```bash theme={null} aws ecs create-service \ --cluster "$CLUSTER" \ --service-name odigos-ecs-proxy \ --task-definition odigos-ecs-proxy \ --desired-count 1 \ --launch-type EC2 \ --enable-execute-command \ --placement-constraints "$PLACEMENT" \ --deployment-configuration "minimumHealthyPercent=0,maximumPercent=100" \ --availability-zone-rebalancing DISABLED ``` Wait for the proxy to come up and connect to Central before continuing: ```bash theme={null} until [ "$(aws ecs describe-services --cluster "$CLUSTER" --services odigos-ecs-proxy --query 'services[0].runningCount' --output text)" = "1" ]; do echo "waiting for proxy..."; sleep 10 done aws logs tail /odigos/ecs-proxy --since 2m --filter-pattern central # want: "Connected to central" ``` ## Step 6: Register the agent task definition Use the **same task definition** as the Agent-only guide, with two differences: the endpoint is the proxy's, and `ODIGOS_ECS_BEHIND_PROXY` is `true`. ```json theme={null} { "family": "odigos-ecs-agent", "requiresCompatibilities": ["EC2"], "networkMode": "host", "pidMode": "host", "executionRoleArn": "", "taskRoleArn": "", "volumes": [ { "name": "docker-sock", "host": { "sourcePath": "/var/run/docker.sock" } }, { "name": "docker-lib", "host": { "sourcePath": "/var/lib/docker" } }, { "name": "sys-kernel-debug", "host": { "sourcePath": "/sys/kernel/debug" } }, { "name": "odigos-config", "host": { "sourcePath": "/etc/odigos-ecs-agent" } }, { "name": "odigos-instrumentations", "host": { "sourcePath": "/var/odigos" } } ], "containerDefinitions": [ { "name": "odigos-ecs-agent", "image": "", "essential": true, "privileged": true, "cpu": 512, "memoryReservation": 512, "memory": 2048, "stopTimeout": 120, "mountPoints": [ { "sourceVolume": "docker-sock", "containerPath": "/var/run/docker.sock", "readOnly": true }, { "sourceVolume": "docker-lib", "containerPath": "/var/lib/docker", "readOnly": true }, { "sourceVolume": "sys-kernel-debug", "containerPath": "/sys/kernel/debug", "readOnly": false }, { "sourceVolume": "odigos-config", "containerPath": "/etc/odigos-ecs-agent", "readOnly": false }, { "sourceVolume": "odigos-instrumentations", "containerPath": "/var/odigos", "readOnly": false } ], "environment": [ { "name": "ODIGOS_DEFAULT_CONFIG", "value": "{\"central\":{\"enabled\":true,\"endpoint\":\"\"}}" }, { "name": "ODIGOS_ECS_BEHIND_PROXY", "value": "true" } ], "secrets": [ { "name": "ODIGOS_ONPREM_TOKEN", "valueFrom": "" } ], "ulimits": [ { "name": "memlock", "softLimit": -1, "hardLimit": -1 }, { "name": "nofile", "softLimit": 65536, "hardLimit": 65536 } ], "linuxParameters": { "initProcessEnabled": true }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/odigos/ecs-agent", "awslogs-create-group": "true", "awslogs-region": "", "awslogs-stream-prefix": "odigos-ecs-agent" } } } ] } ``` ```bash theme={null} sed -e "s||${REG}/odigos-ecs-agent:${TAG}|" \ -e "s||${PROXY_ENDPOINT}|" \ -e "s||${SECRET_ARN}|" \ -e "s||arn:aws:iam::${ACCOUNT}:role/odigos-ecs-exec-role|" \ -e "s||arn:aws:iam::${ACCOUNT}:role/odigos-ecs-task-role|" \ -e "s||${AWS_REGION}|" \ odigos-ecs-agent.json > /tmp/agent-td.json grep -c '<' /tmp/agent-td.json # want 0 jq -r '.containerDefinitions[0].environment[] | select(.name=="ODIGOS_DEFAULT_CONFIG").value' /tmp/agent-td.json | jq . # → {"central": {"enabled": true, "endpoint": "10.0.1.23:4321"}} ← the PROXY, not Central aws ecs register-task-definition --cli-input-json file:///tmp/agent-td.json >/dev/null && echo "agent task definition registered" ``` **`ODIGOS_ECS_BEHIND_PROXY=true` matters.** The agent cannot tell a proxy address from a Central address — both are just `host:port` — and the two modes report different platform names. Behind the proxy, each agent identifies itself by **node hostname** (the proxy is what reports the cluster name upstream, and its coverage check matches agent names against the instances' private DNS). If you leave it `false` here and every agent registers under the cluster name instead, which makes the coverage check report **all** instances as uncovered. ## Step 7: Create the agent daemon service ```bash theme={null} aws ecs create-service \ --cluster "$CLUSTER" \ --service-name odigos-ecs-agent \ --task-definition odigos-ecs-agent \ --scheduling-strategy DAEMON \ --launch-type EC2 \ --enable-execute-command ``` Because the proxy was already live, agents connect on their first boot. For later changes, use [`update-service`](/vmagent/ecs/maintenance#upgrade) — `create-service` is one-time. ## Step 8: Verify ```bash theme={null} aws ecs describe-services --cluster "$CLUSTER" --services odigos-ecs-agent \ --query 'services[0].{running:runningCount,desired:desiredCount,events:events[:3].message}' TASK=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name odigos-ecs-agent \ --desired-status RUNNING --query 'taskArns[0]' --output text) aws ecs execute-command --cluster "$CLUSTER" --task "$TASK" \ --container odigos-ecs-agent --interactive --command "uname -r" # want 6.1.x ``` ```bash theme={null} aws logs tail /odigos/ecs-agent --since 5m ``` Expect a connection to `` and **no** `runtime-detector stopped` errors. The proxy's `/healthz` returns the full membership view: connected agents (id, node, version, connected-at), tombstones for departed nodes, and the last coverage result. ```bash theme={null} PTASK=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name odigos-ecs-proxy \ --query 'taskArns[0]' --output text) aws ecs execute-command --cluster "$CLUSTER" --task "$PTASK" \ --container odigos-ecs-proxy --interactive --command "/bin/bash" ``` The proxy image is slim and ships **no `curl`** — use bash's `/dev/tcp` inside the session: ```bash theme={null} exec 3<>/dev/tcp/127.0.0.1/4321 printf 'GET /healthz HTTP/1.0\r\n\r\n' >&3 cat <&3 ``` The number of connected agents should match your container-instance count. Open the Central UI: **one** platform named after the ECS cluster (not one per node), with workloads aggregated across all nodes. ## Caveat: the pinned proxy address With **Option B**, if the pinned instance is replaced (instance failure, ASG replacement, manual termination) the proxy's address changes. You must: 1. Re-run [Step 3](#step-3-give-the-proxy-a-stable-address) against a live instance to get a new `EC2_ID` and `PROXY_ENDPOINT`. 2. Re-create the proxy service with the new placement constraint. 3. Re-point every agent's `ODIGOS_DEFAULT_CONFIG` at the new endpoint, register a new revision, and `update-service --force-new-deployment`. If that fragility is unacceptable, use **Option A** (internal NLB) — it survives instance and task replacement at the cost of one extra AWS resource. ## Persist the config journal on EFS By default the journal is a file on the proxy task's local storage (`/var/lib/odigos-ecs-proxy/journal.jsonl`). Task replacement loses it. Agents already persist their own config, so a healthy cluster is unaffected — the journal only closes the gap for a node that was **absent** when a change was made. If that gap matters, put the journal on EFS. `ODIGOS_PROXY_JOURNAL` is a **file path**, not a directory. Mount EFS at a directory and point the variable at a file inside it. Per-agent replay cursors are written next to that file (`journal.jsonl.cursors.json`), so the whole directory must be on the volume. Add a volume and mount to the [proxy task definition](#step-4-register-the-proxy-task-definition), and set the environment variable: ```json theme={null} { "volumes": [ { "name": "odigos-proxy-journal", "efsVolumeConfiguration": { "fileSystemId": "", "transitEncryption": "ENABLED" } } ], "containerDefinitions": [ { "mountPoints": [ { "sourceVolume": "odigos-proxy-journal", "containerPath": "/mnt/odigos-journal", "readOnly": false } ], "environment": [ { "name": "ODIGOS_PROXY_JOURNAL", "value": "/mnt/odigos-journal/journal.jsonl" } ] } ] } ``` Merge those fields into `odigos-ecs-proxy.json` — do not replace the rest of the task definition. Then: * Create the file system in the same VPC as the proxy, with a mount target in each subnet the task can land in. * Allow **NFS (TCP 2049)** from the proxy's security group to the EFS mount targets (Option A: the task's `awsvpc` security group; Option B: the container instances' security group). * Grant the **task role** `elasticfilesystem:ClientMount` and `elasticfilesystem:ClientWrite` on that file system. If you enable IAM authorization on the volume (`authorizationConfig.iam: ENABLED`), also set `elasticfilesystem:ClientRootAccess` or use an access point whose POSIX user the proxy can write as. An existing EFS file system is fine. This is optional: skip it if replaying missed mutations from Central after a proxy replacement is acceptable. ## Current limits * **Journal durability** — without EFS, the config journal is lost when the proxy task is replaced. See [persist the config journal on EFS](#persist-the-config-journal-on-efs). * **Per-node workload entries** — workload lists are concatenated across nodes, so a service with tasks on several nodes currently appears once per node. * **Single-node targeting** — profiling jobs and diagnose bundles are served by one agent; targeting a specific node is not yet supported. * **Coverage matching** — instances are correlated to agents by hostname; an instance whose private DNS cannot be resolved is conservatively reported as uncovered. ## Next steps Choose which ECS tasks to instrument — cluster-wide. Send the telemetry to your observability backend. Agent and proxy environment variables. Logs, `odictl`, `/healthz`, and common failures. # Upgrade and Uninstall on ECS Source: https://docs.odigos.io/vmagent/ecs/maintenance Roll out a new Odigos ECS Agent version and remove Odigos from an ECS cluster. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). ## Upgrade An upgrade is a new image tag plus a new task definition revision. The agent's configuration and identity live on the container instance at `/etc/odigos-ecs-agent`, so they survive the rollout. Pull the new version and push it to your ECR repository, as in [System Requirements](/vmagent/ecs/requirements#license-and-images). Use the architecture that matches your container instances. Re-run the `sed` and `register-task-definition` steps from your install guide with the new `TAG`. This creates a new revision of the same family (`odigos-ecs-agent`). ```bash theme={null} grep -c '<' /tmp/agent-td.json # want 0 aws ecs register-task-definition --cli-input-json file:///tmp/agent-td.json >/dev/null ``` ```bash theme={null} aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-agent \ --task-definition odigos-ecs-agent --force-new-deployment ``` ECS replaces the agent task on each instance. Instrumented workloads are not restarted — instrumentation is direct-attach and the agent re-attaches to the running tasks when it comes back up. Same pattern for `odigos-ecs-proxy`: ```bash theme={null} aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-proxy \ --task-definition odigos-ecs-proxy --force-new-deployment ``` With a **pinned** proxy (host networking), the service must already have `minimumHealthyPercent=0,maximumPercent=100` and `--availability-zone-rebalancing DISABLED`, or the rollout gets stuck: ECS tries to place the new task before stopping the old one, and both want port `4321` on the same instance. If the deployment is already stuck, apply the deployment configuration and stop the old task once to unblock it. Upgrade the proxy and agents to the same version. Changing an environment variable follows the exact same flow: new revision, then `update-service --force-new-deployment`. Remember that values already persisted in `config.yaml` on the instance [take precedence](/vmagent/ecs/configuration#configuration-precedence-and-persistence) over `ODIGOS_DEFAULT_CONFIG`. To move an existing Agent-only cluster onto the proxy without reinstalling, see [Switch to Multi Node](/vmagent/ecs/installation-switch-to-multi-node). ## Uninstall Removing Odigos does not touch your workloads: instrumentation is attached at runtime, so stopping the agent stops the telemetry — your tasks keep running untouched. ```bash theme={null} aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-agent --desired-count 0 aws ecs delete-service --cluster "$CLUSTER" --service odigos-ecs-agent --force # Agent + Proxy mode aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-proxy --desired-count 0 aws ecs delete-service --cluster "$CLUSTER" --service odigos-ecs-proxy --force ``` ```bash theme={null} for family in odigos-ecs-agent odigos-ecs-proxy; do for arn in $(aws ecs list-task-definitions --family-prefix "$family" --query 'taskDefinitionArns[]' --output text); do aws ecs deregister-task-definition --task-definition "$arn" >/dev/null done done ``` The agent's config directory and the synced instrumentation files are host directories and outlive the task. On each instance: ```bash theme={null} sudo rm -rf /etc/odigos-ecs-agent /var/odigos ``` `/etc/odigos-ecs-agent` holds the agent's identity, license, and configuration. Removing it makes any future install start fresh — leave it in place if you plan to reinstall and want the configuration back. ```bash theme={null} aws logs delete-log-group --log-group-name /odigos/ecs-agent aws logs delete-log-group --log-group-name /odigos/ecs-proxy aws secretsmanager delete-secret --secret-id odigos-onprem-token --force-delete-without-recovery aws iam delete-role-policy --role-name odigos-ecs-exec-role --policy-name odigos-extras aws iam detach-role-policy --role-name odigos-ecs-exec-role \ --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy aws iam delete-role --role-name odigos-ecs-exec-role aws iam delete-role-policy --role-name odigos-ecs-task-role --policy-name ssm-exec aws iam delete-role-policy --role-name odigos-ecs-task-role --policy-name odigos-coverage 2>/dev/null || true aws iam delete-role --role-name odigos-ecs-task-role aws ecr delete-repository --repository-name odigos-ecs-agent --force aws ecr delete-repository --repository-name odigos-ecs-proxy --force ``` Also delete the internal NLB, listener, and target group if you created them for the proxy. Deleting and recreating the license secret gives it a **new** random ARN suffix. The install guides' IAM policy uses a `odigos-onprem-token-*` wildcard so it keeps working — but any task definition still referencing the old ARN fails to start with `ResourceNotFoundException ... can't find the specified secret`. Your ECS cluster and container instances are untouched by all of the above. # Odigos on Amazon ECS Source: https://docs.odigos.io/vmagent/ecs/overview Instrument Amazon ECS tasks with eBPF using the Odigos ECS Agent — the VM Agent packaged as an ECS daemon service. No task restarts, no application changes. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). The **Odigos ECS Agent** is the same agent as the [VM Agent](/vmagent/overview), packaged as a container image and deployed as an ECS **daemon service** — one agent task on every EC2 container instance in the cluster. Each agent discovers the ECS tasks running on its instance through the instance's Docker daemon, instruments them **in place** with eBPF, and runs the `odigos-otelcol` collector as a supervised child process to export the telemetry. Instrumentation is **direct attach only**: the agent never restarts, recreates, or modifies your task definitions. Deploying Odigos does not redeploy your services. ## How it works ```mermaid theme={null} %%{init: {'themeVariables': {'fontSize': '22px'}}}%% flowchart LR Tasks["Your ECS tasks"] Agent["odigos-ecs-agent
+ odigos-otelcol"] Central["Odigos Central"] Agent == "eBPF attach
no restart" ==> Tasks Agent -. "direct, or via proxy
see below" .-> Central classDef centralStyle fill:#30638E,color:#fff,stroke:#123C60,stroke-width:2px classDef agentStyle fill:#C0E0DE,color:#123C60,stroke:#30638E,stroke-width:2px classDef taskStyle fill:transparent,color:#475569,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 4 4 class Central centralStyle class Agent agentStyle class Tasks taskStyle ``` One agent per container instance, repeated across every instance in the cluster. Each agent connects upstream either **directly to Central** (single node) or **through the proxy** (multi node) — see [Choose a deployment mode](#choose-a-deployment-mode) below. * **Discovery** — the agent reads the instance's Docker socket to map processes to ECS tasks, services, and task ARNs. Only ECS-managed containers are instrumented; host processes and `systemd` services on the container instance are ignored (unlike the [VM Agent](/vmagent/setup/installation) and [Docker deployment](/vmagent/setup/docker)). * **Instrumentation** — Java, Go, and C++ attach through their dynamic eBPF SDKs. Every other language is covered by [OBI](/oss/instrumentations/obi), which also attaches with no restart. Restart-based instrumentation (Python/Node.js/.NET environment injection) is intentionally not used on ECS, since ECS owns the container lifecycle. * **Export** — the collector runs inside the agent container with host networking, so instrumented tasks reach it on the instance's network namespace. * **Configuration** — the task definition only bootstraps the agent (Central or proxy endpoint, license). Sources, destinations, actions, and instrumentation rules are managed from Odigos Central or with [`odictl`](/vmagent/setup/configuration/odictl), exactly as on any VM Agent. On ECS, `odictl` runs inside the agent container over ECS Exec. ## Choose a deployment mode Both modes run the same agent daemon and instrument workloads identically. The difference is **how the cluster appears in Odigos Central** and whether you deploy one extra component. | | **Agent only** | **Agent + Proxy** | | - | - | - | | What runs | the agent daemon service | the agent daemon service **+** one `odigos-ecs-proxy` task per cluster | | Upstream connections | every agent dials Central directly | only the proxy dials Central; agents dial the proxy | | How it appears in Central | one platform **per node** | **one** platform, named after the cluster | | Cluster-wide workload list | no — per node | yes, aggregated across nodes | | Config replay for new nodes | each node is configured on its own | yes — the proxy journals config changes and replays them to nodes that join later | | Coverage check (nodes missing an agent) | — | yes, opt-in | | Extra AWS resources | none | a stable address for the proxy: none if pinned to an instance, or one internal NLB | | Guide | [Single Node](/vmagent/ecs/installation-agent-only) | [Multi Node](/vmagent/ecs/installation-with-proxy) | Select your cluster shape for the installation flow: **One container instance — the agent alone is all you need.** The agent daemon runs on the instance and connects straight to Odigos Central, which shows the cluster as a single platform named after it. No proxy, no load balancer, no extra AWS resources. AWS Secrets Manager (or a plain env var for a test cluster). Task execution role, plus a task role for `odictl` over ECS Exec. Privileged, `pidMode: host`, host networking, with your Central endpoint in `ODIGOS_DEFAULT_CONFIG`. One agent task per container instance, now and for instances that join later. Kernel 5.10+, agent connected, platform visible in Central. Follow the full Agent-only flow. This flow also works unchanged on a **multi-node** cluster — you simply get one platform per node instead of one per cluster. Choose it whenever you don't need cluster-wide aggregation. **Several container instances that should behave as one cluster.** The agent daemon still runs on every instance, but the agents dial a single `odigos-ecs-proxy` task instead of Central. The proxy is the cluster's only upstream connection, so Central shows **one** platform with aggregated workloads, cluster-wide configuration, config replay for nodes that join later, and an optional coverage check. Same two roles as the single-node flow, plus optional coverage permissions for the proxy. Pin it to one instance (no extra AWS resources) or front it with an internal NLB (survives instance replacement). One task per cluster, listening on TCP 4321. Endpoint set to the proxy, plus `ODIGOS_ECS_BEHIND_PROXY=true`. Agents connect to the already-live proxy on first boot. Agents connected, proxy `/healthz` lists every node, one platform in Central. Follow the full Agent + Proxy flow. **Not sure? Start single node.** The flow is identical on a 1-node or a 50-node cluster, and moving to Agent + Proxy later is just re-pointing the agents' endpoint at the proxy and rolling the daemon service — no reinstall. See [Switch to Multi Node](/vmagent/ecs/installation-switch-to-multi-node). ## What the proxy adds `odigos-ecs-proxy` is the ECS counterpart of the Kubernetes `central-proxy`: one task per cluster that represents the whole cluster to Odigos Central as a single compute platform. ```mermaid theme={null} %%{init: {'themeVariables': {'fontSize': '22px'}}}%% flowchart LR agents["odigos-ecs-agent
one per node"] proxy["odigos-ecs-proxy
one per cluster"] central["Odigos Central"] agents == "unchanged
protocol" ==> proxy == "one connection
per cluster" ==> central classDef centralStyle fill:#30638E,color:#fff,stroke:#123C60,stroke-width:2px classDef proxyStyle fill:#123C60,color:#fff,stroke:#30638E,stroke-width:2px classDef agentStyle fill:#C0E0DE,color:#123C60,stroke:#30638E,stroke-width:2px class central centralStyle class proxy proxyStyle class agents agentStyle ``` * **Request routing** — configuration queries go to any one healthy agent (the config is identical on all of them); live state such as workload reports fans out to every agent and is merged; configuration changes are broadcast to all agents. * **Config journal** — configuration changes are recorded, so a node that joins the cluster later (or a task that migrates to a fresh instance) converges to the cluster configuration automatically. * **Membership and coverage** — `GET /healthz` lists the connected agents, departed nodes, and — when [coverage checking](/vmagent/ecs/configuration#proxy-environment-variables) is enabled — ACTIVE container instances that have no agent connected. The proxy is unprivileged: no host PID, no Docker socket, no host mounts. ## Not supported: Fargate The ECS Agent requires the **EC2 launch type**. It needs host PID access, the instance's Docker socket, and the ability to load eBPF programs — none of which exist on Fargate, where AWS controls the host. There is no Fargate deployment mode. ## Next steps Launch type, kernel and AMI requirements, IAM, and images. Agent only — daemon service, direct to Central. Agent + Proxy — one platform per cluster, with aggregation and config replay. Every environment variable, platform naming, and where configuration is persisted. # System Requirements Source: https://docs.odigos.io/vmagent/ecs/requirements What your ECS cluster, container instances, IAM roles, and registry need before installing the Odigos ECS Agent. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). Review this page before installing. The launch type and kernel requirements are hard requirements — the agent cannot work around them. ## Cluster and launch type * An existing **Amazon ECS cluster** with **EC2 container instances**. * **EC2 launch type only.** [Fargate is not supported](/vmagent/ecs/overview#not-supported-fargate) — the agent needs host PID, the instance's Docker socket, and eBPF. * Ability to run a **privileged task with `pidMode: host`** and host bind mounts on the container instances (see [required permissions](#required-permissions)). ## Required permissions Each elevated setting in the agent task definition maps to a specific capability. These are hard requirements. | Setting | Reason | | - | - | | `pidMode: host` | Classifying and attaching to other tasks' processes via `/proc/`. | | `privileged: true` | eBPF program loading, perf events, and writing instrumentation files into target containers via `/proc//root`. | | `networkMode: host` | The collector and OpAMP server listen on the instance's network namespace, reachable by instrumented workloads on localhost or the instance IP. | | `/var/run/docker.sock` (read-only) | Container metadata: mapping PIDs to ECS tasks, service names, task ARNs. | | `/var/lib/docker` (read-only) | Resolving executable paths through overlay2 for language detection. | | `/var/odigos` (read-write) | Instrumentation files synced to the instance on startup; the source seeded into workload containers when attaching Java. | | `/etc/odigos-ecs-agent` (read-write) | Agent configuration and identity, persisted across task restarts and upgrades. | | `/sys/kernel/debug` | kprobes and tracepoints used by the eBPF instrumentations. | | `ulimits.memlock: -1` | eBPF maps on kernels that account them against `RLIMIT_MEMLOCK`. | | `linuxParameters.initProcessEnabled` | Reaps child processes left by attach tooling. | The **proxy** needs none of these. ## Container instance kernel and AMI The container instance kernel must be **5.10 or newer with BTF** (`/sys/kernel/btf/vmlinux` present). The agent's process detector and eBPF probes require BTF/CO-RE and a modern verifier. **Amazon Linux 2's stock kernel (4.14) does not work.** On it, the agent logs `runtime-detector stopped: ... verifier error ... permission denied`, never discovers processes, and nothing is instrumented. * **Recommended:** the **Amazon Linux 2023** ECS-optimized AMI (kernel 6.1). * **Minimum:** the Amazon Linux 2 **kernel-5.10** ECS-optimized AMI. * Both `amd64` and `arm64` (Graviton) instances are supported — pull the image for the matching architecture. Run this preflight check on a container instance before deploying: ```bash theme={null} uname -r # want 5.10+ (6.1.x on AL2023) — 4.14.x will fail ls /sys/kernel/btf/vmlinux # must exist ``` ## Resources per instance The agent task reserves modest resources on each container instance. The defaults in the reference task definition are a good starting point: | Setting | Default | Notes | | - | - | - | | `cpu` | `512` | CPU units for the agent and its collector child process. | | `memoryReservation` | `512` | Soft limit. | | `memory` | `2048` | Hard limit. Raise it on instances with many high-throughput tasks. | Make sure your instances have that headroom free, otherwise the daemon task cannot be placed. ## License and images * An **Odigos Pro license token** for the VM Agent, delivered to the task as `ODIGOS_ONPREM_TOKEN` (recommended: from AWS Secrets Manager). * The **`odigos-ecs-agent`** container image, and — for [Agent + Proxy](/vmagent/ecs/installation-with-proxy) — the **`odigos-ecs-proxy`** image. Both are published multi-arch (`amd64`, `arm64`). Obtain the images and registry credentials from your Odigos support team. * We recommend **mirroring both images into your own Amazon ECR repository** in the cluster's account and region. ECS then pulls them with the task execution role and needs no external registry credentials, and pulls stay in-region: ```bash theme={null} export REGION= export ACCOUNT=$(aws sts get-caller-identity --query Account --output text) export REG=${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com export TAG= aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin "$REG" for r in odigos-ecs-agent odigos-ecs-proxy; do aws ecr create-repository --repository-name "$r" --region "$REGION" >/dev/null 2>&1 || true docker pull --platform linux/amd64 "/${r}:${TAG}" # or linux/arm64 docker tag "/${r}:${TAG}" "${REG}/${r}:${TAG}" docker push "${REG}/${r}:${TAG}" done ``` Pull the architecture that matches your container instances. The proxy image is only needed for the Agent + Proxy mode. ## IAM roles Two roles, both trusting `ecs-tasks.amazonaws.com`: Used by ECS to pull the image, write logs, and read the license secret. * Managed policy `AmazonECSTaskExecutionRolePolicy` * `logs:CreateLogGroup` (the reference task definitions set `awslogs-create-group: "true"`) * `secretsmanager:GetSecretValue` on the license secret, when the token comes from Secrets Manager Used by the running task itself. * `ssmmessages:CreateControlChannel`, `ssmmessages:CreateDataChannel`, `ssmmessages:OpenControlChannel`, `ssmmessages:OpenDataChannel` — required for [ECS Exec](/vmagent/ecs/troubleshooting#inspect-a-running-agent-with-odictl), which is how you run `odictl` against a running agent. * **Proxy only, for the optional coverage check:** `ecs:ListContainerInstances`, `ecs:DescribeContainerInstances`, `ec2:DescribeInstances`. The install guides create both roles with copy-pasteable commands. ## Network * Outbound connectivity from the container instances to your **Odigos Central** endpoint (`host:port`; `:443` implies `wss`). The connection is **outbound-only** — a Central instance in another AWS account or a public endpoint works with no inbound rules on the cluster. * **Agent + Proxy mode only:** the instances' security group must allow TCP **4321** to the proxy (instance-to-instance when the proxy is pinned to an instance, or from the internal NLB). ## Local tooling For the install and verification steps: ```bash theme={null} # AWS CLI v2, jq, and the Session Manager plugin (for ECS Exec / odictl) brew install jq brew install --cask session-manager-plugin ``` **zsh users:** always brace variables when a `:` follows them — write `"${REG}/odigos-ecs-agent:${TAG}"`, not `"$REG/odigos-ecs-agent:$TAG"`, and `"arn:aws:iam::${ACCOUNT}:role/..."`, not `"arn:aws:iam::$ACCOUNT:role/..."`. Unbraced `$VAR:letter` is parsed by zsh as a history modifier and **silently corrupts the value** — producing image names like `odigos-ecs-agentatest` and truncated role ARNs. ## Next steps Daemon service, each node connects to Central directly. One aggregated platform per cluster. # Export profiles to Pyroscope Source: https://docs.odigos.io/vmagent/instrumentations/profiling/export-pyroscope Configure a Pyroscope destination on the VM Agent and view OTLP Profiles in the Pyroscope UI. The VM Agent exports continuous profiles to external backends using the same **destinations** model as traces. For profiling, add a destination whose **signals** include `PROFILES`. Any backend which accepts OTLP Profiles is supported. ## Pyroscope destination fields See the [Pyroscope backend reference](/enterprise/backends/pyroscope) for all fields. Defaults for a local Pyroscope container on the same host: | YAML / env key | UI label | Example | Notes | | - | - | - | - | | `PYROSCOPE_URL` | Pyroscope Endpoint | `localhost:4040` | Distributor `host:port`. Odigos appends `/v1development/profiles` — do **not** include any path suffix. | | `PYROSCOPE_TLS_ENABLED` | Enable TLS | `false` | Set to `true` when Pyroscope is fronted by TLS. | | `signals` | Profiles checkbox | `[PROFILES]` | Required — Pyroscope only emits profiles. | ## Add destination with odictl ```shell theme={null} odictl ``` Press `d` to focus **Destinations**, then `Enter` (or click **+ Add Destination**) and search for **profiles** to filter the list to backends that support the `PROFILES` signal. odictl destination type selector filtered by profiles

odictl destination type selector filtered by profiles

Highlight **Pyroscope** and press `Enter`. Fill in **Pyroscope Endpoint** and toggle **Enable TLS** if needed. Leave **Profiles: Yes**. odictl Pyroscope destination form with localhost:4040 and TLS disabled

odictl Pyroscope destination form with localhost:4040 and TLS disabled

Press `Ctrl+S` (or focus **Apply** with `Tab` and press `Enter`) to save. ```shell theme={null} sudo journalctl -u odigos-vmagent | grep 'Destination signals configured' ``` Look for a line containing `signals=[PROFILES]` for the new destination. ## Add destination with YAML Create a file under `/var/lib/odigos-agent/config/destinations.d/`, for example `pyroscope.yaml`: ```yaml theme={null} - name: pyroscope type: pyroscope signals: - PROFILES config: PYROSCOPE_URL: "localhost:4040" PYROSCOPE_TLS_ENABLED: "false" ``` The destination controller picks the file up automatically. ## Run Pyroscope on the host Quick local run, useful for testing: ```shell theme={null} docker run -d --name pyroscope -p 4040:4040 grafana/pyroscope:latest ``` If Pyroscope listens only on localhost, port-forward from your laptop: ```shell theme={null} ssh -L 4040:localhost:4040 ``` ## View profiles Open Pyroscope at `http://localhost:4040`. Pick profile type **`process_cpu`** and filter by **`service.name`** matching one of your enabled sources. Pyroscope UI showing a process_cpu flame graph for odigos-demo-coupon

Pyroscope UI showing a process_cpu flame graph for odigos-demo-coupon

## Other OTLP Profiles backends Any backend that accepts **OTLP Profiles** can be used with destination type `otlp` (gRPC) or `otlphttp` and `signals: [PROFILES]`. Pyroscope is documented here because it is easy to setup self-hosted target for VM deployments, the configuration shape for other backends is identical aside from their own endpoint fields. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Enable continuous profiling Source: https://docs.odigos.io/vmagent/instrumentations/profiling/getting-started Enable the VM Agent eBPF profiling pipeline, add sources, and export profiles to Pyroscope. This guide walks you through enabling the eBPF profiling pipeline on a host, and adding a destination that accepts OTLP profiles. ## Step 1: Enable the profiling pipeline Turn on the eBPF profiles pipeline on **odigos-otelcol**. You can use `odictl` or edit `/var/lib/odigos-agent/config/config.yaml` directly. ```shell theme={null} odictl ``` Press c to open the Configuration Editor, then navigate to **Profiling**. Set **profiling.enabled** to `true` and press Ctrl+S to save. The collector reloads automatically. Add to `/var/lib/odigos-agent/config/config.yaml`: ```yaml theme={null} profiling: enabled: true ``` ## Step 2: Enable sources for the workloads you want to profile Profiling only retains samples for **enabled sources**. Open `odictl`, focus the **Sources** pane (press `o`), and instrument the services you want sampled. The screenshot below shows the demo workloads enabled (`Instrumented`) alongside other host processes that stay untouched (`Not Instrumented`). Sources panel in odictl showing the demo services instrumented

Sources panel in odictl showing the demo services instrumented

See [Sources](../../setup/configuration/add-sources) for the full walkthrough (TUI and YAML). Toggling a source on or off takes effect within seconds — no vm-agent restart is needed. The collector keeps streaming profiles, but only samples for currently-enabled PIDs are exported. ## Step 3: Add a destination that accepts PROFILES Open **Destinations → + Add Destination** and search for **profiles**. Any destination whose **Signals** column includes `PROFILES` or they accept OTLP Profiles signal works. odictl destination type selector filtered by profiles

As an example for step-by-step Pyroscope configuration, jump to [Export to Pyroscope](./export-pyroscope). For any other OTLP Profiles backend (e.g. `otlp` or `otlphttp`), follow the standard [Add destinations](../../setup/configuration/add-destinations). ## Step 4: View profiles in the destination Open your destination's UI and you should be able to access the application profiles collected. eg in Pyroscope, select profile type **`process_cpu`** and filter by **`service.name`** values that match your sources. See [Export to Pyroscope → View profiles](./export-pyroscope#view-profiles) for detailed example. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Continuous profiling overview Source: https://docs.odigos.io/vmagent/instrumentations/profiling/overview How continuous CPU profiling works on the Odigos VM Agent: eBPF collection, service attribution, and export to Pyroscope. ## See it in action! The clip below shows the full path on a host: enabling `profiling.enabled` via `odictl`, adding a Pyroscope destination, and the resulting flame graph in the Pyroscope UI. End-to-end VM Agent profiling demo: enable profiling, add destination, view flame graph

End-to-end VM Agent profiling demo: enable profiling, add destination, view flame graph

## What is continuous CPU profiling? CPU profiling samples what your processes are doing on the CPU. Over time, those samples form a **flame graph** that shows which functions and libraries consume the most CPU time. Unlike traces, which tell you *that* a request was slow, profiles show *where* CPU time was spent. Use profiles after traces (or metrics) point you at a hot service. ## How it works on the VM Agent 1. You enable the profiling pipeline in agent config (`profiling.enabled: true`). 2. **odigos-otelcol** runs the OpenTelemetry [eBPF profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) receiver on the host. 3. Profiles are exported as using **OTLP Profiles** protocol to a destination which supports OTLP profiles. ## Prerequisites * Linux host meeting [VM Agent system requirements](../../setup/system-requirements) (eBPF-capable kernel). * VM Agent installed and running ([Installation](../../setup/installation)). * At least one [source](../../setup/configuration/add-sources) enabled for the workloads you want to profile. * A destination that accepts the **PROFILES** signal (see [Export to Pyroscope](./export-pyroscope) for the recommended setup). ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # View profiles in odictl Source: https://docs.odigos.io/vmagent/instrumentations/profiling/view-in-odictl Open, navigate, and inspect continuous CPU profiles directly in the odictl terminal interface. `odictl` includes a built-in profiling view so you can inspect continuous CPU profiles directly in the terminal—without opening an external backend. Use it to find the functions consuming the most CPU in a service, and to drive [profile-driven custom instrumentation](../../setup/configuration/instrumentation-rules/instrument-from-profiles). ## Prerequisites * VM Agent installed and running ([Installation](../../setup/installation)). * At least one [source](../../setup/configuration/add-sources) enabled for the workloads you want to profile. * The profiling pipeline enabled. You can enable it ahead of time (see [Getting started](./getting-started) or the Configuration Editor's **Profiling** section in [odictl](../../setup/configuration/odictl)), or directly from the profiling view as described below. ## Open the profiling view From the main `odictl` dashboard, press Shift+P to open the profiling view. If profiling is not yet enabled, `odictl` shows an **Odigos profiling is not enabled** prompt. Press Enter to enable it. odictl profiling view prompting to enable profiling

odictl profiling view prompting to enable profiling

You can also enable profiling without opening this view by setting **profiling.enabled** in the Configuration Editor. See [Getting started](./getting-started) for the full setup. After profiling is enabled, the agent collects samples before any data appears. This can take up to a minute. The first load may be empty while samples are gathered. Wait a few moments and the services and functions populate automatically—no restart is needed. odictl profiling view collecting profile data

## Navigating the profiling view The profiling view has three panes: a **Services** list on the left, a **Filter modules** bar across the top, and a **Functions** list. Use `Tab` to move between them and the arrow keys to move within the focused pane. | Key | Action | | - | - | | `Shift+P` | Open the profiling view from the dashboard | | `Tab` | Cycle between the Services, Filter modules, and Functions panes | | `↑` / `↓` | Move between items in the focused pane | | `Space` | Toggle the highlighted module in the Filter modules pane | | `a` | Show app-only functions | | `i` | Instrument the highlighted function (or remove its instrumentation) | | `Esc` | Return to the main dashboard | Mouse selection is not supported in the profiling view. Use the arrow keys to select items and `Tab` to switch panes. ### Services pane The left pane lists the services that have profile data. Use the arrow keys to highlight the service you want to inspect. ### Filter modules Press `Tab` to focus the **Filter modules** bar at the top. It controls which modules and sample types appear in the Functions list: * **Module** — selects which modules to show in the profile. This is set based on the language of the selected service (for example, Java). You can also choose `vmlinux` or other libraries based on the runtime/language. Press `Space` to toggle the highlighted module. * **Type** — defaults to `app`. You can also select `library` or `kernel`, or press `a` to show app-only functions. odictl profiling view module and type filters

odictl profiling view module and type filters

### Functions pane Press `Tab` again to focus the list of profiled functions for the selected service and filter. Use the arrow keys to highlight a function. From here, press `i` to instrument the highlighted function (the panel then shows **\[i] remove** to undo it)—see [Instrument from Profiles](../../setup/configuration/instrumentation-rules/instrument-from-profiles) for the full workflow. ## Getting Help If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw) # Overview Source: https://docs.odigos.io/vmagent/overview The Odigos VM Agent instruments Linux processes and systemd services using eBPF. Because instrumentation runs in the kernel, impact on applications is negligible. Traces, logs, and profiles are exported to any of the supported destinations of your choice. The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details, [contact us](https://odigos.io/). ## Features * Instrument [systemd](https://systemd.io/) services and Linux processes * Automatic OpenTelemetry instrumentation via eBPF with no code changes * Distributed traces with seamless context propagation * Collect application logs from instrumented sources (process, Docker, systemd) and export them when a destination accepts the `LOGS` signal * Continuous CPU profiling (eBPF) with export to [Pyroscope](/vmagent/instrumentations/profiling/export-pyroscope) or other OTLP Profiles backends ## Deployment options The VM Agent can be deployed in three ways. All of them instrument workloads with the same eBPF agent and are configured the same way (via Odigos Central or `odictl`): * **Linux package** — install as an RPM/deb system package managed by `systemd`. This is the standard option for a normal Linux host. See [Installation](/vmagent/setup/installation). * **Docker container** — run the agent as a privileged container. Use this for hosts where installing a system package is not an option, such as immutable hosts or Compose-managed fleets. See [Docker Deployment](/vmagent/setup/docker). * **Amazon ECS daemon service** — run the agent as an ECS daemon so every EC2 container instance instruments the ECS tasks on it, with no task restarts. See [Odigos on Amazon ECS](/vmagent/ecs/overview). ## Key concepts * **Sources**: The systemd services or Linux processes you instrument with the VM Agent. See [Add Sources](/vmagent/setup/configuration/add-sources) to get started. * **Destinations**: The backends that receive your telemetry data. See [Add Destinations](/vmagent/setup/configuration/add-destinations) to get started. * **Actions**: Processors that transform, filter, or enrich telemetry before it is sent to destinations. See [Actions overview](/vmagent/setup/configuration/actions/overview) to add and manage actions. * **Instrumentation rules**: Rules that control how telemetry is recorded from your sources—for example, which attributes, headers, or payloads are collected and how custom instrumentations behave. See [Instrumentation rules overview](/vmagent/setup/configuration/instrumentation-rules/overview) to add and manage rules. # Overview Source: https://docs.odigos.io/vmagent/setup/configuration/actions/overview What Odigos VM Agent actions are, their benefits, and the three types: processor addition, span metrics, and URL templatization. Actions modify OpenTelemetry Signals produced by your sources before it is exported to destinations. They run in the VM Agent, so processing is decoupled from your applications and applies consistently to all signals from the agent. ## Benefits * **Language and source agnostic** — Actions run in the OpenTelemetry pipeline and can transform, filter, or enrich OpenTelemetry signals regardless of the instrumented process or language. * **Lower impact on workloads** — Processing happens in the agent instead of inside your services, so CPU and memory stay on the agent. * **Control cost and volume** — You can reduce or shape the data sent to backends (e.g., sampling, dropping attributes, aggregating spans into metrics) to limit export volume and cost. * **Reusable building blocks** — The VM Agent supports the OpenTelemetry Collector processors defined in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61); you can combine and configure those processors to match your needs. ## Action types The VM Agent supports three kinds of actions. Each type has a dedicated page with configuration steps for [odictl](/vmagent/setup/configuration/odictl) and YAML: ### Processor addition Add an OpenTelemetry Collector processor to transform, filter, or enrich signals (e.g., add or delete attributes, batch, sample, mask PII). Processors defined in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported; for each processor's configuration options, see the [OpenTelemetry Collector Contrib documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor). See [Processor addition](/vmagent/setup/configuration/actions/processor-addition) for configuration steps. ### Span metrics Convert tracing spans into aggregated metrics (e.g., request counts, latency histograms) to reduce volume and simplify dashboards and alerts. See [Span metrics](/vmagent/setup/configuration/actions/span-metrics) for configuration steps. ### URL templatization Replace variable parts of URLs with parameter placeholders (e.g., `/users/123` → `/users/{id}`) before recording in signals to control cardinality and group traffic by endpoint pattern. See [URL templatization](/vmagent/setup/configuration/actions/url-templatization) for configuration steps. # Processor Addition Source: https://docs.odigos.io/vmagent/setup/configuration/actions/processor-addition Add OpenTelemetry Collector processors to transform, filter, or enrich OpenTelemetry signals in the VM Agent [Processor addition](/vmagent/setup/configuration/actions/overview#processor-addition) lets you add an OpenTelemetry Collector processor to transform, filter, or enrich signals (e.g., add or delete attributes, batch, sample, mask PII). Processors defined in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported. There are two ways to add a processor addition action: using `odictl` or using YAML files. ```shell theme={null} odictl ``` Use `Tab` to focus on the Actions pane or press `a`, then press `Enter` or click `+ Add Action` with your mouse. Select Actions menu

Use the arrow keys to move through the list of action types. When **Processor addition** is highlighted, press `Enter`. Select Processor addition action

1. Press `i` to enter INSERT mode. 2. Set the processor type and options. Only processors listed in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported. Set the processor name (e.g., `attributes`, `batch`) and fill in any processor-specific configuration. 3. When finished, press `Esc`, then type `:wq` to save and exit. Configure Processor addition

To cancel creating the action, press `Esc` if you are in INSERT mode, then type `:q!` to exit without saving. Verify the action has been created

```shell theme={null} cd /var/lib/odigos-agent/config/actions.d ``` Create a YAML file for your action using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi). ```shell theme={null} sudo vi processor-addition.yaml ``` Add a processor addition action with the desired processor type and config. Processors listed in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported. Set `config.type` to one of those processor names (e.g., `attributes`, `batch`), and use `config.processorConfig` for processor-specific options. See the [OpenTelemetry Collector Contrib documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor) for each processor's configuration. For example: ```yaml theme={null} name: processor-addition actionName: processor-addition notes: "" signals: - TRACES # pipelines: [] # optional: target specific pipelines (e.g. traces/otlp-datadog) orderHint: 0 type: processorAddition config: type: "" processorName: "" processorConfig: {} ``` Replace `config.type` with the processor type (e.g., `attributes`, `batch`), set `config.processorName` if desired, and fill `config.processorConfig` with the processor's configuration. ```shell theme={null} :wq! ``` ```shell theme={null} sudo journalctl -u odigos-vmagent | grep 'Action created' ``` ``` Mar 11 21:59:43 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T21:59:43.580Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/tower/mutations/create_action_handler.go:41 msg="Action created" name=processor-addition ``` # Span Metrics Source: https://docs.odigos.io/vmagent/setup/configuration/actions/span-metrics Convert tracing spans into aggregated metrics in the Odigos VM Agent [Span metrics](/vmagent/setup/configuration/actions/overview#span-metrics) is a processor that turns tracing spans into aggregated metrics (e.g., request counts, latency histograms), reducing export volume and making it easier to build dashboards and alerts. There are two ways to add a span metrics action: using `odictl` or using YAML files. ```shell theme={null} odictl ``` Use `Tab` to focus on the Actions pane or press `a`, then press `Enter` or click `+ Add Action` with your mouse. Select Actions menu

Use the arrow keys to move through the list of action types. When **Span metrics** is highlighted, press `Enter`. Select Span metrics action

1. Press `i` to enter INSERT mode. 2. Adjust options such as `interval`, `metricsExpiration`, and histogram settings as needed. 3. When finished, press `Esc`, then type `:wq` to save and exit. Configure Span metrics action

To cancel creating the action, press `Esc` if you are in INSERT mode, then type `:q!` to exit without saving. Verify the action has been created

```shell theme={null} cd /var/lib/odigos-agent/config/actions.d ``` Create a YAML file for your action using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi). ```shell theme={null} sudo vi span-metrics.yaml ``` Add a span metrics action. You can adjust `interval`, `metricsExpiration`, `additionalDimensions`, and other resource attributes to match your needs. For example: ```yaml theme={null} name: span-metrics actionName: span-metrics signals: - TRACES - METRICS # pipelines: [] # optional: target specific pipelines type: spanMetrics config: disabled: false interval: "60s" metricsExpiration: "5m" additionalDimensions: [] histogramDisabled: false explicitHistogramBuckets: [] resourceMetricsKeyAttributes: [] includedProcessInDimensions: false excludedResourceAttributes: [] ``` ```shell theme={null} :wq! ``` ```shell theme={null} sudo journalctl -u odigos-vmagent | grep 'Action created' ``` ``` Mar 11 21:59:43 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T21:59:43.580Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/tower/mutations/create_action_handler.go:41 msg="Action created" name=span-metrics ``` # URL Templatization Source: https://docs.odigos.io/vmagent/setup/configuration/actions/url-templatization Replace dynamic URL parts with parameter placeholders in VM Agent OpenTelemetry signals [URL templatization](/vmagent/setup/configuration/actions/overview#url-templatization) replaces variable parts of URLs with parameter placeholders before the URL is recorded in OpenTelemetry signals (e.g., `/users/123` → `/users/{id}`), keeping cardinality under control and making it easier to group traffic by endpoint pattern. There are two ways to add a URL templatization action: using `odictl` or using YAML files. ```shell theme={null} odictl ``` Use `Tab` to focus on the Actions pane or press `a`, then press `Enter` or click `+ Add Action` with your mouse. Select Actions menu

Use the arrow keys to move through the list of action types. When **URL templatization** is highlighted, press `Enter`. Select URL Templatization action

1. Press `i` to enter INSERT mode. 2. Add one or more templatization rules with a `template` (e.g., `/api/v1/resource/{id}`) and optional `examples`. 3. When finished, press `Esc`, then type `:wq` to save and exit. Configure URL templatization

To cancel creating the action, press `Esc` if you are in INSERT mode, then type `:q!` to exit without saving. Verify the action has been created

```shell theme={null} cd /var/lib/odigos-agent/config/actions.d ``` Create a YAML file for your action using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi). ```shell theme={null} sudo vi url-templatization.yaml ``` Add a URL templatization action with one or more rule groups. Each rule has a `template` (e.g., `/api/v1/resource/{id}`), optional `examples` of matching URLs, and optional `notes`. For example: ```yaml theme={null} name: url-templatization actionName: url-templatization signals: - TRACES # pipelines: [] # optional: target specific pipelines (e.g. traces/otlp-datadog) type: urlTemplatization config: templatizationRulesGroups: - notes: "" templatizationRules: - template: "/api/v1/resource/{id}" examples: - "/api/v1/resource/123" notes: "" ``` ```shell theme={null} :wq! ``` ```shell theme={null} sudo journalctl -u odigos-vmagent | grep 'Action created' ``` ``` Mar 11 21:59:43 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T21:59:43.580Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/tower/mutations/create_action_handler.go:41 msg="Action created" name=url-templatization ``` # Destinations Source: https://docs.odigos.io/vmagent/setup/configuration/add-destinations Add destinations to the Odigos VM Agent so telemetry can be sent to your backends. Enable the signals you want to export on each destination (`TRACES`, `LOGS`, `PROFILES`). The agent turns on collection for those signals automatically—for example, configuring `LOGS` starts [log collection](/vmagent/setup/configuration/overview#log-collection) for instrumented sources. There are two ways to add [destinations](/vmagent/overview#key-concepts) to the Odigos VM Agent: use `odictl` or use YAML files. ```shell theme={null} odictl ``` Use `Tab` to focus on the Destinations pane or press `d`, then press `Enter` or click `+ Add Destination` with your mouse. Select Destination section of the odictl interface

Select Destination section of the odictl interface

Type the name of the destination in the search bar at the top, and/or press `Tab` and scroll through the list of destinations. Once your destination is highlighted, press `Enter`. Select Destination

Set each field according to your destination type. See [Destinations](/enterprise/backends-overview) for field details and requirements. Configure Destination

Fill in the required fields, then press `Ctrl+S` to save your destination. Your destination appears in the `Destinations` section in `odictl`. Verify Destination

```shell theme={null} cd /var/lib/odigos-agent/config/destinations.d ``` Create a YAML file for your destination's configuration using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi). ```shell theme={null} sudo vi example.yaml ``` Set each field according to your destination type. See [Destinations](/enterprise/backends-overview) for field details and requirements. For example: ```yaml theme={null} - name: otlphttp type: otlphttp config: OTLP_HTTP_COMPRESSION: none OTLP_HTTP_ENDPOINT: https://otlp-example.com OTLP_HTTP_INSECURE_SKIP_VERIFY: "false" OTLP_HTTP_OAUTH2_ENABLED: "false" OTLP_HTTP_TLS_ENABLED: "false" signals: - TRACES - LOGS ``` The above is an example of an [OTLP http](/enterprise/backends-overview) destination. For continuous CPU profiling, add a [Pyroscope](/enterprise/backends/pyroscope) destination with `signals: [PROFILES]`. See [Export to Pyroscope](/vmagent/instrumentations/profiling/export-pyroscope). You can define multiple destinations in a single YAML file, and you can organize destinations across multiple YAML files. ```shell theme={null} :wq! ``` ```shell theme={null} sudo journalctl -u odigos-vmagent | grep 'Destination signals configured' ``` ``` Mar 11 20:50:18 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T20:50:18.401Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/destination/controller.go:177 msg="Destination signals configured" destination=otlphttp signals=[TRACES LOGS] ``` # Sources Source: https://docs.odigos.io/vmagent/setup/configuration/add-sources Add or instrument sources in the Odigos VM Agent so the agent can collect their telemetry. There are two ways to add [sources](/vmagent/overview#key-concepts) to the Odigos VM Agent: use `odictl` or use YAML files. It is recommended to [add a destination](/vmagent/setup/configuration/add-destinations) before adding a source. **Instrumenting a source restarts its underlying service or process.** Regardless of whether you use `odictl` or YAML, the VM Agent restarts each service, process, or container when it applies instrumentation, so expect a brief interruption. **The only exception is Go and Java processes running directly on the host (not in a container).** These are instrumented in place and are **not** restarted. A **non-root** agent requires additional permissions to restart services, containers, and processes. See [Additional permissions](/vmagent/setup/permissions#additional-permissions). ```shell theme={null} odictl ``` Use `Tab` to focus on the Sources pane or press `o`, then press `Enter` or click `+ Add Source` with your mouse. Select Sources pane of the odictl interface

Select Sources pane of the odictl interface

Once the source list opens, choose how you want to instrument. Select a **single source** to review and adjust its properties before instrumenting, or select **multiple sources** to instrument them all at once with default settings. Click in the search bar at the top and type the name of the Linux process, and/or press `Tab` and scroll through the list of sources. Once your source is highlighted, press `Enter`. Select Source

Review and adjust the properties if needed: The identity the VM Agent uses to match this source to the discovered process, service, or container. It is auto-populated at discovery time and is required — it also drives source lookup and log collection, so avoid changing it (contact support if you must). The OpenTelemetry `service.name` reported for this source's telemetry — the service that its spans, metrics, and logs are attributed to in your backend (for example, the service node in a service map). It is auto-populated from `Name` at discovery time, but you can edit it to report telemetry under a different service name. If left empty, it defaults to `Name`. This does not change process matching or the individual span (operation) names. The process type: either `systemd` for services managed by systemd, `process` for standalone Linux processes, or `docker` for docker containers. This field is auto-populated at discovery time — contact support before changing it. Controls whether the VM Agent instruments this source. Check to enable instrumentation, or uncheck to pause it without removing the source. The programming language of the process as detected by the VM Agent. This field is auto-populated at discovery time — contact support before changing it. The version of the runtime language as detected by the VM Agent. This field is auto-populated at discovery time — contact support before changing it.
Instrument Source

Press `Ctrl+S` to save and instrument your source. Instrumentation can take a few moments to complete. Once it is applied, the source appears in the `Sources` list with a green `Health` status. Verify Instrumentation

Move through the list with the arrow keys or `Tab`, and press `Space` to select each process, service, or container you want to instrument. Repeat until all the sources you want are selected. Select Multiple Sources

Press `Enter` to instrument every selected source at once. This flow does not show the properties dialog — each source is instrumented with its default settings, which is what most deployments want. Instrumentation can take a few moments to complete. Once it is applied, the sources appear in the `Sources` list with a green `Health` status. Verify Instrumentation

```shell theme={null} cd /var/lib/odigos-agent/config/sources.d ``` Create a YAML file for your source's configuration using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi). ```shell theme={null} sudo vi example.yaml ``` Define each service or process you want to instrument using the properties below. The identity the VM Agent uses to match this source to the discovered process, service, or container. Required, and also used for source lookup and log collection, so it should match the detected workload. Controls whether the VM Agent instruments this source. Set to `true` to enable instrumentation, or to `false` to pause it without removing the source. Optional. The OpenTelemetry `service.name` reported for this source's telemetry — the service that its spans, metrics, and logs are attributed to in your backend. If omitted, it defaults to `name`. This does not change process matching or the individual span (operation) names. The process type: either `systemd` for services managed by systemd, `process` for standalone Linux processes, or `docker` for Docker containers. Optional runtime configuration (e.g., language and version). Auto-populated at discovery time — contact support before changing this field. For example: ```yaml theme={null} # serviceName omitted: telemetry is reported under service.name = odigos-demo-frontend (the name) - name: odigos-demo-frontend enabled: true kind: systemd config: null # serviceName set: telemetry is reported under service.name = membership-service - name: odigos-demo-membership enabled: true serviceName: membership-service kind: systemd config: null ``` You can define multiple sources in a single YAML file, and you can organize sources across multiple YAML files. ```shell theme={null} :wq! ``` ```shell theme={null} sudo journalctl -u odigos-vmagent | grep 'Successfully attached eBPF probes to process' ``` ``` Mar 10 20:14:35 ip-10-0-1-51 odigos-vmagent[4203]: time=2026-03-10T20:14:35.846Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/instrumentor/ebpf_manager.go:179 msg="Successfully attached eBPF probes to process" pid=7251 service=odigos-demo-frontend Mar 10 20:21:42 ip-10-0-1-51 odigos-vmagent[4203]: time=2026-03-10T20:21:42.098Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/instrumentor/ebpf_manager.go:179 msg="Successfully attached eBPF probes to process" pid=8308 service=odigos-demo-membership ``` # Custom Instrumentation Source: https://docs.odigos.io/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation Define custom eBPF-based instrumentations for arbitrary functions in your Java, Go, or C++ application or its dependencies using the VM Agent. [Custom instrumentation](/vmagent/setup/configuration/instrumentation-rules/overview#custom-instrumentation) lets you define custom eBPF-based instrumentations for specific functions or methods in your application or its dependencies. Unlike automatic instrumentation, which covers common frameworks and libraries, custom instrumentation targets the exact code you choose—for example, a business-critical function or an internal helper that you want to see as a span in your traces. The VM Agent supports **Java**, **Go**, and **C++**. For Go, you can instrument either a **function** (by package and function name) or a **receiver method** (by package, receiver type, and method name); you must use one form or the other, not both. For C++, you target a function by its **signature**. There are two ways to add a custom instrumentation rule: using `odictl` or using YAML files. Prefer to find hot functions visually instead of typing class and method names? Use the odictl profiling view to instrument a function straight from a CPU profile. See [Instrument from Profiles](/vmagent/setup/configuration/instrumentation-rules/instrument-from-profiles). ```shell theme={null} odictl ``` Use `Tab` to cycle to the **Instrumentation Rules** pane, or click it with your mouse. Press `Enter` or click **+ Add Rule** to add a new rule. Open the Instrumentation Rules pane

Select **Custom instrumentation** as the rule type and press `Enter`. Choose Custom instrumentation

1. Press `i` to enter INSERT mode. 2. For **Java**: set `className` (fully qualified class name) and `methodName` (method name). 3. For **Go function**: set `packageName` and `functionName`. Do not set `receiverName` or `receiverMethodName`. 4. For **Go receiver method**: set `packageName`, `receiverName`, and `receiverMethodName`. Do not set `functionName`. 5. For **C++**: set `signature` (the function signature, for example `std::vector::push_back` or `SSL_write`). 6. When finished, press `Esc`, then type `:wq` to save and exit. **Examples** Instrument a method by fully qualified class and method name: ```yaml theme={null} name: my-custom-instrumentation type: CustomInstrumentation config: java: - className: "com.example.MyClass" methodName: "myMethod" ``` Instrument a function by package and function name: ```yaml theme={null} name: my-custom-instrumentation type: CustomInstrumentation config: golang: - packageName: "github.com/example/ecommerce" functionName: "ProcessOrder" ``` Instrument a method by package, receiver type, and method name: ```yaml theme={null} name: my-custom-instrumentation type: CustomInstrumentation config: golang: - packageName: "github.com/example/ecommerce" receiverName: "OrderProcessor" receiverMethodName: "ProcessOrder" ``` Instrument a function by its signature: ```yaml theme={null} name: my-custom-instrumentation type: CustomInstrumentation config: cpp: - signature: "ecommerce::OrderProcessor::processOrder" ``` The `java:`, `golang:`, and `cpp:` keys accept a list; add multiple entries to instrument several methods or functions in one rule. Use only one language per rule (do not combine `java:`, `golang:`, and `cpp:`). For Go, use either functions or receiver methods in a single rule, not both. To cancel, press `Esc` if in INSERT mode, then type `:q!` to exit without saving. The rule appears in the **Rules** list. Rules list with the new rule

```shell theme={null} cd /var/lib/odigos-agent/config/rules.d ``` Create a YAML file for your custom instrumentation rule. The example below uses [vi](https://en.wikipedia.org/wiki/Vi). ```shell theme={null} sudo vi custom-instrumentation.yaml ``` Add a rule with `CustomInstrumentation` and one of `java`, `golang`, or `cpp`. For Go, use either a **function** (package + functionName) or a **receiver method** (package + receiverName + receiverMethodName), not both. ```yaml theme={null} name: custom-instrumentation type: CustomInstrumentation config: java: - className: "com.example.ecommerce.OrderService" methodName: "processOrder" ``` ```yaml theme={null} name: custom-instrumentation type: CustomInstrumentation config: golang: - packageName: "github.com/example/ecommerce" functionName: "ProcessOrder" ``` ```yaml theme={null} name: custom-instrumentation type: CustomInstrumentation config: golang: - packageName: "github.com/example/ecommerce" receiverName: "OrderProcessor" receiverMethodName: "ProcessOrder" ``` ```yaml theme={null} name: custom-instrumentation type: CustomInstrumentation config: cpp: - signature: "ecommerce::OrderProcessor::processOrder" ``` ```shell theme={null} :wq! ``` ```shell theme={null} sudo journalctl -u odigos-vmagent | grep -i rules ``` ``` Mar 16 21:49:38 ip-10-0-1-51 odigos-vmagent[36872]: {"level":"info","ts":"2026-03-16T21:49:38.580Z","caller":"logger/logger.go:177","msg":"Publishing rules snapshot","component":"vm-agent","controller":"rules","files":4,"total_rules":4} Mar 16 21:49:38 ip-10-0-1-51 odigos-vmagent[36872]: {"level":"info","ts":"2026-03-16T21:49:38.580Z","caller":"logger/logger.go:177","msg":"Received RulesChange update","component":"vm-agent","controller":"coordinator","rules":4,"has_rules":true} ``` # Instrument from Profiles Source: https://docs.odigos.io/vmagent/setup/configuration/instrumentation-rules/instrument-from-profiles Create custom instrumentation rules from a function in the odictl continuous profiling view. The odictl [profiling view](/vmagent/instrumentations/profiling/view-in-odictl) lets you turn a profiled function into a [custom instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation) rule with a single keystroke. Instead of typing a class, package, or method name by hand, you highlight a hot function in the profile and press `i`—`odictl` generates the matching `CustomInstrumentation` rule for you. This UI-driven flow supports **Java**, **Go**, and **C++**. This page covers the profile-driven, UI-only workflow. To define a rule manually (entering class/method or package/function yourself) or with YAML files, see [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation). ## Before you start * Profiling must be enabled and collecting data. See [View profiles in odictl](/vmagent/instrumentations/profiling/view-in-odictl). * A [source](/vmagent/setup/configuration/add-sources) must be enabled for the service you want to instrument. ## Create a rule from a profile From the main `odictl` dashboard, press Shift+P to open the [profiling view](/vmagent/instrumentations/profiling/view-in-odictl). Use `Tab` to move between the three panes—**Services**, **Filter modules**, and **Functions**—and the arrow keys to move within the focused pane. 1. In the **Services** pane, select a service. Its runtime (Java, Go, or C++) determines the modules shown. 2. In the **Filter modules** pane, adjust the module or type filter if needed. 3. In the **Functions** pane, highlight the function you want to instrument. In this example, the `odigos-demo-frontend` service is selected and the highlighted function is the Java method `sun.net.www.protocol.http.HttpURLConnection.getHeaderField`. odictl profiling view with the getHeaderField function selected, before instrumenting

odictl profiling view with the getHeaderField function selected, before instrumenting

Press i. The bottom-right panel updates to show the function is **Instrumented**. In this example, `getHeaderField` is now instrumented under the `odigos-profiling-odigos-demo-frontend` rule. odictl profiling view showing the getHeaderField function marked as Instrumented

odictl profiling view showing the getHeaderField function marked as Instrumented

Press Esc to return to the main dashboard. A new rule appears in the **Rules** section, named `odigos-profiling-` for the service you selected—`odigos-profiling-odigos-demo-frontend` in this example. Highlight the rule to view its configuration. The generated config matches the runtime of the profiled function: * **Java** — the fully qualified class name and method name. * **Go** — the package and function name, or the package, receiver type, and method name. * **C++** — the function signature. For the underlying field reference, see [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation). odictl Rules section showing the generated profiling rule and its config

odictl Rules section showing the generated profiling rule and its config

The rule name and config are generated automatically from the profile—you don't enter the class, package, or method by hand. To author a rule manually instead, see [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation). Once the rule is applied, new traces include a span for the instrumented function. Open your trace backend and search for the function name (for example, `getHeaderField`) to confirm the span is being recorded. ## Remove a profile-driven instrumentation To stop instrumenting a function, reopen the profiling view (`Shift+P`), highlight the same function, and press i again. The panel shows **\[i] remove** for an already-instrumented function. ## Related * [View profiles in odictl](/vmagent/instrumentations/profiling/view-in-odictl) — open and navigate the profiling view. * [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation) — define rules manually with `odictl` or YAML. * [Instrumentation rules overview](/vmagent/setup/configuration/instrumentation-rules/overview) — all rule types and their benefits. # Overview Source: https://docs.odigos.io/vmagent/setup/configuration/instrumentation-rules/overview What Odigos VM Agent instrumentation rules are, their benefits, and rule types: code attributes, headers collection, payload collection, and custom instrumentation. Instrumentation rules control how telemetry is recorded from your [sources](/vmagent/overview#key-concepts). You can apply rules to specific sources and instrumentation libraries to control which attributes, headers, and payloads are collected and how custom instrumentations behave. ## Benefits * **Control what is recorded** — Decide which code attributes, HTTP headers, and payloads are added to spans so you capture the right context without excess volume. * **Enrich without code changes** — Get better context without modifying or redeploying your application. * **Trace back to code** — Code attributes link spans to the code that produced them, so you can debug and analyze with confidence. * **Custom instrumentation** — Define custom eBPF-based instrumentations for arbitrary functions in your application or its dependencies. ## Rule types You can manage instrumentation rules via [odictl](/vmagent/setup/configuration/odictl) or YAML. ### Custom instrumentation Define custom eBPF-based instrumentations for arbitrary functions in your application or its dependencies. Use custom instrumentation with Java, Go, and C++ applications to instrument application-specific code. See [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation) for configuration, or [Instrument from Profiles](/vmagent/setup/configuration/instrumentation-rules/instrument-from-profiles) to create a rule directly from a function in the odictl profiling view. ### Code attributes Add [semantic code attributes](https://opentelemetry.io/docs/specs/semconv/attributes-registry/code/) to spans so you can trace an operation back to the code that generated it. **Planned for a future release.** ### Headers collection Add span attributes by collecting HTTP header data from requests and responses. **Planned for a future release.** ### Payload collection Add span attributes by collecting payload data from HTTP requests and responses, database queries, and messaging systems. **Planned for a future release.** # odictl Source: https://docs.odigos.io/vmagent/setup/configuration/odictl Launching and navigating the terminal user interface (TUI); `odictl` ## Introduction The Odigos VM Agent is installed with a terminal user interface (TUI) called `odictl`. `odictl` is one method you can use to instrument processes and services, set up destinations, and add actions to Odigos. Want centralized visibility and management? See [Connecting VM Agent](/central/adding-connections/vmagent) to connect this host to Odigos Central. To launch `odictl`, run the following command from the command-line: ```shell theme={null} odictl ``` You will then see a screen similar to the one below: Initial odictl TUI screen

## Install a license Apply the Odigos Pro token the support team issued. `odictl` validates the JWT before writing `/var/lib/odigos-agent/config/license.jwt`, then you restart the agent: ***Alternative***: Create a file called `/var/lib/odigos-agent/config/license.jwt` with the JWT token and restart the agent. ```shell theme={null} sudo odictl -odigos-license '' sudo systemctl restart odigos-vmagent ``` See [Installation](/vmagent/setup/installation#apply-the-license). ## Navigation The odictl interface has five main sections: Services, Logs, Sources, Destinations, and Actions. Use the keyboard to move between them; if you are connected through an `ssh` session from a terminal that supports mouse input, you can navigate the interface with your mouse. ## Navigating the Interface The following describes how to navigate using the keyboard: | Key | Action | | - | - | | `Tab` / `Shift+Tab` | Cycle panes | | `↑` / `↓` | Move between items in the current section | | `Enter` | Select the highlighted item | | `q` / `Esc` | Quit | | `s` | Focus Services | | `o` | Focus Sources | | `d` | Focus Destinations | | `a` | Focus Actions | | `c` | Open Configuration Editor | | `r` | Reload status + config | | `e` | Edit selected | | `Del` / `Backspace` | Delete selected | | `l` | Show service logs | | `y` | Show YAML config | | `?` / `h` | Display full keymap | The currently selected item has a secondary border around it. Press `Enter` to select it. #### Requirements Mouse support requires: * A terminal emulator that supports mouse input * An SSH session that forwards mouse events Most modern terminal emulators support mouse interaction automatically. To navigate: * Click a **section** to switch sections * Click an **item** to select it ## Services Panel The Services section shows the status of the two services that run the Odigos VM Agent: odigos-vmagent and odigos-otelcol. When you highlight a service, the log window shows that service's logs. Services section of the odictl interface

Services section of the odictl interface

The following animation shows switching between viewing logs for odigos-vmagent and odigos-otelcol in the Logs section.