# Cloud Connectors
Source: https://docs.odigos.io/central/adding-connections/cloud-connectors

Discover and instrument cloud workloads from Odigos Central. Full documentation lives in the Cloud Connectors product tab.

Cloud Connectors let Odigos Central discover and instrument workloads in cloud accounts (AWS, GCP, and more) from the same control plane you use for remote Kubernetes clusters and VM hosts.

Each connector manages **one** cloud account boundary (for example an AWS account or a GCP project). From Central you discover workloads in that account, create Sources for the ones you want to observe, and track instrumentation progress.

> **[Cloud Connectors documentation](/cloud-connectors/overview)**
>
> Enable the feature, add AWS or GCP connectors, instrument workloads, and manage connectors.


# Connecting Remote Clusters
Source: https://docs.odigos.io/central/adding-connections/remote-clusters

Configure remote Kubernetes clusters to connect to Odigos Central using the CLI or Helm, including TLS configuration for production.

Once Odigos Central is running, you need to configure your remote clusters to connect to it. This is done by installing Odigos in each remote cluster with the Central Proxy configuration.

Set **`centralProxy.centralBackendURL`** (and **`clusterName`**). That URL is what the Central Proxy uses to open a WebSocket to Central at `/ws/proxy`. It must be reachable **from the remote cluster**, include a scheme (`https://` in production), and match the hostname you expose for `central-backend`. If UI and backend share one ingress host, use that host — and make sure `/ws/proxy` is routed to `central-backend`, not `central-ui`. See [Authentication](/central/authentication#production-ingress-configuration) and [Architecture](/central/architecture#backend-urls-helm-values).

> [!NOTE]
>
> To connect VM hosts, see [VM Agent](/central/adding-connections/vmagent).
> For cloud accounts (AWS, GCP, and more), see [Cloud Connectors](/cloud-connectors/overview).

> [!NOTE]
>
> Remote clusters require the same Odigos Enterprise license token as the
> central installation.

## Connect a remote cluster

**CLI**

```bash theme={null}
odigos install \
  --onprem-token <token> \
  --set centralProxy.centralBackendURL=<your-central-backend-url> \
  --set clusterName=<your-cluster-name>
```

**Example:**

```bash theme={null}
odigos install \
  --onprem-token <token> \
  --set centralProxy.centralBackendURL=https://central-backend.example.com:8081 \
  --set clusterName=production-east-1
```

**Helm**

Add the following to your remote cluster's `values.yaml`:

```yaml theme={null}
clusterName: 'production-east-1'

centralProxy:
  centralBackendURL: 'https://central-backend.example.com:8081'
```

Then install:

```bash theme={null}
helm upgrade --install odigos odigos/odigos \
  --namespace odigos-system \
  --create-namespace \
  -f values.yaml
```

## TLS Configuration

For production deployments, configure TLS for secure communication between the Central Proxy and Central Backend:

```yaml theme={null}
centralProxy:
  centralBackendURL: 'https://central-backend.example.com:8081'
  tls:
    # Skip certificate verification (for testing/self-signed certificates only)
    skipVerify: false

    # Secret name containing CA certificate (key: 'ca.crt')
    caSecretName: 'central-ca-cert'

    # Secret name for mTLS client certificate (keys: 'tls.crt', 'tls.key')
    clientCertSecretName: 'central-client-cert'
```

**Create the CA certificate secret:**

```bash theme={null}
kubectl create secret generic central-ca-cert \
  --from-file=ca.crt=/path/to/ca.crt \
  -n odigos-system
```

**Create the client certificate secret (for mTLS):**

```bash theme={null}
kubectl create secret generic central-client-cert \
  --from-file=tls.crt=/path/to/client.crt \
  --from-file=tls.key=/path/to/client.key \
  -n odigos-system
```

The proxy connects to **`/ws/proxy`** (not `/wss/proxy`). It uses the `wss` scheme when the backend URL host ends with `:443`. Verify the connection with:

```bash theme={null}
kubectl logs deploy/central-proxy -n odigos-system --tail=20
```

Look for `WebSocket connection established`.


# Connecting VM Agent
Source: https://docs.odigos.io/central/adding-connections/vmagent

Connect existing VM Agent hosts to Odigos Central for centralized visibility and management.

Powered by **eBPF**, the **Odigos VM Agent** automatically instruments systemd services, processes, and containers across your Linux hosts, seamlessly exporting telemetry to your observability backends. **Odigos Central** then brings it all together, providing a single, unified interface to view and manage all your VM Agent hosts from one place.

> [!NOTE]
>
> If you are looking to connect Kubernetes clusters, see [Remote Clusters](/central/adding-connections/remote-clusters).
> For cloud accounts (AWS, GCP, and more), see [Cloud Connectors](/cloud-connectors/overview).

## Prerequisites

Before you connect a host, make sure you:

* Install VM Agent on your Linux hosts. If VM Agent is not installed yet, start with [VM Agent installation](/vmagent/setup/installation).
* Expose Odigos Central externally (for example, via an Ingress, LoadBalancer Service, or another supported exposure method).
* Have a Central endpoint URL that is reachable from your VM hosts.

To connect VM Agent to Odigos Central, use one of the following methods:

**odictl**

Use `odictl` to connect the VM Agent to Odigos Central.

**1. Open odictl on the host**

```shell theme={null}
odictl
```

![odictl home screen](https://mintlify.s3.us-west-1.amazonaws.com/odigos/images/central/vmagent/odictl-home.png)

**2. Go to the Configuration screen**

Press `c` to navigate to the configuration screen.

![odictl config screen](https://mintlify.s3.us-west-1.amazonaws.com/odigos/images/central/vmagent/odictl-config.png)

**3. Navigate to the Central section**

Use the arrow keys or mouse to navigate to the Central section on the left side of the screen.

![odictl Central screen](https://mintlify.s3.us-west-1.amazonaws.com/odigos/images/central/vmagent/odictl-central.png)

**4. Configure Central settings**

Use the `Tab` key or mouse to navigate and configure the following fields:

**central.enabled** `boolean` *required*

Enable the Central controller. If false, the controller will not start.\\

Set this value to `true`.

**central.endpoint** `string` *required*

WebSocket endpoint of the Central server in host:port or URL format.\\

Set this to the Central endpoint URL from the [Prerequisites](#prerequisites) section above.

**central.agent\_name** `string`

Agent name identifier for this VM Agent instance. (Defaults to IP of the host where dots are replaced with dashes, with an ip- prefix)\\

Set this to the display name you want to see in Odigos Central.

![odictl Central screen](https://mintlify.s3.us-west-1.amazonaws.com/odigos/images/central/vmagent/odictl-central-configured.png)

**5. Save the configuration**

Press `Ctrl+S` to save the configuration.

**6. Exit odictl**

Press `Ctrl+C` to exit `odictl`.

**7. Restart the odigos-vmagent service**

Restart the VM Agent service:

```shell theme={null}
sudo systemctl restart odigos-vmagent.service
```

**YAML**

Use `YAML` configuration files to connect VM Agent to Odigos Central.

**1. Navigate to the /var/lib/odigos-agent/config folder**

```shell theme={null}
cd /var/lib/odigos-agent/config
```

**2. Edit the config.yaml file**

Edit the `config.yaml` file using an editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi config.yaml
```

**3. Add Odigos Central to the configuration**

Modify the following fields:

**central.enabled** `boolean` *required*

Enable the Central controller. If false, the controller will not start.\\

Set this value to `true`.

**central.endpoint** `string` *required*

WebSocket endpoint of the Central server in host:port or URL format.\\

Set this to the Central endpoint URL from the [Prerequisites](#prerequisites) section above.

**central.agent\_name** `string`

Agent name identifier for this VM Agent instance. (Defaults to IP of the host where dots are replaced with dashes, with an ip- prefix)\\

Set this to the display name you want to see in Odigos Central.

```yaml theme={null}
log_level: info
log_format: console
cli:
  port: 23678
central:
  enabled: true
  endpoint: "odigoscentral.example.com"
  agent_name: ip-10-0-1-51
  platform_type: vm
  version: v1.6.0
...
```

> [!NOTE]
>
> The `YAML` above is only a snippet of the full `config.yaml` file.

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Restart the odigos-vmagent service**

Restart the VM Agent service:

```shell theme={null}
sudo systemctl restart odigos-vmagent.service
```

You should now see your VM Agent listed under Connections in the Central UI.

## Next steps

> **[VM Agent overview](/vmagent/overview)**
>
> Learn more about VM Agent capabilities and core concepts.


# Architecture
Source: https://docs.odigos.io/central/architecture

Learn about the architecture of Odigos Central, including the central cluster components, remote cluster proxies, VM agents, and optional Cloud Connectors.

Odigos Central consists of components deployed in a **central (management) cluster**, plus connections to remote platforms: a lightweight proxy in each **remote Kubernetes cluster**, [VM Agent](/central/adding-connections/vmagent) hosts, and optional [Cloud Connectors](/cloud-connectors/overview) for cloud accounts.

```mermaid theme={null}
flowchart TB
    subgraph central [Central Cluster]
        direction TB
        UI[Central UI<br/>Port 3000]
        Backend[Central Backend<br/>Port 8081]
        Redis[Redis<br/>Port 6379]
        Keycloak[Keycloak<br/>Port 8080]
        CC[Cloud Connectors]
        UI --> Backend
        Backend --> Redis
        Backend --> Keycloak
    end

    subgraph remotes [Remote Kubernetes Clusters]
        direction TB
        Proxy[Central Proxy]
        Odigos[Odigos Components]
        Proxy --> Odigos
    end

    subgraph vms [VM Hosts]
        VMA[VM Agent]
    end

    subgraph cloud [Cloud Accounts]
        CloudAcct[AWS / GCP / …]
    end

    Backend <-->|WebSocket| Proxy
    Backend <-->|WebSocket| VMA
    Backend <-->|WebSocket| CC
    CC -->|Cloud APIs| CloudAcct

    central ~~~ remotes
    remotes ~~~ vms
    vms ~~~ cloud
```

When [Cloud Connectors are enabled](/cloud-connectors/enable), the Central cluster also runs the connector-runtime controller, a dedicated PostgreSQL store for connector state, and per-account provider connector workloads (for example AWS, GCP, and other supported providers) that you create from the UI.

## Backend URLs (Helm values)

Several Helm values are named `centralBackendURL`. They are **not interchangeable**.

| Value | Chart | Who uses it | When to set |
| - | - | - | - |
| `centralProxy.centralBackendURL` | `odigos` (each **remote** cluster) | Central Proxy WebSocket to `/ws/proxy` | **Required** to connect a remote cluster. See [Connecting Remote Clusters](/central/adding-connections/remote-clusters). |
| `centralUI.centralBackendURL` | `odigos-central` | Central UI → backend HTTP | Optional. Default is in-cluster `http://central-backend.<namespace>:8081`. |
| `auth.externalUrl` | `odigos-central` | Browser SSO redirects (Keycloak issuer / proxy URLs) | Set to the **browser-accessible** Central Backend URL (ingress hostname). See [Authentication](/central/authentication). |

`centralProxy.centralBackendURL` is the URL **remote clusters** use to reach Central. `auth.externalUrl` is the URL **browsers** use for SSO. They are often the same public hostname when you terminate TLS on ingress, but they serve different clients.

## Components

| Component | Description |
| - | - |
| **Central UI** | Web interface for managing connected clusters, VM agents, cloud connectors, sources, destinations, and sampling configurations |
| **Central Backend** | API server that stores configuration in Redis and communicates with remote platforms |
| **Central Proxy** | Lightweight service deployed in each remote cluster that bridges the central backend to local Odigos components |
| **Cloud Connectors** (optional) | Provider runtimes in the Central cluster that discover and instrument cloud workloads for one cloud account boundary each (for example an AWS account or GCP project). See [Cloud Connectors](/cloud-connectors/overview). |


# Authentication
Source: https://docs.odigos.io/central/authentication

Configure authentication for Odigos Central using OIDC or SAML identity providers, including production ingress setup.

Odigos Central includes a bundled **Keycloak** instance as its identity provider. Keycloak is **not exposed externally** — the Central Backend acts as a reverse proxy, forwarding authentication requests (`/realms/*` and `/resources/*`) to Keycloak internally. This means you only need to expose the Central Backend and Central UI; no separate ingress or service is needed for Keycloak.

Odigos Central supports two types of external SSO providers:

* **OIDC** — for providers like Azure AD, Auth0, or any OpenID Connect-compatible IdP
* **SAML** — for providers like Okta that use SAML 2.0

Both are configured through the Central UI — Odigos will create the required Keycloak configuration automatically.

> [!NOTE]
>
> Helm `auth.adminUsername` and `auth.adminPassword` are the Keycloak master admin used by `central-backend`. They are not Central UI login credentials. Sign in with SSO (the first SSO user becomes an admin) or with users created in the UI. See [Troubleshooting](/central/troubleshooting#admin-credentials-vs-central-ui-login).

## Configuring the External URL

For SSO redirects to work, Keycloak and the Central Backend must know the **browser-accessible URL** of the Central Backend. This is set via the `auth.externalUrl` Helm value. The Central UI URL used for OAuth `redirect_uri` (`/signin`) is configured separately via `auth.externalUIUrl` when it differs from the backend URL (e.g. local port-forward).

| Environment | `auth.externalUrl` value | `auth.externalUIUrl` value |
| - | - | - |
| Local dev / port-forward | `http://localhost:8081` (default) | `http://localhost:3000` (auto) |
| Ingress with HTTPS | `https://central.example.com` | same as `externalUrl` (auto) |
| LoadBalancer service | `https://central-lb.prod.internal` | same as `externalUrl` (auto) |

**CLI**

```bash theme={null}
odigos pro central install \
  --set onPremToken=<token> \
  --set auth.externalUrl=https://central.example.com
```

**Helm**

```yaml theme={null}
auth:
  externalUrl: "https://central.example.com"
```

Or via `--set`:

```bash theme={null}
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --create-namespace \
  --set onPremToken=<token> \
  --set auth.externalUrl=https://central.example.com
```

> [!NOTE]
>
> When `auth.externalUrl` is empty (the default), it falls back to `http://localhost:8081`, which works for local development with `odigos pro central ui` port-forwarding.

## OIDC Provider Setup (Azure AD, Auth0, etc.)

**1. Create the OIDC Identity Provider in the Central UI**

Open the Central UI sign-in page, choose your OIDC provider (e.g., Azure AD), and fill in:

* **Client ID** — from your IdP application
* **Client Secret** — from your IdP application
* **Discovery URL** — your IdP's OpenID Connect discovery endpoint
* **Tenant ID** — required for Azure AD (Entra ID)

> [!NOTE]
>
> Odigos Central will create/update the required configuration in the bundled Keycloak for you. Avoid configuring the identity provider directly in Keycloak unless you're troubleshooting.

**2. Copy the Callback URL**

After the provider is created successfully, the Central UI will display a **Callback URL**. Copy this URL and add it to your IdP application settings as the **Redirect URI** (or **Callback URL**).

> [!WARNING]
>
> The Login button is disabled until you copy the Callback URL. This ensures you configure your IdP correctly before attempting to sign in.

**3. Sign in via your OIDC provider**

Click the **Login** button. You will be redirected to your IdP's sign-in page. After authenticating, you will be redirected back to Odigos Central and the first user account will be created as the admin.

> [!NOTE]
>
> To grant roles automatically based on Azure AD group membership, see [Role Mappings](./role-mappings).

### Azure AD (Entra ID) network access

Keycloak must reach Microsoft from inside the cluster. Allow egress to at least:

* `login.microsoftonline.com` — authorization, token, JWKS, and OpenID discovery
* `graph.microsoft.com` — OIDC userinfo

Also allow any tenant-specific discovery or token URLs you configure. If domains are blocked, or the Entra app lacks permissions, Keycloak often shows only `couldNotSendAuthenticationRequestMessage` with no error class. See [Troubleshooting](/central/troubleshooting#azure-oauth).

## SAML Provider Setup (Okta, etc.)

**1. Create the SAML Identity Provider in the Central UI**

Open the Central UI sign-in page, choose SAML, and fill in the IdP details from your SAML provider:

* **IdP Entity ID / Issuer**
* **Single Sign-On Service URL**
* **X.509 Signing Certificate**

> [!NOTE]
>
> Odigos Central will create/update the required configuration in the bundled Keycloak for you. Avoid configuring the identity provider directly in Keycloak unless you're troubleshooting.

**2. Copy the Callback URL (ACS URL)**

After the provider is created, the Central UI will display the **Callback URL (ACS URL)**. Copy this URL and configure it in your SAML provider:

* **Single sign-on URL** (ACS URL)
* **Audience URI (SP Entity ID)** — use the same URL or the realm URL depending on your provider

> [!WARNING]
>
> The Login button is disabled until you copy the Callback URL. Make sure to add it to your SAML application before proceeding.

**3. Sign in via your SAML provider**

Click the **Login with SAML** button. You will be redirected to your IdP's sign-in page. After authenticating, you will be redirected back to Odigos Central and the first user account will be created as the admin.

## Production Ingress Configuration

When deploying behind an ingress, route traffic to the Central Backend and Central UI services only. Keycloak does **not** need its own ingress — the Central Backend handles `/realms/*` and `/resources/*` internally.

`/resources/*` serves Keycloak theme and static assets (login page CSS, JS, and images). Browsers request it during sign-in; if it is routed to `central-ui`, login pages break.

Example routing (single hostname, path-based):

| Path | Service | Port | Notes |
| - | - | - | - |
| `/` | `central-ui` | 3000 | Central UI. Next.js also rewrites some API paths when the request hits this service. |
| `/graphql` | `central-backend` | 8081 | GraphQL API |
| `/api/*` | `central-backend` | 8081 | Sign-in / sign-out helpers |
| `/realms/*` | `central-backend` | 8081 | Keycloak OIDC/SAML (proxied) |

> [!WARNING]
>
> The WebSocket path is `/ws/proxy`, not `/wss/proxy`. `wss` is the TLS **scheme**. Central Proxy uses `wss` when the backend URL host ends with `:443`; otherwise it uses `ws`. Enable WebSocket upgrades (HTTP `Upgrade`) on this path.

Set `auth.externalUrl` (and `auth.externalUIUrl` if the UI origin differs) to the public HTTPS URL of this hostname. Remote clusters should set `centralProxy.centralBackendURL` to the same reachable backend URL. See [Architecture](/central/architecture#backend-urls-helm-values) and [Connecting Remote Clusters](/central/adding-connections/remote-clusters).

> [!NOTE]
>
> If you're looking to enable OIDC for the **non-central** Odigos UI (not Odigos
> Central), see the dedicated OIDC documentation: [`OIDC (OpenID
>     Connect)`](../../enterprise/features/oidc).


# Installation
Source: https://docs.odigos.io/central/installation

Install Odigos Central using the CLI or Helm chart to manage multi-cluster observability from a single control plane.

You can install Odigos Central using the CLI or Helm chart.

**CLI**

Use the `odigos pro central install` command with `--set` flags:

```bash theme={null}
odigos pro central install --set onPremToken=<token>
```

This installs:

* central-backend
* central-ui
* redis (for state)
* keycloak (as identity provider)

### CLI Flags

| Flag | Default | Description |
| - | - | - |
| `--set onPremToken` | (required) | Your Odigos Enterprise license token |
| `--set auth.externalUrl` | `http://localhost:8081` | Browser-accessible URL of the Central Backend for SSO redirect flows |
| `--set auth.externalUIUrl` | (derived from `auth.externalUrl`) | Browser-accessible URL of the Central UI for OAuth `redirect_uri` (`/signin`). When empty, defaults to `auth.externalUrl`, except for local port-forward (`http://localhost:8081`) where the UI is on port 3000 |
| `--set auth.adminPassword` | (auto-generated) | Keycloak **master** admin password for backend API access. If not provided, a random password is generated. Not used for Central UI login. See [Troubleshooting](/central/troubleshooting#admin-credentials-vs-central-ui-login) |
| `--set auth.adminUsername` | `admin` | Keycloak master admin username |
| `--set auth.persistence.enabled` | `false` | Persist Keycloak data on a PVC. Recommended for production. See [Keycloak persistence](#keycloak-persistence) |
| `--set cloudConnectors.enabled` | `false` | Enable [Cloud Connectors](/cloud-connectors/enable) (controller, Postgres, CRD) |
| `--namespace`, `-n` | `odigos-central` | Target namespace for installation |
| `--version` | (current) | Specify a specific version to install |

**Example with all options:**

```bash theme={null}
odigos pro central install \
  --set onPremToken=<token> \
  --namespace odigos-central \
  --version v1.0.0
```

**Helm**

Add the Odigos Helm repository and install:

```bash theme={null}
helm repo add odigos https://odigos-io.github.io/odigos/
helm repo update
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --create-namespace \
  --set onPremToken=<token>
```

For production deployments, consider using a `values.yaml` file:

```yaml theme={null}
onPremToken: "<your-token>"

auth:
  externalUrl: "https://central.example.com"  # Browser-accessible URL for SSO redirects
  adminUsername: admin
  adminPassword: "<your-password>"
  persistence:
    enabled: true
    size: 2Gi
    storageClassName: "gp3"

centralBackend:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 1000m
      memory: 1Gi

centralUI:
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi
```

Then install with:

```bash theme={null}
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --create-namespace \
  -f values.yaml
```

> [!TIP]
>
> `centralProxy.centralBackendURL` is not a Central install flag. Set it when you install the `odigos` chart on each **remote** cluster so the Central Proxy can reach this backend. See [Connecting Remote Clusters](/central/adding-connections/remote-clusters).

## Keycloak image

Odigos Central deploys Keycloak from the image in `auth.image` (see the [`odigos-central` Helm values file](https://github.com/odigos-io/odigos/blob/main/helm/odigos-central/values.yaml)). That image is based on official Keycloak with health and metrics enabled at build time, which is required for `start --optimized`.

The chart starts Keycloak with `start --optimized --http-enabled=true` and probes `/health/live` and `/health/ready`. Those endpoints are not available on a stock official image in optimized mode.

> [!WARNING]
>
> Do not replace `auth.image` in the helm chart with `quay.io/keycloak/keycloak` (or another unmodified official tag) unless you also change the container args and probes. An official image can appear Ready while `central-backend` stays on "Waiting for Keycloak to become ready" and `/readyz` never succeeds. See [Troubleshooting](/central/troubleshooting#waiting-for-keycloak-to-become-ready).

When mirroring images to a private registry, also pull the Keycloak image from `auth.image` in the Helm values file. See [Custom Docker Registry](#custom-docker-registry).

## Custom Docker Registry

Odigos Central images are hosted at `registry.odigos.io`. To install from a private registry, mirror at least:

| Image | Notes |
| - | - |
| `registry.odigos.io/odigos-enterprise-central-backend:$VERSION` | Central Backend. Tag matches the Odigos Central version. |
| `registry.odigos.io/odigos-enterprise-central-ui:$VERSION` | Central UI. Tag matches the Odigos Central version. |
| Keycloak (`auth.image`) | Bundled Keycloak. Tag is **not** the Odigos version. Do not substitute the official Keycloak image. |
| Redis (`redis.image`) | Bundled Redis. |

The Keycloak and Redis image versions are defined in the [`odigos-central` Helm values file](https://github.com/odigos-io/odigos/blob/main/helm/odigos-central/values.yaml) (`auth.image` and `redis.image`). Use those values as the source of truth when mirroring; do not rely on tags copied from this page.

If Cloud Connectors are enabled, also mirror `odigos-enterprise-connector-runtime`, `odigos-enterprise-connector-aws`, `odigos-enterprise-connector-gcp`, `odigos-enterprise-connector-azure`, and `odigos-enterprise-connector-postgres`.

Keycloak is set with `auth.image`, not `imagePrefix`. After you push the mirrored Keycloak image, set `auth.image` to that full URL. Copy `auth.image` from the Helm values file first:

```bash theme={null}
# Example: registry.odigos.io/keycloak:<tag> from helm/odigos-central/values.yaml
export KEYCLOAK_IMAGE=<auth.image from values.yaml>
export KEYCLOAK_TAG=<tag from auth.image>

docker pull --platform $PLATFORM $KEYCLOAK_IMAGE
docker tag $KEYCLOAK_IMAGE $CUSTOM_DOCKER_REGISTRY/keycloak:$KEYCLOAK_TAG
docker push $CUSTOM_DOCKER_REGISTRY/keycloak:$KEYCLOAK_TAG
```

```bash theme={null}
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --create-namespace \
  --set onPremToken=$ODIGOS_TOKEN \
  --set imagePrefix=$CUSTOM_DOCKER_REGISTRY \
  --set image.tag=$VERSION \
  --set auth.image=$CUSTOM_DOCKER_REGISTRY/keycloak:$KEYCLOAK_TAG
```

For mirroring **K8s agent** images (`odigos` chart), see [Custom Docker Registry](/enterprise/setup/docker-registry).

## Keycloak persistence

Persistence is **disabled by default**. Without a PVC, Keycloak stores data in the container filesystem. Restarting or recreating the Keycloak pod wipes SSO configuration and users stored in Keycloak.

For production, enable a PVC:

```yaml theme={null}
auth:
  persistence:
    enabled: true
    size: 2Gi
    storageClassName: "gp3"  # omit to use the cluster default
```

After resetting Keycloak, restart `central-backend` if login does not recover. With persistence enabled, a bad identity-provider config in the database can leave pods Ready while OAuth still fails until the PVC is recreated. See [Troubleshooting](/central/troubleshooting#keycloak-persistence-and-resets).

> [!NOTE]
>
> `auth.adminUsername` / `auth.adminPassword` are the Keycloak master admin used by `central-backend`, not Central UI login credentials. On upgrade, Helm reuses the existing `keycloak-admin-credentials` secret and ignores a new `auth.adminPassword` value.

## Optional: Cloud Connectors

Cloud Connectors are disabled by default. To discover and instrument cloud workloads (AWS, GCP, and more) from Central, enable the feature at install or upgrade time:

**CLI**

```bash theme={null}
odigos pro central install \
  --set onPremToken=<token> \
  --set cloudConnectors.enabled=true \
  --set cloudConnectors.postgres.password=<strong-password> \
  --set cloudConnectors.postgres.persistence.enabled=true \
  --set cloudConnectors.postgres.persistence.size=20Gi \
  --set cloudConnectors.postgres.persistence.storageClassName=
```

On an existing install:

```bash theme={null}
odigos pro central upgrade \
  --set cloudConnectors.enabled=true \
  --set cloudConnectors.postgres.password=<strong-password> \
  --set cloudConnectors.postgres.persistence.enabled=true \
  --set cloudConnectors.postgres.persistence.size=20Gi \
  --set cloudConnectors.postgres.persistence.storageClassName=
```

**Helm**

```bash theme={null}
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --reuse-values \
  --set cloudConnectors.enabled=true \
  --set cloudConnectors.postgres.password=<strong-password> \
  --set cloudConnectors.postgres.persistence.enabled=true \
  --set cloudConnectors.postgres.persistence.size=20Gi \
  --set cloudConnectors.postgres.persistence.storageClassName=
```

See [Enable Cloud Connectors](/cloud-connectors/enable) for PostgreSQL settings, verification steps, and full values.


# Overview
Source: https://docs.odigos.io/central/overview

Odigos Central is the centralized deployment mode of Odigos, allowing organizations to manage observability from a single UI across Kubernetes clusters, VM hosts, and cloud connectors.

> [!NOTE]
>
> This feature is available in Odigos Enterprise tier only. Please contact the
> Odigos team to inquire about access to the Enterprise version.

Odigos Central is a centralized management layer for multi-environment observability. Instead of configuring instrumentation, [sampling rules](/enterprise/pipeline/sampling/rules/overview), and destinations separately in each place, Odigos Central provides a **single pane of glass** to manage:

* [Remote Kubernetes clusters](/central/adding-connections/remote-clusters)
* [VM Agent](/central/adding-connections/vmagent) hosts
* [Cloud Connectors](/cloud-connectors/overview) for cloud accounts (AWS, GCP, and more)

## Why Use Odigos Central?

* **Unified Management** - Configure instrumentation, sampling rules, and observability pipelines across clusters, VMs, and cloud accounts from a single UI
* **Reduced Operational Overhead** - No need to SSH or `kubectl` into each environment separately
* **Centralized Access Control** - Manage user authentication and authorization with support for SSO providers like Okta
* **Consistent Configuration** - Apply the same sampling and instrumentation rules uniformly across environments
* **Audit and Compliance** - Single point of control for observability configuration changes

After [installation](/central/installation), use [port-forward](/central/ui) for local access, or expose Central with ingress for production ([Authentication](/central/authentication#production-ingress-configuration)).


# Prerequisites
Source: https://docs.odigos.io/central/prerequisites

Requirements for installing Odigos Central, including Kubernetes version, license token, and network connectivity.

Before installing Odigos Central, ensure you have:

* **Kubernetes** version 1.19.0 or higher
* **Odigos Enterprise license token** (contact Odigos team to obtain)
* **Helm** 3.x (if using Helm installation method)
* **Network connectivity** from remote clusters to the Central Backend endpoint

## Network Requirements

In production, users typically reach a **single hostname** (ingress) that path-routes to Central UI and Central Backend. See [Authentication](/central/authentication#production-ingress-configuration) for the full path table.

Ensure the following network connectivity:

| Source | Destination | Port / path | Protocol |
| - | - | - | - |
| Users (browser) | Ingress hostname → Central UI | 443 (UI on 3000 behind ingress) | HTTPS |
| Users (browser) | Ingress hostname → Central Backend | 443: `/graphql`, `/api/*`, `/realms/*` | HTTPS |
| Central UI | Central Backend | 8081 | HTTP (in-cluster) |
| Central Backend | Redis | 6379 | TCP |
| Central Backend | Keycloak | 8080 | HTTP (in-cluster) |
| Keycloak | Identity provider (e.g. Azure) | 443 | HTTPS (`login.microsoftonline.com`, `graph.microsoft.com` for Entra ID) |
| Remote cluster proxies, VM agents, cloud connectors | Central Backend | 443: `/ws/proxy` | WebSocket (`wss` when the URL host ends with `:443`) |

> [!WARNING]
>
> Odigos Central is an Enterprise feature and requires a valid license token.
> Make sure your `--onprem-token` is valid and has the necessary permissions.


# Role Mappings
Source: https://docs.odigos.io/central/role-mappings

Automatically assign Odigos Central roles to users based on their Azure AD (Entra ID) group membership.

Role mappings let you grant Odigos Central roles automatically based on the **groups** a user belongs to in Azure AD (Entra ID). For example, members of your `odigos-admins` group can be granted the **Admin** role on sign-in, with no per-user configuration.

Roles are evaluated on **every login**, so changes to a user's group membership (or to the mappings themselves) take effect the next time they sign in.

> [!NOTE]
>
> Role mappings require Azure AD configured as an OIDC SSO provider. See
> [Authentication](./authentication) to set that up first.

## Roles

| Role | Capabilities |
| - | - |
| **Admin** | Full access, including managing users and role mappings |
| **Maintainer** | Manage sources, destinations, and configuration |
| **Viewer** | Read-only access |

When a user belongs to multiple mapped groups, the **highest** role wins. Precedence is: **Owner > Admin > Maintainer > Viewer**.

## Configuring Azure AD group claims

For role mappings to work, Azure must include the user's groups in the token it sends to Odigos.

**1. Emit the groups claim**

In the Azure portal, open your App Registration → **Token configuration** → **Add groups claim**. Choose the groups to emit (e.g. *Security groups*) for both the **ID** and **Access** tokens.

![Azure App Registration Token configuration with the Edit groups claim panel, Security groups selected for ID and Access tokens](https://mintcdn.com/odigos/9rV-UTkDfyqYKqN-/images/central/azure-groups-claim.png?fit=max\&auto=format\&n=9rV-UTkDfyqYKqN-\&q=85\&s=08c57a9e8d06264ec3640e7065bd7760)

**2. Note the group identifier**

The value you map must exactly match what Azure puts in the `groups` claim. By **default** Azure emits the group **Object ID** (a GUID) — copy it from Azure AD → **Groups**. If you've configured the groups claim to emit names instead (e.g. *Cloud-only group display names*, or `sAMAccountName` for on-prem-synced groups), use the group **name** instead. Either way, paste that exact value into the **Group** field.

> [!NOTE]
>
> Azure uses `groups` as the claim name by default, which matches the Odigos
> default. If you've customized the claim name, set it in the **Claim Name**
> field when creating a mapping.

## Creating a role mapping

**1. Open Access Management**

In the Central UI, go to **Access Management** and select the **Role Mappings** tab.

**2. Add a mapping**

Click **Create Role Mapping** and fill in:

* **Group** — the value from the previous step: the group **Object ID** (GUID) by default, or the group **name** if Azure emits names. It must exactly match what Azure puts in the claim.
* **Role** — the Odigos role to grant: **Admin**, **Maintainer**, or **Viewer**.
* **Claim Name** — the token claim that carries group membership. Defaults to `groups`.
* **Display Name** *(optional)* — a friendly label shown in the table instead of the raw group value. This does **not** affect matching.

![Create Role Mapping dialog in the Central UI with Group, Role, Claim Name, and Display Name fields](https://mintcdn.com/odigos/9rV-UTkDfyqYKqN-/images/central/create-role-mapping.png?fit=max\&auto=format\&n=9rV-UTkDfyqYKqN-\&q=85\&s=d4a5d7defa08f8e73c154b8948c0e0c9)

**3. Save and sign in**

Save the mapping. Affected users receive their role the next time they sign in.

> [!NOTE]
>
> Only **Admins** can create, edit, or delete role mappings. Viewers can see the
> mappings in read-only mode.

## How it works

Odigos manages the group-to-role assignment in the bundled Keycloak for you — you don't need to configure Keycloak directly. On each login, Keycloak re-evaluates the user's group claim against your mappings and assigns the matching role (highest privilege wins). Removing a mapping strips the corresponding role on the user's next sign-in.


# Troubleshooting
Source: https://docs.odigos.io/central/troubleshooting

Diagnose Odigos Central startup, Keycloak, SSO, and remote-cluster connection issues.

Use this page when Central installs, but pods stay NotReady, SSO fails, or remote clusters cannot connect. For production URL and ingress routing, see [Authentication](/central/authentication). For remote cluster Helm values, see [Connecting Remote Clusters](/central/adding-connections/remote-clusters).

## Waiting for Keycloak to become ready

`central-backend` logs `Waiting for Keycloak to become ready` while it is still initializing. During that time:

* **Liveness** (`/healthz`) returns 200 as soon as the process binds port 8081, so Kubernetes does not restart the pod.
* **Readiness** (`/readyz`) stays 503 until Keycloak and Redis init finish. Treat `/readyz` as the signal that Central is usable, not `/healthz`.

The backend waits until both succeed:

1. `GET http://keycloak:8080/realms/master/.well-known/openid-configuration`
2. A password grant against the Keycloak **master** admin (`admin-cli`) using the credentials in the `keycloak-admin-credentials` secret

Keycloak itself is only Ready when `/health/live` and `/health/ready` succeed. Those endpoints exist because Odigos ships a **custom Keycloak image** with health checks enabled at **build time**. Use `auth.image` from the [`odigos-central` Helm values file](https://github.com/odigos-io/odigos/blob/main/helm/odigos-central/values.yaml) as the source of truth. Replacing it with the official Keycloak image while keeping `start --optimized` and the chart probes typically leaves `central-backend` stuck in this wait. See [Installation](/central/installation#keycloak-image).

```bash theme={null}
kubectl logs deploy/central-backend -n odigos-central --tail=50
kubectl logs deploy/keycloak -n odigos-central --tail=50
kubectl get pods -n odigos-central
```

## Keycloak persistence and resets

Keycloak persistence is **disabled by default** (`auth.persistence.enabled: false`). Without a PVC, restarting or recreating the Keycloak pod wipes realm data, identity-provider config, and users stored in Keycloak.

After you reset Keycloak **without** persistence:

1. Confirm Keycloak becomes Ready.
2. Restart `central-backend` so it can recreate Keycloak clients, roles, and broker configuration:

```bash theme={null}
kubectl rollout restart deployment/central-backend -n odigos-central
```

The backend watches Keycloak health and can re-apply some state automatically, but if login still fails after a Keycloak reset, restart `central-backend`.

When persistence is enabled, a bad identity-provider configuration written into the Keycloak database can leave **all pods Ready** while OAuth still fails. Logs may not mention the PVC. Recover by deleting the Keycloak PVC (after backing up if needed), recreating Keycloak, and restarting `central-backend`.

Helm `auth.adminPassword` is used only on **first install** when the `keycloak-admin-credentials` secret does not exist. On upgrade, the chart **reuses the existing secret** and ignores the value in Helm. Read the live password with:

```bash theme={null}
kubectl get secret keycloak-admin-credentials -n odigos-central \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo
```

If you recreate the secret without wiping Keycloak data, the password in the secret and the password Keycloak stored will not match, and the backend cannot obtain an admin token.

## Admin credentials vs Central UI login

`auth.adminUsername` and `auth.adminPassword` configure the **Keycloak master admin**. `central-backend` uses that account to manage Keycloak. They are **not** Central UI users.

Sign in to the Central UI with:

* An SSO provider configured in the UI (the first successful SSO user becomes an admin), or
* A user created later in Central Access Management

Using the Helm admin username and password on the Central UI sign-in form returns **invalid credentials**.

## Azure OAuth

Keycloak (inside the cluster) must reach Microsoft Entra ID. Allow egress at least to:

* `login.microsoftonline.com` (authorization, token, JWKS, discovery)
* `graph.microsoft.com` (OIDC userinfo)

Also allow the exact discovery, token, and authorization URLs configured for your tenant.

Azure Tenant ID is required for the Azure OIDC provider. See [Authentication](/central/authentication) and [Role Mappings](/central/role-mappings).

Keycloak often shows only `couldNotSendAuthenticationRequestMessage` for network, DNS, TLS, and permission failures. That string does not distinguish a blocked domain from insufficient Entra app permissions. Check Keycloak logs and your egress policy.

## Remote cluster cannot connect (WebSocket)

Remote proxies, VM agents, and cloud connectors connect to `central-backend` at **`/ws/proxy`** (not `/wss/proxy`). `wss` is the TLS scheme when the backend URL host ends with `:443`.

If your ingress sends `/` to `central-ui`, you must add an explicit `/ws/proxy` rule to `central-backend` and enable WebSocket upgrades. See [Authentication](/central/authentication#production-ingress-configuration).


# Accessing the Central UI
Source: https://docs.odigos.io/central/ui

Access the Odigos Central web interface using port-forwarding or a custom address.

After installation, access the Central UI using port-forward:

```bash theme={null}
odigos pro central ui
```

This forwards:

* **Central UI** to `http://localhost:3000`
* **Central Backend** to `http://localhost:8081`

## Options

| Flag | Default | Description |
| - | - | - |
| `--address` | `localhost` | Address to bind the port-forward to |
| `--namespace`, `-n` | `odigos-central` | Namespace where Odigos Central is installed |

**Example:** To access from other machines on your network:

```bash theme={null}
odigos pro central ui --address 0.0.0.0
```

For production, expose Central with an ingress instead of port-forward. Route UI and backend paths as described in [Authentication](/central/authentication#production-ingress-configuration), and set `auth.externalUrl` to the public HTTPS URL.


# Uninstall
Source: https://docs.odigos.io/central/uninstall

Remove Odigos Central from your cluster using the CLI or Helm.

To completely remove Odigos Central:

## Using CLI

```bash theme={null}
odigos pro central uninstall
```

**Options:**

| Flag | Default | Description |
| - | - | - |
| `--namespace`, `-n` | `odigos-central` | Target namespace |
| `--yes` | `false` | Skip confirmation prompt |

**Example:**

```bash theme={null}
odigos pro central uninstall --namespace odigos-central --yes
```

## Using Helm

```bash theme={null}
helm uninstall odigos-central --namespace odigos-central
kubectl delete namespace odigos-central
```

> [!WARNING]
>
> Uninstalling Odigos Central will disconnect all remote clusters. Make sure to
> plan for this accordingly.


# Upgrade
Source: https://docs.odigos.io/central/upgrade

Upgrade Odigos Central to a newer version using the CLI or Helm.

To upgrade Odigos Central to a newer version:

## Using CLI

```bash theme={null}
odigos pro central upgrade --version <version>
```

**Options:**

| Flag | Default | Description |
| - | - | - |
| `--version` | (required) | Version to upgrade to (e.g., `v1.2.0`) |
| `--namespace`, `-n` | `odigos-central` | Target namespace |
| `--yes` | `false` | Skip confirmation prompt |
| `--image-pull-secrets` | (existing) | Update image pull secrets |
| `--central-max-message-size` | (existing) | Update max gRPC message size |

**Example:**

```bash theme={null}
odigos pro central upgrade --version v1.2.0 --yes
```

## Using Helm

```bash theme={null}
helm repo update
helm upgrade odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --reuse-values \
  --version <chart-version>
```

> [!NOTE]
>
> When upgrading, the Keycloak admin password is preserved from the existing
> secret to prevent credential mismatch.


# Add an AWS Connector
Source: https://docs.odigos.io/cloud-connectors/add-connectors/add-aws-connector

Connect an AWS account to Odigos Central, configure authentication and capabilities, and verify discovery.

Add an AWS Cloud Connector to manage AWS accounts. The connector runs in the Central cluster, discovers resources across every region enabled in that account, and can instrument AWS Lambda functions when you create Sources.

> [!NOTE]
>
> [Enable Cloud Connectors](/cloud-connectors/enable)
> before you add a connector.

## Resource types

| Type | Domain | Discovery | Instrumentation |
| - | - | - | - |
| `aws.lambda` | Workload | Yes | Yes (OpenTelemetry Lambda layer) |
| `aws.fargate-task` | Workload | Yes | Yes (agent-staging sidecar in the task definition) |
| `aws.ecs-cluster` | Compute platform | Yes | Coming soon |
| `aws.fargate-cluster` | Compute platform | Yes | Coming soon |

> [!NOTE]
>
> `aws.ecs-cluster` and `aws.fargate-cluster` are discovered for coverage today;
> Odigos does not yet install a separate agent onto them. `aws.fargate-task` — an
> individual Fargate **service** — is the ECS resource Odigos instruments directly.

aws.lambda and aws.fargate-task support two instrumentation approaches:

* Odigos-managed: The connector automatically applies and maintains the required changes.

* Self-managed: You maintain full control by provisioning through your own CloudFormation, CDK, or Terraform pipelines.

(For a deeper comparison, see [Odigos-managed vs. self-managed instrumentation](/cloud-connectors/overview#odigos-managed-vs-self-managed-instrumentation).)
For the self-managed path, artifact preloading, and per-language CDK/CloudFormation snippets, see the
[AWS Connector](/cloud-connectors/aws/overview) section.

## Add the connector

**1. Open Add connector and choose AWS**

In the Central UI, open Connections / Connectors and start **Add connector**. Choose **AWS**.

**2. Choose access level**

Choose how much access this connector should have. The UI shows an **example IAM permissions policy** for the level you pick. You will attach that policy when you create the IAM role or user in the next step.

![Create Cloud Connector dialog with AWS selected and the Discovery + instrumentation access level showing an IAM policy preview](https://mintcdn.com/odigos/cif9LffayXl8I7ih/images/cloud-connectors/aws-access-level.png?fit=max\&auto=format\&n=cif9LffayXl8I7ih\&q=85\&s=bc0d72296568b3ad94f87bd343d0f87b)

**Discovery only**

The connector can list and discover resources (e.g. Lambda functions and ECS/Fargate resources) but cannot change them or attach instrumentation.

Use this when you want inventory and visibility first, or if you are managing the environment with Infrastructure as code (IaC).

Typical policy actions include:

| Purpose | Actions |
| - | - |
| Authentication | `sts:GetCallerIdentity` |
| Region enumeration | `ec2:DescribeRegions` |
| Lambda discovery | `lambda:ListFunctions`, `lambda:ListTags` |
| ECS / Fargate discovery | `ecs:ListClusters`, `ecs:DescribeClusters`, `ecs:ListServices`, `ecs:DescribeServices`, `ecs:ListTasks`, `ecs:DescribeTasks`, `ecs:DescribeTaskDefinition` |

> [!NOTE]
>
> `ecs:DescribeTaskDefinition` is a discovery permission, not only an
> instrumentation one — it's what lets Odigos see whether a self-managed
> Fargate service already carries your own instrumentation, so grant it
> even if you only ever plan to self-manage instrumentation.

**Discovery + instrumentation**

The connector can discover resources and instrument supported workloads. Prefer this for a full setup in one step.

Includes all discovery actions, plus Odigos-managed instrumentation for Lambda and Fargate:

| Purpose | Actions |
| - | - |
| Read and update function config | `lambda:GetFunctionConfiguration`, `lambda:UpdateFunctionConfiguration` |
| Manage OTel layers | `lambda:GetLayerVersion`, `lambda:ListLayers`, `lambda:PublishLayerVersion`, `lambda:DeleteLayerVersion` |
| Manage Fargate task definitions | `ecs:RegisterTaskDefinition`, `ecs:UpdateService` |
| Pass task/execution roles when registering a task definition | `iam:PassRole` (scope with condition `iam:PassedToService=ecs-tasks.amazonaws.com`) |

Odigos-managed instrumentation also requires an [OTLP destination](/cloud-connectors/instrument-workloads#add-an-otlp-destination) after the connector is online.

**Custom**

Pick which resource types may be discovered and which may be instrumented. The UI updates the **policy preview** to match your selections so the IAM policy stays aligned with what verification will validate.

Use Custom when you need a narrower or broader mix than the presets (e.g. discover ECS clusters but only instrument Lambda).

> [!NOTE]
>
> Use the policy preview as the source of truth for the permissions policy.
> This keeps connector access scoped — not blanket account admin. See
> [You control the access scope](/cloud-connectors/overview#you-control-the-access-scope).

Click **Next** to continue to **Setup credentials**.

**3. Setup credentials**

On the **Setup credentials** screen, apply the IAM policy from the previous step in your AWS account and enter the credentials Odigos will use.

![Setup credentials screen for an AWS connector with the authentication method set to Assume Role and CloudFormation selected as the setup method](https://mintcdn.com/odigos/cif9LffayXl8I7ih/images/cloud-connectors/aws-setup-credentials.png?fit=max\&auto=format\&n=cif9LffayXl8I7ih\&q=85\&s=43b278c4028d105ceb887a417bc7a6a0)

#### 1. Choose authentication method

**Assume Role**

Odigos assumes a scoped IAM role. Prefer this for production: you attach the permissions policy to the role, and Odigos does not rely on long-lived admin credentials in the connector.

You still provide base access keys that are allowed to call `sts:AssumeRole` on that role (plus the Role ARN and External ID).

#### 2. Choose setup method

Use the tool that best fits your workflow — **CloudFormation**, **Terraform**, **AWS CLI**, or **Manual**. Each path ends with the same role (trust policy + access-level permissions) and a Role ARN to paste back into Odigos.

#### 3. Create IAM role

Follow the instructions the UI shows for the method you picked, then copy the **Role ARN**.

**CloudFormation**

1. Open the CloudFormation instructions in the UI.
2. Deploy the provided template (or stack) in the target AWS account.
3. Confirm the stack created the IAM role with the UI trust policy and the permissions policy from the access level step.
4. Copy the **Role ARN** from the stack outputs or the IAM console.

**Terraform**

1. Open the Terraform snippet in the UI.
2. Apply it in the target account or module.
3. Confirm the role was created with the UI trust policy and the permissions policy from the access level step.
4. Copy the **Role ARN** from the Terraform output or the IAM console.

**AWS CLI**

1. Open the AWS CLI commands in the UI.
2. Run them against the target account (creates the role, trust policy, and attaches permissions).
3. Confirm the commands completed successfully.
4. Copy the **Role ARN** from the command output or the IAM console.

**Manual**

1. Open the AWS Console → IAM → Roles → Create role.
2. Select **Custom trust policy**.
3. Paste the trust policy from the UI (includes the External ID condition when present).
4. Attach the **permissions policy from the access level step**.
5. Name the role and create it.
6. Open the role summary and copy the **Role ARN**.

![Create IAM role step showing CloudFormation instructions and the Enter Role ARN form with account ID, access keys, Role ARN, and External ID fields](https://mintcdn.com/odigos/cif9LffayXl8I7ih/images/cloud-connectors/aws-create-iam-role.png?fit=max\&auto=format\&n=cif9LffayXl8I7ih\&q=85\&s=7e4821c52bd4387ce42f761133d4d80b)

#### 4. Enter Role ARN

Paste the values from the previous step into the form — account ID, base access keys, the Role ARN you copied, and the External ID from the UI trust policy.

**AWS Account ID** `string` *required*

The 12-digit account ID where the IAM role lives.

**Access Key ID** `string` *required*

Access key for an identity that can assume the role.

**Secret Access Key** `string` *required*

Secret access key for that identity.

**Session Token** `string`

Required only for temporary access keys.

**IAM Role ARN** `string` *required*

ARN of the cross-account role Odigos assumes.

**External ID** `string`

Must match the External ID in the role trust policy (shown in the UI, e.g. `odigos-connector`).

**Static keys**

Use IAM user access keys that already have the permissions policy from the access level step attached. There is no separate role setup flow in the UI for this method.

1. In AWS, create or select an IAM user.
2. Attach the **permissions policy from the access level step**.
3. Create an access key for that user.
4. Enter the credentials in Odigos:

**AWS Account ID** `string` *required*

The 12-digit account ID where the IAM user lives.

**Access Key ID** `string` *required*

IAM user access key ID.

**Secret Access Key** `string` *required*

IAM user secret access key.

**Session Token** `string`

Required only for temporary access keys.

Credentials are stored in a Kubernetes Secret in the Central cluster — never in the connector spec.

**4. Verify and create**

Click **Verify & Create Connector**. Central runs connection checks, then creates the credentials Secret and `OdigosCloudConnector` resource when verification succeeds.

> [!NOTE]
>
> Verification covers authentication and discovery permissions for the access
> level you selected. For **Discovery + instrumentation**, also ensure the
> instrumentation actions from the policy preview are on the role before you
> create Sources.

**5. Wait for Connected**

After create succeeds, wait until the connector status is **Connected** and initial discovery has completed.

> [!TIP]
>
> When the connector shows Connected, you can browse discovered workloads and
> start [instrumenting](/cloud-connectors/instrument-workloads).

## Next steps

> **[Instrument workloads](/cloud-connectors/instrument-workloads#add-an-otlp-destination)**
>
> Add an OTLP destination, create Sources for discovered Lambdas and Fargate services, and track instrumentation.

> **[AWS Connector](/cloud-connectors/aws/overview)**
>
> Self-managed instrumentation, artifact preloading, and CDK/CloudFormation for every language.


# Add a GCP Connector
Source: https://docs.odigos.io/cloud-connectors/add-connectors/add-gcp-connector

Connect a Google Cloud project to Odigos Central, configure authentication and capabilities, and verify discovery.

Add a GCP Cloud Connector to manage **one Google Cloud project**. The connector runs in the Central cluster, discovers resources across every location in that project, and can instrument Cloud Run services when you create Sources.

> [!NOTE]
>
> [Enable Cloud Connectors](/cloud-connectors/enable)
> before you add a connector.

## Resource types

| Type | Domain | Discovery | Instrumentation |
| - | - | - | - |
| `gcp.cloud-run` | Workload | Yes | Yes (Java, Python, Node.js, .NET) |
| `gcp.gke` | Compute platform | Yes | Agent install coming soon |
| `gcp.gce` | Compute platform | Yes | Agent install coming soon |

## Add the connector

**1. Open Add connector and choose GCP**

In the Central UI, open Connections / Connectors and start **Add connector**. Choose **GCP**.

**2. Choose access level**

Choose how much access this connector should have. The UI shows an **example IAM policy** for the level you pick. You will grant that role to the connector identity in the next step.

![Create Cloud Connector dialog with GCP selected and the Discovery + instrumentation access level showing a custom role policy preview](https://mintcdn.com/odigos/cif9LffayXl8I7ih/images/cloud-connectors/gcp-access-level.png?fit=max\&auto=format\&n=cif9LffayXl8I7ih\&q=85\&s=34c8e9fee0eca77f9ad43c78af69d7ac)

**Discovery only**

The connector can list and read resources (e.g. Cloud Run services, GKE clusters, and Compute Engine instances) but cannot change them or attach instrumentation.

Use this when you want inventory and visibility first, or when a separate identity will handle instrumentation later.

Typical permissions include:

| Purpose | Permissions |
| - | - |
| Authentication / project access | `resourcemanager.projects.get` |
| Cloud Run discovery | `run.services.list`, `run.services.get` |
| GKE discovery | `container.clusters.list`, `container.clusters.get` |
| Compute Engine discovery | `compute.instances.list` |

**Discovery + instrumentation**

The connector can discover resources and instrument supported workloads. Prefer this for a full setup in one step.

Includes all discovery permissions, plus Cloud Run instrumentation:

| Purpose | Permissions |
| - | - |
| Update services | `run.services.get`, `run.services.update`, `run.operations.get` |
| Act as runtime service account | **`iam.serviceAccounts.actAs`** |
| Language detection (optional) | `artifactregistry.repositories.downloadArtifacts` (or `roles/artifactregistry.reader`) |

> [!WARNING]
>
> `iam.serviceAccounts.actAs` is required to update Cloud Run services that
> run as a runtime service account. Without it, instrumentation fails even
> when `run.services.update` is granted. Bind it in a custom role, or grant
> `roles/iam.serviceAccountUser` on the runtime service account.

Cloud Run instrumentation also requires an [OTLP destination](/cloud-connectors/instrument-workloads#add-an-otlp-destination) after the connector is online. That endpoint must be reachable **from Cloud Run**, because instrumented services export OTLP directly to it.

**Custom**

Pick which resource types may be discovered and which may be instrumented. The UI updates the **policy preview** to match your selections so the IAM role stays aligned with what verification will validate.

Use Custom when you need a narrower or broader mix than the presets (e.g. discover GKE clusters but only instrument Cloud Run).

> [!NOTE]
>
> Use the policy preview as the source of truth for the custom role. This
> keeps connector access scoped — not blanket project owner. See
> [You control the access scope](/cloud-connectors/overview#you-control-the-access-scope).

Enable the Cloud Run, Cloud Resource Manager, Container, Compute, and IAM APIs in the project as needed, then click **Next** to continue to **Setup credentials**.

**3. Setup credentials**

On the **Setup credentials** screen, identify the project and enter the identity Odigos uses to access it.

![Setup credentials screen for a GCP connector with GCP Project ID, authentication method toggle, and Service Account Key JSON fields](https://mintcdn.com/odigos/cif9LffayXl8I7ih/images/cloud-connectors/gcp-setup-credentials.png?fit=max\&auto=format\&n=cif9LffayXl8I7ih\&q=85\&s=f6095c846e2453948da297a0b3632f0f)

#### 1. Enter Google Cloud project

**GCP Project ID** `string` *required*

The Google Cloud project that contains the resources Odigos should discover.

#### 2. Choose authentication method

Use a service account key for local Central, or choose workload identity only when it is configured on the connector pod.

**Workload identity**

Odigos uses the connector pod's ambient credentials — [GKE Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) or Application Default Credentials. Prefer this when Central runs on GKE and you do not want a long-lived key in the cluster.

1. Create a Google Cloud service account (or reuse one) and grant it the **IAM role from the access level step**.
2. Bind that service account to the connector pod's Kubernetes service account via Workload Identity (or ensure ADC is otherwise available to the pod).
3. Select **Workload identity** in Odigos. Do **not** paste a JSON key — no credentials Secret is created for this method.

> [!NOTE]
>
> A kind or non-GKE Central cluster has no ambient GCP identity. Use
> **Service account key** instead.

**Service account key**

Use a JSON key for a dedicated service account. Prefer this for local Central (for example kind) or any cluster without Workload Identity / ADC.

1. Create a Google Cloud service account in the target project.
2. Grant it the **IAM role from the access level step**.
3. Create a JSON key for that service account.
4. Select **Service account key** in Odigos and paste the complete JSON key.

**Service Account Key (JSON)** `string` *required*

The full JSON key for a service account with the required roles.

The key is stored in a Kubernetes Secret in the Central cluster — never in the connector spec.

**4. Verify and create**

Click **Verify & Create Connector**. Central runs connection checks, then creates the `OdigosCloudConnector` resource when verification succeeds. For **Service account key**, it also writes the credentials Secret; for **Workload identity**, no Secret is created.

> [!NOTE]
>
> Verification covers authentication and discovery permissions for the access
> level you selected. For **Discovery + instrumentation**, also ensure the
> instrumentation permissions from the policy preview (including
> `iam.serviceAccounts.actAs`) are granted before you create Sources.

**5. Wait for Connected**

After create succeeds, wait until the connector status is **Connected** and initial discovery has completed.

> [!TIP]
>
> When the connector shows Connected, you can browse discovered workloads and
> start [instrumenting](/cloud-connectors/instrument-workloads).

## Next steps

> **[Instrument workloads](/cloud-connectors/instrument-workloads#add-an-otlp-destination)**
>
> Add an OTLP destination, create Sources for discovered Cloud Run services, and track instrumentation.


# CDK: Destinations
Source: https://docs.odigos.io/cloud-connectors/aws/cdk/destinations

Configure where the Odigos AWS CDK constructs send telemetry, and which signals to enable.

Both Lambda and Fargate constructs take a `destination` — where your telemetry is sent, and which OpenTelemetry
signals to include. Configure it once on the construct; every workload that construct instruments uses the
same settings.

> [!NOTE]
>
> Read [Odigos AWS CDK](/cloud-connectors/aws/cdk/overview) first for install and when to use the package.
> This page covers only the shared `destination` options.

## Choosing a destination

You give the construct an endpoint in one of two formats, depending on which protocol your destination
speaks:

**destination.endpoint** `string`

OTLP over **gRPC**, written as a bare `host:port` with **no** `https://` prefix — for example
`otlp.example.com:4317`. Port 4317 is the convention.

**destination.httpEndpoint** `string`

OTLP over **HTTP**, written as a full URL **including** the scheme — for example
`https://otlp.example.com:4318`. Port 4318 is the convention.

The two formats aren't interchangeable, and getting them mixed up is the most common mistake here — so the
construct checks: a scheme in `endpoint`, or a missing one in `httpEndpoint`, stops the build immediately
rather than deploying something that starts up fine and then quietly delivers nothing. Using a port that
belongs to the other protocol (4318 with gRPC, or 4317 with HTTP) produces a warning.

> [!NOTE]
>
> **How many endpoints to set differs between the two constructs.** Lambda takes
> **exactly one**. Fargate accepts **either or both**, and often should get both —
> see [Fargate → Transport](/cloud-connectors/aws/cdk/fargate#transport) for why.

Two more optional settings:

**destination.insecure** `boolean` (default: false)

Skip TLS verification. Applies to the gRPC `endpoint` only — the HTTP exporter takes its transport from
the URL scheme.

**destination.headers** `object`

Headers sent with every request, which in practice means your destination's credential (an API key or
token).

> [!WARNING]
>
> These are stored as a plain environment variable on the workload — readable by
> anyone who can describe the function or task definition, and visible in the AWS
> console. For a long-lived credential, prefer a destination that authenticates by
> network position (a private VPC endpoint, or mTLS), or send telemetry to a
> collector you run that holds the real credential.

## Choosing which signals to send

OpenTelemetry produces three kinds of data, and you choose which ones to send:

**destination.traces** `boolean` (default: true)

Traces — the request-by-request timelines. On by default.

**destination.metrics** `boolean` (default: false)

Metrics — numeric measurements over time. **Off unless you turn it on.**

**destination.logs** `boolean` (default: false)

Logs. **Off unless you turn it on.**

```typescript theme={null}
destination: {
  endpoint: 'otlp.example.com:4317',
  metrics: true,   // off unless you ask
  logs: true,      // off unless you ask
}
```

Traces are on by default because that is the primary signal most teams set up OpenTelemetry for.
Metrics and logs are opt-in: enabling a signal you are not ready to receive is not free — refused
batches are retried on a backoff, and on Lambda that retrying happens inside billed duration. Turn them
on when you want those signals and your destination is configured to accept them.

> [!WARNING]
>
> **Match this to what your destination in Odigos has enabled.** Odigos compares
> the workload's full telemetry configuration against what it expects, so a
> traces-only workload checked against a destination configured for all three
> signals is reported as *drifted* — even though the workload is working fine.
> Read the enabled signals off the destination in Odigos and state the same ones
> here.

> [!NOTE]
>
> Turning all three off stops the build — a workload with instrumentation but
> nowhere to send anything would be silent, which is never what you meant.

## Next steps

> **[Lambda construct](/cloud-connectors/aws/cdk/lambda)**
>
> Layer sources, web-server mode, Java handler interfaces.

> **[Fargate construct](/cloud-connectors/aws/cdk/fargate)**
>
> Languages, service naming, the agents image.


# CDK: Fargate
Source: https://docs.odigos.io/cloud-connectors/aws/cdk/fargate

Instrument ECS Fargate services from your CDK app with OdigosFargateInstrumentation — every option, and the edge cases to know about.

`OdigosFargateInstrumentation` instruments containers running in ECS Fargate services defined in your CDK
app.

> [!NOTE]
>
> Read [Odigos AWS CDK](/cloud-connectors/aws/cdk/overview) first — it covers
> installing the package and how Odigos recognizes an instrumented workload — then
> [Destinations](/cloud-connectors/aws/cdk/destinations) for the shared `destination`
> options. This page covers only what's specific to Fargate.

## Basic usage

```typescript theme={null}
import { OdigosFargateInstrumentation, OdigosFargateLanguage } from '@odigos/aws-cdk';

const odigos = new OdigosFargateInstrumentation(this, 'Odigos', {
  destination: { httpEndpoint: 'https://otlp.example.com:4318' },
});

odigos.instrument({
  container: apiContainer,               // the container to instrument
  language: OdigosFargateLanguage.JAVA,  // stated, not detected — see below
  service: apiService,                   // used to read the service name
});
```

You create the construct once, then call `instrument()` for each container you want instrumented. There's
no automatic sweep of a whole stack — you name each container explicitly.

## How it works

The construct sets up four things that work together:

1. **A shared scratch volume** on the task definition, named `odigos-agent`.
2. **An extra container**, also named `odigos-agent`, that copies the right language agent into that volume
   and then exits. It's marked non-essential, because finishing and exiting is its whole job — it isn't
   part of your running application.
3. **A read-only mount** of that volume at `/var/odigos` in your application container, which is set to
   wait until the copy step has finished successfully before it starts.
4. **Environment variables** on your application container that load the agent and configure where
   telemetry goes.

> [!WARNING]
>
> All four are one mechanism, which is why they're one construct call rather than
> four options. A container can have a perfectly correct set of environment
> variables while the copy step is missing — the application then starts, finds
> nothing at `/var/odigos`, and sends no telemetry while still *reporting* as
> instrumented.

## Where telemetry goes

**Straight from your application to your destination.** There's no intermediate collector inside the task,
so your application's own OpenTelemetry exporter is what makes the network call — which means a slow or
unreachable destination is felt by the application itself.

Two practical consequences: the destination endpoint has to be reachable from the task's network, and it's
worth pointing it somewhere with headroom, since your application is doing the sending.

## Stating the language

**language** `OdigosFargateLanguage` *required*

One of `JAVA`, `PYTHON`, `NODEJS`, `DOTNET`, `PHP`, `RUBY`.

| Language | Loaded through | `runtimeVersion` prop |
| - | - | - |
| `JAVA` — also Kotlin, Scala | `JAVA_TOOL_OPTIONS` | Ignored — one agent covers the supported JVM range |
| `PYTHON` | `PYTHONPATH` | Ignored — one agent covers the supported Python range |
| `NODEJS` | `NODE_OPTIONS` | Ignored — one agent covers the supported Node.js range |
| `DOTNET` | `CORECLR_*` / `DOTNET_*` | Ignored — one agent covers the supported .NET range |
| `PHP` | `PHP_INI_SCAN_DIR` | **Required** — picks the per-version agent folder |
| `RUBY` | `RUBYOPT` | **Required** — picks the per-version agent folder |

There's no auto-detection here, and that's a platform constraint rather than a preference. A container
declares only an *image* — nothing in the task definition says which language runs inside it. Working that
out would mean downloading and inspecting the image, which is network I/O a build can't do. So you state it.

> [!WARNING]
>
> A wrong language here fails quietly. Staging the Java agent into a Python
> container leaves it uninstrumented, and setting `JAVA_TOOL_OPTIONS` on an image
> with no JVM is simply ignored — nothing errors loudly enough to notice.

**runtimeVersion** `string`

Your application's language version as `major.minor` — for example `"8.3"` or `"3.3"`. Required for PHP
and Ruby (their agents are laid out per version in the bundle). Ignored for Java, Python, Node.js, and
.NET — those ship one agent that covers a range of runtime versions, so you don't declare which one.

> [!WARNING]
>
> PHP and Ruby agents are stored per version inside the bundle, and this value
> selects the folder the loader points at. Anything other than an exact
> `major.minor` names a folder that doesn't exist, and the container starts with
> no agent at all.

> [!NOTE]
>
> **Go can't be instrumented on Fargate at all** — not an omission, a platform
> limit. Go's only zero-code instrumentation is eBPF-based, and Fargate doesn't
> allow the privileged access eBPF needs. Use the OpenTelemetry Go SDK in your code
> instead.

## Naming your service

`OTEL_SERVICE_NAME` here **must be the ECS service's name** — not the task definition family, and not the
container name. Odigos compares it against the workload's name, and a mismatch is reported as drift rather
than treated as a naming preference. There's no default, so you set one of these two:

**service** `ecs.BaseService`

The ECS service running this task definition. Only its name is read.

**serviceName** `string`

The name directly — use this when the service is defined somewhere else (another stack, Terraform, or the
console).

> [!WARNING]
>
> The service's name has to be one your stack **sets explicitly**. A name
> CloudFormation generates is a reference to the service, and a task definition
> referring back to the service that runs it is a circular dependency
> CloudFormation rejects. A generated name stops the build and points you at
> `serviceName`.

## Variables your container already sets

For Java, Node.js, and Python, the construct reads whatever your container already has in its loader
variable and **adds to it** — joined by that language's separator (a space for Java and Node.js, a colon
for Python):

```typescript theme={null}
taskDefinition.addContainer('App', {
  image,
  environment: { JAVA_TOOL_OPTIONS: '-Xmx512m' },
});
// result: -Xmx512m -javaagent:/var/odigos/java/javaagent.jar
```

**existingLoaderValue** `string`

Only needed as an override — pass this when the existing value isn't visible to CDK at build time,
because it comes from an `environmentFiles` entry, the image's own `ENV`, or `secrets`.

> [!WARNING]
>
> **PHP and Ruby work differently.** Odigos compares `PHP_INI_SCAN_DIR` and
> `RUBYOPT` exactly, so they're set outright rather than added to. If your container
> already sets one, you get a build warning — fold whatever your application needs
> into the value Odigos sets.

## Transport

Fargate accepts **both** endpoint formats at once, and often should get both:

```typescript theme={null}
destination: {
  endpoint: 'otlp.example.com:4317',              // OTLP/gRPC
  httpEndpoint: 'https://otlp.example.com:4318',  // OTLP/HTTP
}
```

Every agent in the bundle prefers HTTP. Java, Python, Node.js, and .NET will fall back to gRPC if that's all
you provide. Giving both lets each language use what it prefers.

> [!WARNING]
>
> **PHP and Ruby can only speak HTTP** —
> Instrumenting a PHP or Ruby container against a gRPC-only destination stops the
> build, rather than deploying a task that silently sends nothing.

## Instrumenting several containers

A task definition holding more than one application — in more than one language — is a normal shape. Call
`instrument()` once per container, or use `instrumentAll()`:

```typescript theme={null}
odigos.instrumentAll([
  { container: apiContainer,    language: OdigosFargateLanguage.JAVA,   service: svc },
  { container: workerContainer, language: OdigosFargateLanguage.PYTHON, service: svc },
]);
```

There's still only **one** copy-step container for the whole task definition — it just copies every language
that's needed.

> [!NOTE]
>
> AWS allows at most 10 containers per task definition, and the copy step needs one
> of them. A task definition already at the limit stops the build, rather than
> failing later during deployment.

## The agents image

The copy step pulls its files from an image. Leave this unset unless your tasks cannot reach the public
registry — the construct defaults to `registry.odigos.io/odigos-fargate-agents` at the tag this package
release ships (multi-arch: Intel and ARM/Graviton).

> [!WARNING]
>
> **This image is pulled by your task's execution role every time a task starts.** A
> task in a private subnet with no NAT gateway and no registry endpoint can't reach
> the default public registry, and the pull failure will block your application
> container from starting.

**agentsImage** `string`

Only set this if tasks cannot pull from `registry.odigos.io` (or your org requires images to live in a
registry you own). Point the copy step at your own mirror:

```typescript theme={null}
agentsImage: '111122223333.dkr.ecr.eu-west-1.amazonaws.com/odigos-fargate-agents:v0.0.15',
```

See [Preload the Fargate agents image](/cloud-connectors/aws/workloads/fargate/preload) for how to build
that mirror.

The image is a normal container image whose filesystem holds the language agents (under
`/instrumentations/<language>/…`). The construct sets environment variables that point at those exact
paths after the copy step stages them into `/var/odigos`. A private-registry override must therefore be a
**byte-for-byte mirror** of that image — retag and push the published `odigos-fargate-agents` image; do not
rebuild or rearrange it. See [Preload the Fargate agents image](/cloud-connectors/aws/workloads/fargate/preload)
for the mirror steps (and [Amazon ECR private registries](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html)
if you need the AWS side of hosting it).

The agents themselves are Linux and glibc only: a Windows task definition stops the build, and Alpine/musl
application images have no .NET profiler available.

Whichever image you use is recorded in the `ODIGOS_IAC` marker, and Odigos holds the task to *that* image
rather than to whatever the connector's own build happens to use. Two useful consequences:

* Your mirror is judged against itself, not flagged for differing from the default.
* Upgrading this package ahead of the connector isn't drift either — the service stays conformant until you
  redeploy.

> [!NOTE]
>
> What *is* reported: two containers in the same task definition recording different
> agents images. There's one copy step and it can't stage two different images.

## API reference

| Export | What it is |
| - | - |
| `OdigosFargateInstrumentation` | The construct. Methods: `instrument(target)`, `instrumentAll(targets)`. |
| `OdigosFargateTarget` | One container to instrument: `container`, `language`, `service` or `serviceName`, `runtimeVersion`, `existingLoaderValue`. |
| `OdigosFargateLanguage` | `JAVA` \| `PYTHON` \| `NODEJS` \| `DOTNET` \| `PHP` \| `RUBY` |
| `sidecarCommand`, `evaluateAppendPattern`, `appendSeparator` | The copy-step command and loader-value rendering, if you want to inspect what they produce. |
| `FARGATE_AGENTS_IMAGE` | The agents image this package version uses by default. |
| `FARGATE_DISTROS_VERSION` | The Odigos distros release the activation recipes were mirrored from. |

## Next steps

> **[CDK overview](/cloud-connectors/aws/cdk/overview)**
>
> Install, constructs, and how Odigos recognizes instrumented workloads.

> **[Destinations](/cloud-connectors/aws/cdk/destinations)**
>
> Endpoint formats, credentials, and which signals to enable.

> **[Preload the agents image](/cloud-connectors/aws/workloads/fargate/preload)**
>
> Mirror the image into a registry your tasks can reach.


# CDK: Lambda
Source: https://docs.odigos.io/cloud-connectors/aws/cdk/lambda

Instrument AWS Lambda functions from your CDK app with OdigosLambdaInstrumentation — every option, and the edge cases to know about.

`OdigosLambdaInstrumentation` instruments AWS Lambda functions defined in your CDK app.

> [!NOTE]
>
> Read [Odigos AWS CDK](/cloud-connectors/aws/cdk/overview) first — it covers
> installing the package and how Odigos recognizes an instrumented workload — then
> [Destinations](/cloud-connectors/aws/cdk/destinations) for the shared `destination`
> options. This page covers only what's specific to Lambda.

## Basic usage

```typescript theme={null}
import { OdigosLambdaInstrumentation } from '@odigos/aws-cdk';

new OdigosLambdaInstrumentation(this, 'Odigos', {
  destination: { endpoint: 'otlp.example.com:4317' },
  functions: [apiFn, workerFn],
});
```

Typically you define **one** `OdigosLambdaInstrumentation` per CDK stack and pass it every function you
want instrumented. One construct does everything for every function you list: attaches the right
OpenTelemetry layer for that function's language and CPU architecture, sets the variable that loads it,
configures where telemetry goes, and writes the `ODIGOS_IAC` marker.

`serviceName` (and other construct-level options) apply to **every** function on that construct. If each
function needs its own `serviceName`, define **one construct per function** instead of listing them
together. The same applies whenever any other setting must differ. To instrument every function in the
stack without listing them, use the [Aspect](#instrumenting-every-function-in-a-stack) instead.

Languages supported: **Python, Node.js, Java, Ruby.** Other runtimes — Go, .NET, and custom runtimes —
have no OpenTelemetry Lambda layer, so they can't be instrumented this way.

> [!NOTE]
>
> You can also call `instrument(fn)` or `instrumentAll([fn1, fn2])` on the construct
> after creating it, instead of passing `functions` up front.

## How telemetry leaves the function

This is worth understanding because it explains several of the options below.

Your function's OpenTelemetry SDK does **not** send data to your destination directly. It sends it to a
small collector that ships inside the Odigos layer and runs in the same Lambda sandbox, reachable on
`localhost`. That collector then forwards the data to your real destination — **after** your handler has
already returned its response to the caller.

That extra hop is the entire point: if your destination is slow or briefly unreachable, the delay lands on
billed duration rather than on the response your user is waiting for. (A `decouple` processor inside the
collector keeps the sandbox alive just long enough to finish sending before Lambda freezes it, so a
low-traffic function doesn't lose its telemetry.)

> [!WARNING]
>
> This is why the construct never sets `OTEL_EXPORTER_OTLP_ENDPOINT`. If you set it
> yourself, you point the SDK straight at your destination and put the network call
> back on the critical path of every single invocation — undoing the whole design.

## Where the layer comes from

**layerSource** `OdigosLayerSource` (default: ODIGOS\_PUBLIC)

Which copy of the OpenTelemetry layer your functions attach.

| | `ODIGOS_PUBLIC` (default) | `IN_ACCOUNT` |
| - | - | - |
| What it adds to your stack | Just a reference to a layer | A nested stack per language/architecture in use |
| Who owns the layer | Odigos' AWS account | **Your** account, created by your own stack |
| When it's resolved | At build time, from a table inside the package | At deploy time, from AWS Serverless Application Repository |
| Works without a fixed region | No — needs a concrete region | Yes |
| First deploy | Unchanged | Slower |

```typescript theme={null}
new OdigosLambdaInstrumentation(this, 'Odigos', {
  destination: { endpoint: 'otlp.example.com:4317' },
  functions: [fn],
  layerSource: OdigosLayerSource.IN_ACCOUNT,
});
```

Both options install byte-for-byte identical instrumentation. **`IN_ACCOUNT` is the SAR preload path for
CDK**: the construct nests an AWS Serverless Application Repository application into your stack, which
publishes an `AWS::Lambda::LayerVersion` owned by **your** account and wires its ARN onto the function. You
do not deploy SAR by hand when using this option.

Use `IN_ACCOUNT` if your organization's policy forbids attaching a Lambda layer owned by an outside AWS
account. It needs the principal running `cdk deploy` to hold `serverlessrepo:CreateCloudFormationTemplate`
and `serverlessrepo:GetApplication`.

For the same preload outside CDK (Console, CloudFormation/SAM) — or to see the Application IDs and
`SemanticVersion` the construct pins — see
[Preload the Lambda layer](/cloud-connectors/aws/workloads/lambda/preload).

**layerVersion** `lambda.ILayerVersion`

Escape hatch: attach a specific layer you already own, and **skip `layerSource` entirely** — neither
`ODIGOS_PUBLIC` nor `IN_ACCOUNT` is consulted when this is set.

Pass any CDK [`ILayerVersion`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.ILayerVersion.html)
— typically `LayerVersion.fromLayerVersionArn(...)` pointing at a layer you mirrored into your account, or
a `LayerVersion` construct you already defined. There is no separate allow-list: the construct attaches
whatever you pass.

The layer **must** be the Odigos OpenTelemetry layer (same bytes / script layout as the public or SAR
layer). Environment variables assume paths inside that layout; a different layer will attach but fail to
instrument at runtime.

```typescript theme={null}
import * as lambda from 'aws-cdk-lib/aws-lambda';

new OdigosLambdaInstrumentation(this, 'Odigos', {
  destination: { endpoint: 'otlp.example.com:4317' },
  functions: [fn],
  layerVersion: lambda.LayerVersion.fromLayerVersionArn(
    this,
    'OdigosLayer',
    'arn:aws:lambda:eu-west-1:111122223333:layer:odigos-otel-python-arm64:1',
  ),
});
```

One `layerVersion` is used for **every** function on this construct, so those functions must share a
language and architecture that match that layer. Prefer `layerSource` when you need the construct to pick
the right layer per function.

## Normal functions vs. web-server functions

**mode** `OdigosInstrumentationMode` (default: HANDLER)

`HANDLER` is for an ordinary Lambda function — one that exports a handler and gets invoked per event.

`WEB_SERVER` is for a function that runs an actual web server behind the
[AWS Lambda Web Adapter](https://github.com/awslabs/aws-lambda-web-adapter).

The difference matters because both Odigos and the Web Adapter want to use the same Lambda setting
(`AWS_LAMBDA_EXEC_WRAPPER`) to hook into startup. In `WEB_SERVER` mode, Odigos leaves that setting to the
adapter and loads instrumentation a different way instead.

> [!WARNING]
>
> **Using `HANDLER` on a web-adapter function overwrites the adapter's setting and
> the function stops serving traffic.** There's deliberately no auto-detection:
> spotting the adapter would mean inspecting the function's other layers, which CDK
> often can't resolve until deploy time — and a detector that was right most of the
> time would silently break exactly the functions it got wrong. So you state it.

```typescript theme={null}
new OdigosLambdaInstrumentation(this, 'Odigos', {
  destination: { endpoint: 'otlp.example.com:4317' },
  functions: [webFn],
  mode: OdigosInstrumentationMode.WEB_SERVER,
  existingLoaderValue: '--max-old-space-size=2048',
});
```

**existingLoaderValue** `string`

In `WEB_SERVER` mode, Odigos loads instrumentation through your language's own startup variable —
`NODE_OPTIONS`, `JAVA_TOOL_OPTIONS`, or `PYTHONPATH`. If your function already needs a value in that
variable, pass it here and Odigos adds to it instead of replacing it.

> [!WARNING]
>
> CDK can't read back an environment variable your app set elsewhere, so this
> can't be detected automatically. Omit it and the variable is set to **only**
> the Odigos value — for `NODE_OPTIONS`, that can silently drop a flag your
> application needs to start.

> [!NOTE]
>
> Ruby functions are not supported in WEB\_SERVER mode because a verified web-server loader is currently unavailable.

## Java functions: pick the matching handler type

The Java layer ships four different wrapper scripts, one per Lambda handler interface. This is a
correctness setting, not a level-of-detail setting.

**javaHandlerDistro** `OdigosJavaHandlerDistro` (default: OTEL\_HANDLER)

```typescript theme={null}
javaHandlerDistro: OdigosJavaHandlerDistro.OTEL_STREAM_HANDLER
```

| Value | Use it for |
| - | - |
| `OTEL_HANDLER` (default) | A `RequestHandler` function |
| `OTEL_PROXY_HANDLER` | API Gateway proxy events — produces richer spans |
| `OTEL_SQS_HANDLER` | SQS events — produces richer spans |
| `OTEL_STREAM_HANDLER` | A `RequestStreamHandler` function |

> [!WARNING]
>
> The default installs a `RequestHandler` implementation. A function that actually
> implements `RequestStreamHandler` has a completely different method signature, so
> the default **breaks the invocation** — it doesn't merely produce thinner traces.

## Instrumenting every function in a stack

```typescript theme={null}
Aspects.of(stack).add(new OdigosInstrumentationAspect(stack, {
  destination: { endpoint: 'otlp.example.com:4317' },
}));
```

This visits every Lambda function in the stack.

**skipUnsupportedRuntimes** `boolean` (default: false (true on the aspect))

What to do when a function's runtime has no OpenTelemetry layer (Go, .NET, custom runtimes).

The aspect defaults to `true`: skip it with a build **warning** rather than failing. The single-function
construct defaults to `false`, because naming one specific Go function and asking for it to be
instrumented is a mistake worth stopping the build over.

## Naming your service

`OTEL_SERVICE_NAME` is what your function shows up as in your observability tool.

**serviceName** `string`

Left unset, this becomes the function's real name when your stack sets one explicitly, and the CDK
construct ID otherwise.

That fallback exists for a structural reason: a function name that CloudFormation generates for you is a
reference to the function itself, and a function referring to its own generated name inside its own
configuration is a circular dependency that CloudFormation refuses to deploy.

> [!WARNING]
>
> Odigos compares this against the function's real AWS name. **If your function's
> name is auto-generated, set `serviceName` explicitly** — otherwise Odigos reports
> it as drifted. Note that `serviceName` applies to *every* function in a single
> construct, so use one construct per function when you need different names.

## Other things worth knowing

> **Functions this construct can't instrument**
>
> A container-image function (`DockerImageFunction`) can't carry Lambda layers at all. The construct also
> needs a function your own stack owns — one imported with `Function.fromFunctionArn` can't be modified
> by CDK.

> **Layer limits**
>
> AWS allows a maximum of 5 layers per function. Also don't attach the Odigos layer through more than one
> route at once — two versions of the same layer is reported as a conflict.

## API reference

| Export | What it is |
| - | - |
| `OdigosLambdaInstrumentation` | The construct. Methods: `instrument(fn)`, `instrumentAll(fns)`. |
| `OdigosInstrumentationAspect` | Applies instrumentation to every function in a scope. |
| `OdigosLayerSource` | `ODIGOS_PUBLIC` \| `IN_ACCOUNT` |
| `OdigosInstrumentationMode` | `HANDLER` \| `WEB_SERVER` |
| `OdigosJavaHandlerDistro` | The four Java handler wrappers. |
| `renderCollectorConfig` | The same collector-config renderer the construct uses, if you want to inspect what it produces. |
| `LAYER_ARNS`, `ODIGOS_VERSION` | The public layer-ARN table baked into the package (`ODIGOS_PUBLIC`), and the Odigos release it came from. |
| `SAR_APPLICATIONS`, `SAR_SEMANTIC_VERSION` | What `IN_ACCOUNT` deploys under the hood: SAR Application IDs keyed by `language/architecture`, and the pinned semantic version (same bytes as this package release). You normally do **not** import these — set `layerSource: IN_ACCOUNT` instead. Use them only if you're authoring your own SAR/`AWS::Serverless::Application` template; the step-by-step is in [Preload the Lambda layer](/cloud-connectors/aws/workloads/lambda/preload). |

## Related Topics

> **[CDK overview](/cloud-connectors/aws/cdk/overview)**
>
> Install, constructs, and how Odigos recognizes instrumented workloads.

> **[Destinations](/cloud-connectors/aws/cdk/destinations)**
>
> Endpoint formats, credentials, and which signals to enable.

> **[Preload the layer](/cloud-connectors/aws/workloads/lambda/preload)**
>
> Put the layer in your own account for `IN_ACCOUNT`.


# Odigos AWS CDK
Source: https://docs.odigos.io/cloud-connectors/aws/cdk/overview

What the Odigos AWS CDK package is, when to use it, and how Odigos recognizes instrumented workloads.

The Odigos AWS CDK package adds OpenTelemetry instrumentation to your AWS workloads from inside your own
[AWS CDK](https://aws.amazon.com/cdk/) application — so instrumentation ships through the same pipeline as
the rest of your infrastructure, reviewed in a pull request and applied by `cdk deploy`.

> [!NOTE]
>
> **New to this?** AWS CDK is a tool for describing cloud infrastructure in a
> programming language instead of a YAML template. If your team doesn't use CDK,
> you don't need this section — use the CloudFormation instructions on
> [Instrument Lambda](/cloud-connectors/aws/workloads/lambda/instrument) or
> [Instrument Fargate](/cloud-connectors/aws/workloads/fargate/instrument), or let
> Odigos apply instrumentation for you.

## When you'd use this

Odigos can instrument your workloads two ways, and the CDK package is for the second one:

| | Odigos-managed | Self-managed, with this package |
| - | - | - |
| Who changes the workload | Odigos, automatically | Your CDK app, on `cdk deploy` |
| Where instrumentation is defined | In Odigos | In your own source code |
| What Odigos does | Applies and maintains it | Watches and reports |

Pick self-managed when your infrastructure code should stay the single source of truth for what's deployed.
Pick Odigos-managed when you'd rather not touch your infrastructure code at all. Full comparison:
[Odigos-managed vs. self-managed instrumentation](/cloud-connectors/overview#odigos-managed-vs-self-managed-instrumentation).

> [!WARNING]
>
> Don't do both to the same workload. If a function or service is instrumented by
> your CDK app **and** also enabled as an Odigos-managed source, each side keeps
> reasserting its own configuration — every deploy fights every reconcile, and each
> round trip creates a new Lambda version or ECS task revision. Pick one per
> workload.

## Install

```bash theme={null}
npm install @odigos/aws-cdk
```

> [!NOTE]
>
> **TypeScript and JavaScript only, for now.** A Python distribution
> (`odigos-aws-cdk`) is built but not yet published — it's waiting on package-index
> approval. Until it's out, Python CDK apps should use the
> [CloudFormation instructions](/cloud-connectors/aws/workloads/lambda/instrument#using-cloudformation-directly)
> instead.

Pin an exact version. **The package version *is* the instrumentation version** — `@odigos/aws-cdk@0.0.15`
installs the OpenTelemetry layer and agents bundle built by Odigos release `v0.0.15`, and nothing resolves
or moves at deploy time. Upgrading is a deliberate `npm update` followed by a `cdk diff` that shows you
exactly what changes. An instrumentation agent that silently updated itself under a deployment that changed
nothing is not a property you want.

## The two constructs

| | Lambda | ECS Fargate |
| - | - | - |
| Construct | `OdigosLambdaInstrumentation` | `OdigosFargateInstrumentation` |
| How the agent gets there | A Lambda layer attached to the function | Copied into the task by an extra short-lived container |
| Languages | Python, Node.js, Java, Ruby | Java, Python, Node.js, .NET, PHP, Ruby |
| Where telemetry goes | To a collector inside the same Lambda sandbox, which forwards it on after your handler returns | Straight from your application to your destination |
| How the language is determined | Automatically, from the function's runtime | You state it explicitly |

> **[Lambda construct](/cloud-connectors/aws/cdk/lambda)**
>
> Every option for instrumenting Lambda functions.

> **[Fargate construct](/cloud-connectors/aws/cdk/fargate)**
>
> Every option for instrumenting ECS Fargate services.

Configure where telemetry goes and which signals to enable on
[CDK: Destinations](/cloud-connectors/aws/cdk/destinations) — both constructs share the same `destination`
options.

## How Odigos recognizes your workload

Both constructs write an environment variable called **`ODIGOS_IAC`** onto everything they instrument. This
is the one thing that tells Odigos "this workload is instrumented, and here's what with."

> [!WARNING]
>
> Without that marker, Odigos reports the workload as **awaiting your deployment** —
> no matter how completely everything else was applied. The constructs write it for
> you, so there's nothing to switch on; it matters mainly because you'll see it in
> a `cdk diff`, and because hand-written instrumentation has to include it too.

The marker also records *which version* of instrumentation you deployed — the layer ARN on Lambda, the
agents image on Fargate. That's what lets you pin a version and have Odigos respect it, rather than
reporting your deliberately-pinned workload as out of date the day Odigos publishes something newer.

> [!WARNING]
>
> There's a second marker, **`ODIGOS_MANAGED`**, that these constructs never write —
> and neither should you. That one is Odigos' own record of a change *it* made,
> including the snapshot it would undo that change with. Writing it by hand claims a
> rollback point that doesn't exist.

## Turning on debug logging

**debug** `boolean` (default: false)

Makes the OpenTelemetry agent log verbosely, which helps when telemetry isn't arriving and you need to
see why. Available on both constructs.

> [!WARNING]
>
> Leave it off once you're done. It's expensive at volume, and Odigos expects it
> **off** unless debug telemetry is also enabled on the connector — so leaving it
> on can itself be reported as drift.

## Next steps

> **[Destinations](/cloud-connectors/aws/cdk/destinations)**
>
> Endpoint formats, credentials, and which signals to enable.

> **[Lambda construct](/cloud-connectors/aws/cdk/lambda)**
>
> Layer sources, web-server mode, Java handler interfaces.

> **[Fargate construct](/cloud-connectors/aws/cdk/fargate)**
>
> Languages, service naming, the agents image.


# AWS Connector
Source: https://docs.odigos.io/cloud-connectors/aws/overview

How the AWS Cloud Connector works end to end: resource types, Odigos-managed vs. self-managed instrumentation, languages, and where to go next.

The AWS Connector is one running instance that owns **one AWS account**. It scans every region enabled in
that account, discovers Lambda functions and ECS/Fargate resources, and — for the types you allow — either
instruments them for you or reports on instrumentation you applied yourself.

This page is the map. For hands-on tasks, see:

> **[Add an AWS Connector](/cloud-connectors/add-connectors/add-aws-connector)**
>
> Connect an account, choose an access level, and verify discovery.

> **[Preload Lambda layer](/cloud-connectors/aws/workloads/lambda/preload)**
>
> Preload the Lambda layer into your own account for IaC.

> **[Preload Fargate image](/cloud-connectors/aws/workloads/fargate/preload)**
>
> Mirror the Fargate agents image into a private registry.

> **[Instrument Lambda](/cloud-connectors/aws/workloads/lambda/instrument)**
>
> Self-managed Lambda instrumentation for Java, Python, Node.js, and Ruby.

> **[Instrument Fargate](/cloud-connectors/aws/workloads/fargate/instrument)**
>
> Self-managed Fargate instrumentation with CloudFormation for every supported language.

> **[CDK reference: Lambda](/cloud-connectors/aws/cdk/lambda)**
>
> Every option on the Lambda CDK construct, in depth.

> **[CDK reference: Fargate](/cloud-connectors/aws/cdk/fargate)**
>
> Every option on the Fargate CDK construct, in depth.

## Resource types

The connector splits AWS resources into two domains — **workloads**, which it can instrument directly, and
**compute platforms**, which it can only discover today:

| Type | Domain | Discovery | Instrumentation |
| - | - | - | - |
| `aws.lambda` | Workload | Yes | Yes — Odigos-managed or self-managed |
| `aws.fargate-task` | Workload | Yes | Yes — Odigos-managed or self-managed |
| `aws.ecs-cluster` | Compute platform | Yes | Coming soon (agent install) |
| `aws.fargate-cluster` | Compute platform | Yes | n/a (coverage only) |

> [!NOTE]
>
> `aws.fargate-task` means an ECS **service** running on Fargate — the service, not
> the task, is the unit of instrumentation, because task ARNs are recycled on every
> deploy while the service is a stable identity. A standalone task started outside
> a service (for example by `RunTask` or EventBridge) can be discovered but cannot
> be instrumented — there is no service to roll a new revision onto.

### EC2-launch-type ECS is a separate story

Everything on this page and in the AWS Connector section covers **Fargate**-launch-type ECS. EC2-launch-type
ECS clusters are discovered as `aws.ecs-cluster` compute platforms, but Fargate forbids the privileged
containers eBPF instrumentation needs, so EC2-backed clusters are covered by a different, cluster-wide agent
rather than the per-service sidecar described here. That agent's automatic deployment is not yet available —
today `aws.ecs-cluster` gives you discovery and coverage only.

## Odigos-managed vs. self-managed

Both workload types support two independent modes, chosen per type when you add or edit the connector. This
is a connector-wide concept — see
[Odigos-managed vs. self-managed instrumentation](/cloud-connectors/overview#odigos-managed-vs-self-managed-instrumentation)
for the general rules (the marker system, why you should pick one mode per workload, and so on). On AWS
specifically:

| Mode | Lambda | Fargate |
| - | - | - |
| **Odigos-managed** | The connector attaches the OpenTelemetry layer and sets the required environment directly on the function, and keeps it there. | The connector injects an `odigos-agent` sidecar into the service's task definition and rolls the service to the new revision, then keeps reconciling it. |
| **Self-managed (IaC)** | You attach the layer and environment yourself via CloudFormation, the Odigos CDK construct, Terraform, or the AWS CLI. | You add the sidecar and environment yourself via CloudFormation or the Odigos CDK construct (`@odigos/aws-cdk`). |

Self-managed is the default the moment you enable discovery without instrumentation for a type — nothing
extra to configure. See [Instrument Lambda](/cloud-connectors/aws/workloads/lambda/instrument) and
[Instrument Fargate](/cloud-connectors/aws/workloads/fargate/instrument) for the actual snippets, and
[Preload Lambda layer](/cloud-connectors/aws/workloads/lambda/preload) /
[Preload Fargate image](/cloud-connectors/aws/workloads/fargate/preload) for getting artifacts into your own
account first if your organization requires that.

## Supported languages

| Workload | Languages | Notes |
| - | - | - |
| Lambda (`aws.lambda`) | Java, Python, Node.js, Ruby | Ruby: OTLP HTTP destinations only. Other runtimes (Go, .NET) are discovered but not instrumented. |
| Fargate (`aws.fargate-task`) | Java, Python, Node.js, .NET, PHP, Ruby | PHP and Ruby: OTLP HTTP destinations only. .NET requires a glibc (not musl/Alpine) base image. Go and any other eBPF-based language cannot be instrumented on Fargate at all. |

**If a task runs more than one container** — for example, an nginx or Envoy proxy sitting in front of your
application — Odigos has to pick which one is the actual application to instrument. It does this by looking
at what's really running inside each container, not by which one has a public port. That distinction
matters: the proxy is usually the container with the port mapping, but the proxy has no code to instrument
— it's the application container behind it that needs the agent. Picking by port would target the wrong
container entirely.

If Odigos can't confidently identify exactly one application container — none of them look like an app, or
more than one equally do — it marks the task **unsupported** rather than guessing. The same happens if the
task definition is already at AWS's 10-container limit, since there's no room left for the extra container
Odigos needs to add.

## Telemetry export

Both Lambda and Fargate export OTLP only, and the destination must be reachable from the workload itself:

* **Lambda** ships an in-process collector inside the layer. Your function's SDK exports to `localhost`
  inside the same sandbox; the collector forwards to your configured destination *after* the handler
  returns, so a slow or unreachable destination costs billed duration, not response latency.
* **Fargate** has no node-local collector — each instrumented task exports **directly** to the destination
  over its own network path. The destination endpoint must be reachable from the task's VPC (public IP or
  NAT for a registry pull, plus whatever egress the destination itself needs).

## Next steps

Self-managed instrumentation — pick your workload:

> **[Instrument Lambda](/cloud-connectors/aws/workloads/lambda/instrument)**
>
> CloudFormation or CDK for Lambda functions.

> **[Instrument Fargate](/cloud-connectors/aws/workloads/fargate/instrument)**
>
> CloudFormation for ECS Fargate task definitions.

Using `@odigos/aws-cdk`? Start from the
[CDK overview](/cloud-connectors/aws/cdk/overview) (install, destinations, markers), then the Lambda or
Fargate construct reference.

Still setting up the connector? See
[Add an AWS Connector](/cloud-connectors/add-connectors/add-aws-connector). For Odigos-managed Sources after
the connector is online, create them in Central once an OTLP destination is configured.


# Instrument Fargate with CloudFormation
Source: https://docs.odigos.io/cloud-connectors/aws/workloads/fargate/instrument

Self-managed ECS Fargate instrumentation for Java, Python, Node.js, .NET, PHP, and Ruby using hand-written CloudFormation.

This page shows how to instrument an ECS **service** running on Fargate yourself via CloudFormation, by editing its task
definition directly — no changes made by the connector on your behalf. Odigos still discovers the service
and reports its status once your changes are in place; see
[Odigos-managed vs. self-managed instrumentation](/cloud-connectors/overview#odigos-managed-vs-self-managed-instrumentation)
for how that works.

> [!NOTE]
>
> Prefer the Odigos CDK construct? See
> [CDK: Fargate](/cloud-connectors/aws/cdk/fargate). This page is the
> hand-written CloudFormation / raw ECS form of the same change.

## How it works

Self-managed Fargate instrumentation adds one extra container to your task definition, plus a handful of
environment variables on your application container. Here's what happens each time the task starts:

1. A small helper container, **`odigos-agent`**, starts first. It copies the files your application's
   language agent needs into a shared volume, then exits. It's marked non-essential, since its only job is
   to prepare files — it isn't part of your running application.
2. Your **application container** waits for step 1 to finish successfully before it starts
   (`DependsOn: ... Condition: SUCCESS`) — this guarantees the agent files are already in place by the time
   your code runs.
3. Your application container picks up the agent through environment variables specific to its language —
   for example `JAVA_TOOL_OPTIONS` for Java — and begins exporting telemetry.

> [!NOTE]
>
> There's no intermediate collector inside the task — each Fargate task sends
> telemetry **directly** to your OTLP destination, so that destination's endpoint
> must be reachable from the task's network. See
> [Telemetry export](/cloud-connectors/aws/overview#telemetry-export).

## CloudFormation

Merge these properties into your task definition. Only the exact agent path, activation variable, and network protocol change per language. Pick yours below:

**Java**

```yaml theme={null}
# Merge into Resources.<YourTaskDefinition>.Properties
Volumes:
  - Name: odigos-agent   # ephemeral-storage bind mount, scoped to the task
ContainerDefinitions:
  - Name: odigos-agent
    Image: "registry.odigos.io/odigos-fargate-agents:<version>"
    Essential: false
    Command: ["sh", "-c", "mkdir -p /var/odigos && cp -r /instrumentations/java /var/odigos/java"]
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
  - Name: "<YOUR_CONTAINER>"
    # ... your existing configuration ...
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
        ReadOnly: true
    DependsOn:
      - ContainerName: odigos-agent
        Condition: SUCCESS
    Environment:
      - Name: ODIGOS_IAC
        Value: '{"v":1,"lang":"java"}'
      - Name: OTEL_SERVICE_NAME
        Value: "<YOUR_SERVICE_NAME>"
      - Name: OTEL_RESOURCE_ATTRIBUTES
        Value: "cloud.provider=aws,cloud.platform=aws_ecs,cloud.region=<REGION>"
      - Name: OTEL_EXPORTER_OTLP_ENDPOINT
        Value: "<OTLP_ENDPOINT_URL>"
      - Name: OTEL_EXPORTER_OTLP_PROTOCOL
        Value: "http/protobuf"   # or "grpc" — must match your destination; see note below
      - Name: OTEL_TRACES_EXPORTER
        Value: "otlp"
      - Name: OTEL_METRICS_EXPORTER
        Value: "otlp"
      - Name: OTEL_LOGS_EXPORTER
        Value: "otlp"
      # APPEND after any existing value, joined by a space — assigning discards yours.
      - Name: JAVA_TOOL_OPTIONS
        Value: "<YOUR_EXISTING_VALUE> -javaagent:/var/odigos/java/javaagent.jar"
```

**Python**

```yaml theme={null}
# Merge into Resources.<YourTaskDefinition>.Properties
Volumes:
  - Name: odigos-agent
ContainerDefinitions:
  - Name: odigos-agent
    Image: "registry.odigos.io/odigos-fargate-agents:<version>"
    Essential: false
    Command: ["sh", "-c", "mkdir -p /var/odigos && cp -r /instrumentations/python /var/odigos/python"]
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
  - Name: "<YOUR_CONTAINER>"
    # ... your existing configuration ...
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
        ReadOnly: true
    DependsOn:
      - ContainerName: odigos-agent
        Condition: SUCCESS
    Environment:
      - Name: ODIGOS_IAC
        Value: '{"v":1,"lang":"python"}'
      - Name: OTEL_SERVICE_NAME
        Value: "<YOUR_SERVICE_NAME>"
      - Name: OTEL_RESOURCE_ATTRIBUTES
        Value: "cloud.provider=aws,cloud.platform=aws_ecs,cloud.region=<REGION>"
      - Name: OTEL_EXPORTER_OTLP_ENDPOINT
        Value: "<OTLP_ENDPOINT_URL>"
      - Name: OTEL_EXPORTER_OTLP_PROTOCOL
        Value: "http/protobuf"
      - Name: OTEL_TRACES_EXPORTER
        Value: "otlp"
      - Name: OTEL_METRICS_EXPORTER
        Value: "otlp"
      - Name: OTEL_LOGS_EXPORTER
        Value: "otlp"
      - Name: OTEL_PYTHON_CONFIGURATOR
        Value: "odigos-python-configurator"
      # APPEND after any existing value, joined by ":" — assigning discards yours.
      - Name: PYTHONPATH
        Value: "<YOUR_EXISTING_VALUE>:/var/odigos/python:/var/odigos/python/opentelemetry/instrumentation/auto_instrumentation"
```

**Node.js**

```yaml theme={null}
# Merge into Resources.<YourTaskDefinition>.Properties
Volumes:
  - Name: odigos-agent
ContainerDefinitions:
  - Name: odigos-agent
    Image: "registry.odigos.io/odigos-fargate-agents:<version>"
    Essential: false
    Command: ["sh", "-c", "mkdir -p /var/odigos && cp -r /instrumentations/nodejs /var/odigos/nodejs"]
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
  - Name: "<YOUR_CONTAINER>"
    # ... your existing configuration ...
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
        ReadOnly: true
    DependsOn:
      - ContainerName: odigos-agent
        Condition: SUCCESS
    Environment:
      - Name: ODIGOS_IAC
        Value: '{"v":1,"lang":"nodejs"}'
      - Name: OTEL_SERVICE_NAME
        Value: "<YOUR_SERVICE_NAME>"
      - Name: OTEL_RESOURCE_ATTRIBUTES
        Value: "cloud.provider=aws,cloud.platform=aws_ecs,cloud.region=<REGION>"
      - Name: OTEL_EXPORTER_OTLP_ENDPOINT
        Value: "<OTLP_ENDPOINT_URL>"
      - Name: OTEL_EXPORTER_OTLP_PROTOCOL
        Value: "http/protobuf"
      - Name: OTEL_TRACES_EXPORTER
        Value: "otlp"
      - Name: OTEL_METRICS_EXPORTER
        Value: "otlp"
      - Name: OTEL_LOGS_EXPORTER
        Value: "otlp"
      # APPEND after any existing value, joined by a space — assigning discards yours.
      - Name: NODE_OPTIONS
        Value: "<YOUR_EXISTING_VALUE> --require /var/odigos/nodejs/node_modules/@opentelemetry/auto-instrumentations-node/build/src/register.js"
```

**.NET**

```yaml theme={null}
# Merge into Resources.<YourTaskDefinition>.Properties
Volumes:
  - Name: odigos-agent
ContainerDefinitions:
  - Name: odigos-agent
    Image: "registry.odigos.io/odigos-fargate-agents:<version>"
    Essential: false
    Command: ["sh", "-c", "mkdir -p /var/odigos && cp -r /instrumentations/dotnet /var/odigos/dotnet"]
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
  - Name: "<YOUR_CONTAINER>"
    # ... your existing configuration ...
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
        ReadOnly: true
    DependsOn:
      - ContainerName: odigos-agent
        Condition: SUCCESS
    Environment:
      - Name: ODIGOS_IAC
        Value: '{"v":1,"lang":"dotnet"}'
      - Name: OTEL_SERVICE_NAME
        Value: "<YOUR_SERVICE_NAME>"
      - Name: OTEL_RESOURCE_ATTRIBUTES
        Value: "cloud.provider=aws,cloud.platform=aws_ecs,cloud.region=<REGION>"
      - Name: OTEL_EXPORTER_OTLP_ENDPOINT
        Value: "<OTLP_ENDPOINT_URL>"
      - Name: OTEL_EXPORTER_OTLP_PROTOCOL
        Value: "http/protobuf"
      - Name: OTEL_TRACES_EXPORTER
        Value: "otlp"
      - Name: OTEL_METRICS_EXPORTER
        Value: "otlp"
      - Name: OTEL_LOGS_EXPORTER
        Value: "otlp"
      # --- .NET activation: every value below is SET exactly as shown, not appended ---
      - Name: CORECLR_ENABLE_PROFILING
        Value: "1"
      - Name: CORECLR_PROFILER
        Value: "{918728DD-259F-4A6A-AC2B-B85E1B658318}"
      - Name: CORECLR_PROFILER_PATH
        Value: "/var/odigos/dotnet/linux-glibc/OpenTelemetry.AutoInstrumentation.Native.so"
      - Name: OTEL_DOTNET_AUTO_HOME
        Value: "/var/odigos/dotnet"
      - Name: DOTNET_STARTUP_HOOKS
        Value: "/var/odigos/dotnet/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll"
      - Name: DOTNET_ADDITIONAL_DEPS
        Value: "/var/odigos/dotnet/AdditionalDeps"
      - Name: DOTNET_SHARED_STORE
        Value: "/var/odigos/dotnet/store"
```

> [!WARNING]
>
> The .NET agent is **glibc-only**. A musl-based image (Alpine) can't be
> instrumented this way — the profiler shared object won't load, and the
> container starts uninstrumented with no error.

**PHP**

```yaml theme={null}
# Merge into Resources.<YourTaskDefinition>.Properties
Volumes:
  - Name: odigos-agent
ContainerDefinitions:
  - Name: odigos-agent
    Image: "registry.odigos.io/odigos-fargate-agents:<version>"
    Essential: false
    Command: ["sh", "-c", "mkdir -p /var/odigos && cp -r /instrumentations/php /var/odigos/php"]
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
  - Name: "<YOUR_CONTAINER>"
    # ... your existing configuration ...
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
        ReadOnly: true
    DependsOn:
      - ContainerName: odigos-agent
        Condition: SUCCESS
    Environment:
      - Name: ODIGOS_IAC
        Value: '{"v":1,"lang":"php"}'
      - Name: OTEL_SERVICE_NAME
        Value: "<YOUR_SERVICE_NAME>"
      - Name: OTEL_RESOURCE_ATTRIBUTES
        Value: "cloud.provider=aws,cloud.platform=aws_ecs,cloud.region=<REGION>"
      - Name: OTEL_EXPORTER_OTLP_ENDPOINT
        Value: "<OTLP_ENDPOINT_URL>"     # must be an OTLP/HTTP endpoint — see note below
      - Name: OTEL_EXPORTER_OTLP_PROTOCOL
        Value: "http/protobuf"
      - Name: OTEL_TRACES_EXPORTER
        Value: "otlp"
      - Name: OTEL_METRICS_EXPORTER
        Value: "otlp"
      - Name: OTEL_LOGS_EXPORTER
        Value: "otlp"
      # --- PHP activation ---
      - Name: PHP_INI_SCAN_DIR
        Value: ":/var/odigos/php/<RUNTIME_VERSION>"   # e.g. 8.3 — must match your image's PHP version
      - Name: OTEL_PHP_AUTOLOAD_ENABLED
        Value: "true"
      - Name: OTEL_PROPAGATORS
        Value: "tracecontext,baggage"
```

> [!WARNING]
>
> The PHP agent speaks **OTLP/HTTP only** — it cannot use gRPC. Your OTLP
> destination must offer an HTTP endpoint (standard port 4318), or PHP services
> can't be instrumented at all. `<RUNTIME_VERSION>` must match your container's
> actual PHP major.minor (e.g. `8.3`) — set a runtime override on the Source so
> Odigos judges the container against the path you actually mounted.

**Ruby**

```yaml theme={null}
# Merge into Resources.<YourTaskDefinition>.Properties
Volumes:
  - Name: odigos-agent
ContainerDefinitions:
  - Name: odigos-agent
    Image: "registry.odigos.io/odigos-fargate-agents:<version>"
    Essential: false
    Command: ["sh", "-c", "mkdir -p /var/odigos && cp -r /instrumentations/ruby /var/odigos/ruby"]
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
  - Name: "<YOUR_CONTAINER>"
    # ... your existing configuration ...
    MountPoints:
      - SourceVolume: odigos-agent
        ContainerPath: /var/odigos
        ReadOnly: true
    DependsOn:
      - ContainerName: odigos-agent
        Condition: SUCCESS
    Environment:
      - Name: ODIGOS_IAC
        Value: '{"v":1,"lang":"ruby"}'
      - Name: OTEL_SERVICE_NAME
        Value: "<YOUR_SERVICE_NAME>"
      - Name: OTEL_RESOURCE_ATTRIBUTES
        Value: "cloud.provider=aws,cloud.platform=aws_ecs,cloud.region=<REGION>"
      - Name: OTEL_EXPORTER_OTLP_ENDPOINT
        Value: "<OTLP_ENDPOINT_URL>"     # must be an OTLP/HTTP endpoint — see note below
      - Name: OTEL_EXPORTER_OTLP_PROTOCOL
        Value: "http/protobuf"
      - Name: OTEL_TRACES_EXPORTER
        Value: "otlp"
      - Name: OTEL_METRICS_EXPORTER
        Value: "otlp"
      - Name: OTEL_LOGS_EXPORTER
        Value: "otlp"
      # --- Ruby activation: RUBYOPT is SET, replacing anything already there ---
      - Name: RUBYOPT
        Value: "-r/var/odigos/ruby/<RUNTIME_VERSION>/index.rb"   # e.g. 3.3
      - Name: ODIGOS_GEM_PATH
        Value: "/var/odigos/ruby/<RUNTIME_VERSION>/bundle"
      - Name: OTEL_PROPAGATORS
        Value: "tracecontext,baggage"
```

> [!WARNING]
>
> The Ruby agent speaks **OTLP/HTTP only** — it cannot use gRPC. Your OTLP
> destination must offer an HTTP endpoint (standard port 4318). `<RUNTIME_VERSION>`
> must match your container's actual Ruby major.minor (e.g. `3.3`); set a runtime
> override on the Source so Odigos judges the container against the path you
> actually mounted.

### Rules that apply to every language

> **ODIGOS\_IAC is per container**
>
> It's the only thing marking a container as instrumented, and its recorded language is what the
> activation environment is checked against. Never set `ODIGOS_MANAGED` — that's the connector's own
> rollback record, written only when you hand the workload to Odigos-managed instrumentation instead.

> **Endpoint and protocol are one decision**
>
> Both are compared exactly against what your OTLP destination resolves to on this connector. If it only
> offers OTLP/gRPC, use the gRPC endpoint (standard port 4317) and `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`
> instead of the `http/protobuf` shown above (except PHP and Ruby, which support HTTP only — see their
> tabs). Pairing an endpoint with the wrong transport fails at runtime as an opaque connection reset.

> **One odigos-agent container serves the whole task**
>
> Its `Command` is per task, not per language. A task definition with application containers in more than
> one language needs a chained `&& cp -r /instrumentations/<lang> /var/odigos/<lang>` for each, in
> alphabetical order — Odigos compares the command exactly against the union of every marked container's
> language.

> **Roll the service, not just the revision**
>
> Registering a new task-definition revision isn't enough on its own — update the service to it, or it
> keeps running the old one.

> **Constraints**
>
> A 10-container task-definition limit, Linux-only agents bundle (no Windows tasks), and the
> `odigos-agent` container must stay non-essential — ECS only accepts a `SUCCESS` dependency against a
> non-essential container.

> **Registry reachability**
>
> The image is pulled by the task's execution role at every task start. If your task's subnet can't reach
> `registry.odigos.io`, mirror the image into a registry it can — see
> [Preload artifacts](/cloud-connectors/aws/workloads/fargate/preload).

## Next steps

> **[Preload artifacts](/cloud-connectors/aws/workloads/fargate/preload)**
>
> Mirror the Fargate agents image into a private registry.

> **[CDK: Fargate](/cloud-connectors/aws/cdk/fargate)**
>
> Every option on the Fargate CDK construct, in depth.


# Preload the Fargate Agents Image
Source: https://docs.odigos.io/cloud-connectors/aws/workloads/fargate/preload

Mirror the Fargate agents image into a private registry instead of pulling it from the Odigos registry.

Every instrumented Fargate task pulls the same agents bundle at task-start time — it isn't baked into your
task image, so the pull happens on every new task launch, and a failed pull blocks the whole task (the
application container waits on the stager with a `SUCCESS` dependency).

Most accounts need none of this — with Odigos-managed instrumentation, the connector pulls the image from
Odigos' own registry on your behalf. Preloading matters if your Fargate tasks run in a network that cannot
reach that public registry, or your organization requires every image your tasks pull to live in a registry
you own.

**Odigos registry (default)**

By default, tasks pull `registry.odigos.io/odigos-fargate-agents:<version>` — a public, multi-arch
(amd64 + arm64/Graviton) image. This needs egress from the task's subnet to the public internet (a
public IP, or NAT).

Nothing to configure — this is what Odigos-managed instrumentation uses, and it's also the default a
self-managed CloudFormation task definition references unless you override it.

**Your own private registry**

Mirror the image into a registry your task's execution role can already reach. For example, a private
ECR repository in the same account:

**1. Create a repository**

```bash theme={null}
aws ecr create-repository --repository-name odigos-fargate-agents --region <region>
```

**2. Authenticate Docker to your ECR registry**

```bash theme={null}
aws ecr get-login-password --region <region> \
  | docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com
```

**3. Copy the image, preserving the multi-arch manifest**

Use `docker buildx imagetools create` rather than `pull` + `push` — a plain pull only fetches the
image for your local machine's architecture, and pushing that back would give Graviton (arm64)
Fargate tasks an amd64 image that fails to start.

```bash theme={null}
docker buildx imagetools create \
  --tag <account-id>.dkr.ecr.<region>.amazonaws.com/odigos-fargate-agents:<version> \
  registry.odigos.io/odigos-fargate-agents:<version>
```

> [!TIP]
>
> Pin `<version>` to a specific released tag rather than mirroring on a
> schedule — an unpinned mirror can drift from what the connector expects
> if Odigos ships a newer bundle later.

**4. Point instrumentation at your mirrored image**

**CloudFormation**

Replace the `Image` on the `odigos-agent` container in your task definition with your mirrored
tag — see [Instrument Fargate](/cloud-connectors/aws/workloads/fargate/instrument).

**CDK**

Pass your mirrored tag as `agentsImage` on the construct:

```typescript theme={null}
new OdigosFargateInstrumentation(this, 'Odigos', {
  destination: { httpEndpoint: 'https://otlp.example.com:4318' },
  agentsImage: '<account-id>.dkr.ecr.<region>.amazonaws.com/odigos-fargate-agents:<version>',
});
```

See [CDK reference: Fargate](/cloud-connectors/aws/cdk/fargate#the-agents-image).

**Odigos-managed**

The connector's default Fargate agents image is controlled by the `connector-runtime`
controller (`CONNECTOR_AWS_FARGATE_AGENTS_IMAGE`), which stamps `ODIGOS_FARGATE_AGENTS_IMAGE`
onto every AWS connector pod it creates. This isn't yet a `values.yaml` setting — set it on the
controller directly:

```bash theme={null}
kubectl set env deployment/connector-runtime -n odigos-central \
  CONNECTOR_AWS_FARGATE_AGENTS_IMAGE=<account-id>.dkr.ecr.<region>.amazonaws.com/odigos-fargate-agents:<version>
```

> [!WARNING]
>
> This overrides the Fargate agents bundle for **every** AWS connector the
> controller manages. Confirm the version and IAM access (the task
> execution role needs `ecr:GetDownloadUrlForLayer` / `ecr:BatchGetImage`
> on the repository) with Odigos support before changing this in
> production — a mismatched or unreachable override affects every
> Fargate service across every AWS connector at once.

## Next steps

> **[Instrument Fargate](/cloud-connectors/aws/workloads/fargate/instrument)**
>
> CDK and CloudFormation for every supported Fargate language.


# Instrument Lambda with CDK or CloudFormation
Source: https://docs.odigos.io/cloud-connectors/aws/workloads/lambda/instrument

Self-managed AWS Lambda instrumentation for Java, Python, Node.js, and Ruby using the Odigos CDK construct or hand-written CloudFormation.

AWS Lambda functions can be instrumented two ways: **Odigos-managed**, where the connector attaches
everything for you, or **self-managed**, where you attach it yourself through your own IaC. This page is
about the self-managed path — see
[Odigos-managed vs. self-managed instrumentation](/cloud-connectors/overview#odigos-managed-vs-self-managed-instrumentation)
if you haven't decided which one you want yet.

> [!NOTE]
>
> Terraform and a plain `aws lambda update-function-configuration` CLI command are
> also available for every language shown here — the **Custom** access-level policy
> preview in the Central UI surfaces all four formats for a discovered function.
> This page covers the two most common: CDK and CloudFormation.

## What actually gets added to your function

However you apply it, self-managed Lambda instrumentation is the same four things:

1. **The OpenTelemetry layer** — a prebuilt Lambda layer for your function's language and architecture.
2. **An exec wrapper** (`AWS_LAMBDA_EXEC_WRAPPER`) — tells Lambda to run your handler through the layer's
   instrumentation code first. The exact path is different per language.
3. **The bundled collector's configuration** (`OPENTELEMETRY_COLLECTOR_CONFIG_URI`) — tells the small
   OpenTelemetry collector that ships inside the layer where to send your telemetry. It runs in the same
   Lambda sandbox and forwards data *after* your handler already returned, so a slow destination costs
   extra billed time, not extra response latency.
4. **The `ODIGOS_IAC` marker** — a small JSON value that tells Odigos "this function is instrumented, and
   here's the layer version it's running." Without it, Odigos has no way to tell the difference between an
   instrumented function and one you haven't touched yet — it will keep reporting the function as
   **Not Instrumented / Waiting for you**, even if everything else above is already in place.

The CDK construct sets all four for you. With CloudFormation, you set them by hand — one per-language tab
below shows exactly what to add.

## Using AWS CDK

The easiest option if your functions are already defined in a CDK app. One package covers every supported
language — the construct looks at each function's runtime and figures out the right layer and wrapper on
its own.

```bash theme={null}
npm install @odigos/aws-cdk@<PACKAGE_VERSION>
```

> [!NOTE]
>
> TypeScript and JavaScript only, for now — a Python distribution is built but not
> yet published. Python CDK apps should use the
> [CloudFormation instructions](#using-cloudformation-directly) below.

```typescript theme={null}
import { OdigosLambdaInstrumentation } from '@odigos/aws-cdk';

new OdigosLambdaInstrumentation(this, 'Odigos', {
  destination: {
    endpoint: 'otlp.example.com:4317',   // OTLP/gRPC as bare host:port — no scheme
    // httpEndpoint: 'https://otlp.example.com:4318',  // use this instead, for OTLP/HTTP
  },
  serviceName: 'my-api',   // set explicitly if your stack doesn't name the function directly
  functions: [apiFn, workerFn],
});
```

That's it — this one call attaches the layer, sets the wrapper, configures the bundled collector (using the
same renderer the connector itself uses, so the values match exactly), and writes `ODIGOS_IAC` for every
function you list.

> [!WARNING]
>
> Don't also enable a CDK-instrumented function as an Odigos-managed source. Every
> `cdk deploy` would reassert the CDK's configuration, every connector reconcile
> would reassert its own, and you'd get a new function version on every round trip.

> **[Full CDK reference](/cloud-connectors/aws/cdk/lambda)**
>
> Every option in depth — layer sources, instrumenting a whole stack at once, the AWS Lambda Web Adapter,
> Java handler interfaces, and the complete API.

## Using CloudFormation directly

If you're not using CDK, merge these properties into your function's `AWS::Lambda::Function` resource by
hand. The shape is identical across languages — a layer, a wrapper, a service name, and the collector
configuration — but the wrapper path (and, for the Web Adapter case, the loader variable) is different for
each one. Pick your language below:

**Java**

```yaml theme={null}
# Merge into Resources.<YourFunction>.Properties
Layers:
  - "<OTEL_LAYER_ARN>"    # keep any layers you already have, add this one
Environment:
  Variables:
    ODIGOS_IAC: '{"v":1,"layer":"<OTEL_LAYER_ARN>"}'
    AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-handler"
    OTEL_SERVICE_NAME: "<YOUR_FUNCTION_SERVICE_NAME>"
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: 'yaml:{receivers: {otlp: {protocols: {grpc: {endpoint: "localhost:4317"}, http: {endpoint: "localhost:4318"}}}}, processors: {memory_limiter: {check_interval: 1s, limit_percentage: 50, spike_limit_percentage: 20}, batch: {send_batch_size: 512, timeout: 200ms}, decouple: null}, exporters: {otlp: {endpoint: "<OTLP_ENDPOINT>", tls: {insecure: <OTLP_INSECURE>}}}, service: {telemetry: {metrics: {level: none}}, pipelines: {traces: {receivers: [otlp], processors: [memory_limiter, batch, decouple], exporters: [otlp]}}}}'
```

> [!NOTE]
>
> Java ships three more wrappers for handler interfaces other than the default
> `RequestHandler`: `/opt/otel-proxy-handler` (API Gateway proxy events),
> `/opt/otel-sqs-handler` (SQS), and `/opt/otel-stream-handler`
> (`RequestStreamHandler`). Using the default wrapper on a `RequestStreamHandler`
> function breaks the invocation — it's the wrong interface, not just a
> less-detailed trace.

**Running behind the AWS Lambda Web Adapter?** Remove `AWS_LAMBDA_EXEC_WRAPPER` — the adapter already
owns it — and instead append the agent to `JAVA_TOOL_OPTIONS`. CloudFormation can't read an existing
environment variable back, so add your current value in by hand:

```yaml theme={null}
JAVA_TOOL_OPTIONS: "<YOUR_EXISTING_VALUE> -javaagent:/opt/opentelemetry-javaagent.jar"
```

**Python**

```yaml theme={null}
# Merge into Resources.<YourFunction>.Properties
Layers:
  - "<OTEL_LAYER_ARN>"
Environment:
  Variables:
    ODIGOS_IAC: '{"v":1,"layer":"<OTEL_LAYER_ARN>"}'
    AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-instrument"
    OTEL_SERVICE_NAME: "<YOUR_FUNCTION_SERVICE_NAME>"
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: 'yaml:{receivers: {otlp: {protocols: {grpc: {endpoint: "localhost:4317"}, http: {endpoint: "localhost:4318"}}}}, processors: {memory_limiter: {check_interval: 1s, limit_percentage: 50, spike_limit_percentage: 20}, batch: {send_batch_size: 512, timeout: 200ms}, decouple: null}, exporters: {otlp: {endpoint: "<OTLP_ENDPOINT>", tls: {insecure: <OTLP_INSECURE>}}}, service: {telemetry: {metrics: {level: none}}, pipelines: {traces: {receivers: [otlp], processors: [memory_limiter, batch, decouple], exporters: [otlp]}}}}'
```

**Running behind the AWS Lambda Web Adapter?** Remove `AWS_LAMBDA_EXEC_WRAPPER` and load the agent
through `PYTHONPATH` instead. Unlike Java or Node.js, this value goes at the **front**, not the back —
the layer's OpenTelemetry packages need to resolve before your application's own code does:

```yaml theme={null}
PYTHONPATH: "/opt/python:/opt/python/opentelemetry/instrumentation/auto_instrumentation:/var/task:/var/task/site-packages:<YOUR_EXISTING_VALUE>"
OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
```

**Node.js**

```yaml theme={null}
# Merge into Resources.<YourFunction>.Properties
Layers:
  - "<OTEL_LAYER_ARN>"
Environment:
  Variables:
    ODIGOS_IAC: '{"v":1,"layer":"<OTEL_LAYER_ARN>"}'
    AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-handler"
    OTEL_SERVICE_NAME: "<YOUR_FUNCTION_SERVICE_NAME>"
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: 'yaml:{receivers: {otlp: {protocols: {grpc: {endpoint: "localhost:4317"}, http: {endpoint: "localhost:4318"}}}}, processors: {memory_limiter: {check_interval: 1s, limit_percentage: 50, spike_limit_percentage: 20}, batch: {send_batch_size: 512, timeout: 200ms}, decouple: null}, exporters: {otlp: {endpoint: "<OTLP_ENDPOINT>", tls: {insecure: <OTLP_INSECURE>}}}, service: {telemetry: {metrics: {level: none}}, pipelines: {traces: {receivers: [otlp], processors: [memory_limiter, batch, decouple], exporters: [otlp]}}}}'
```

**Running behind the AWS Lambda Web Adapter?** Remove `AWS_LAMBDA_EXEC_WRAPPER` and append the agent to
`NODE_OPTIONS` instead:

```yaml theme={null}
NODE_OPTIONS: "<YOUR_EXISTING_VALUE> --import /opt/init.mjs"
```

**Ruby**

```yaml theme={null}
# Merge into Resources.<YourFunction>.Properties
Layers:
  - "<OTEL_LAYER_ARN>"
Environment:
  Variables:
    ODIGOS_IAC: '{"v":1,"layer":"<OTEL_LAYER_ARN>"}'
    AWS_LAMBDA_EXEC_WRAPPER: "/opt/otel-handler"
    OTEL_SERVICE_NAME: "<YOUR_FUNCTION_SERVICE_NAME>"
    OPENTELEMETRY_COLLECTOR_CONFIG_URI: 'yaml:{receivers: {otlp: {protocols: {grpc: {endpoint: "localhost:4317"}, http: {endpoint: "localhost:4318"}}}}, processors: {memory_limiter: {check_interval: 1s, limit_percentage: 50, spike_limit_percentage: 20}, batch: {send_batch_size: 512, timeout: 200ms}, decouple: null}, exporters: {otlp: {endpoint: "<OTLP_ENDPOINT>", tls: {insecure: <OTLP_INSECURE>}}}, service: {telemetry: {metrics: {level: none}}, pipelines: {traces: {receivers: [otlp], processors: [memory_limiter, batch, decouple], exporters: [otlp]}}}}'
```

> [!NOTE]
>
> Ruby doesn't have a web-server (AWS Lambda Web Adapter) activation path — the
> exec wrapper above is the only way to instrument it. If you apply this to a
> Ruby function that actually runs behind the adapter, it won't work, and it
> will show up as instrumentation drift rather than a clear error.

> [!TIP]
>
> Substitute only the placeholders in `<ANGLE_BRACKETS>`. Change nothing else in
> `OPENTELEMETRY_COLLECTOR_CONFIG_URI` — Odigos compares the whole value as one
> string, and even a missing space after a colon breaks it (see below).

## Things to watch out for

> **Every space in the collector config matters**
>
> `OPENTELEMETRY_COLLECTOR_CONFIG_URI` is written in a compact YAML style where a colon must always be
> followed by a space. Miss one and the collector doesn't fail with a helpful parse error — it accepts
> the variable, logs the URI it read, and then fails to start with an unrelated-looking
> `otelcol state is Closed`.

> **A few variables must never be set**
>
> `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `ODIGOS_OTLP_ENDPOINT`, and
> `ODIGOS_OTLP_INSECURE` are all compared against **unset** — adding any of them is reported as drift,
> even if the value looks harmless. `OTEL_LOG_LEVEL` is the one exception: leave it unset unless debug
> telemetry is turned on for this connector, in which case it must be exactly `debug`.

> **ODIGOS\_IAC is not optional**
>
> It's the only signal Odigos has that this function is instrumented at all, and the layer ARN recorded
> inside it is what future checks compare against — so a version you deliberately pinned is respected
> instead of being flagged as an available upgrade.

> **Never set ODIGOS\_MANAGED by hand**
>
> That variable belongs to the connector alone — it's how the connector remembers what to roll back to
> after it makes a change itself. Setting it yourself claims a rollback that was never actually recorded.

> **Turning off a signal**
>
> If your destination doesn't accept metrics or logs, set `OTEL_METRICS_EXPORTER` / `OTEL_LOGS_EXPORTER`
> to `"none"` and remove that signal's pipeline from the collector configuration. Leave both untouched
> (unset) for a signal your destination does accept.

> [!TIP]
>
> Once `ODIGOS_IAC` is set and the layer, wrapper, and collector configuration all
> match, Odigos flips the function to **Instrumented** on its own — usually within
> a few minutes, at the next discovery cycle. No action needed on the Odigos side.

## Next steps

> **[Full CDK reference](/cloud-connectors/aws/cdk/lambda)**
>
> Every construct option, in depth.

> **[Preload the layer](/cloud-connectors/aws/workloads/lambda/preload)**
>
> Get the layer into your own account first, if your policy requires it.


# Preload the Lambda Layer
Source: https://docs.odigos.io/cloud-connectors/aws/workloads/lambda/preload

Preload the Lambda OpenTelemetry layer into your own AWS account before instrumenting with IaC.

> [!NOTE]
>
> If you use **Odigos-managed** Lambda instrumentation, you can skip this page — the
> connector publishes the layer version into your account itself
> (`lambda:PublishLayerVersion`) the first time it instruments a function. Nothing to
> preload.

If you **self-manage** instrumentation, you have two choices for where the layer your CloudFormation, CDK,
or Terraform attaches actually lives:

**Reference the public Odigos layer**

Simplest option: attach the public layer ARN Odigos publishes in every region
(`arn:aws:lambda:<region>:<odigos-account>:layer:odigos-otel-<language>-<arch>:<n>`) directly in your
function resource. No preloading, no extra deploy — for next steps see
[Instrument Lambda](/cloud-connectors/aws/workloads/lambda/instrument). If your organization's policy
forbids attaching a Lambda layer owned by another AWS account, use the
**Preload into your own account (SAR)** tab instead.

**Preload into your own account (SAR)**

Deploy the Odigos layer from the AWS Serverless Application Repository (SAR) so the resulting
`AWS::Lambda::LayerVersion` is owned by **your** account. Then attach that stack's `LayerVersionArn`
instead of the public Odigos layer ARN.

Deploying a SAR application is an AWS workflow — follow
[Deploying applications](https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverlessrepo-how-to-consume.html)
(Console or CLI). For nesting the app in your own template, see
[`AWS::Serverless::Application`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-application.html).

What Odigos supplies for that deploy:

* **Application IDs** — `SAR_APPLICATIONS` in `@odigos/aws-cdk`, keyed by `language/architecture`
  (for example `python/arm64`). Search the console for `odigos-otel-<language>-<arch>` if you prefer.
* **Semantic version** — `SAR_SEMANTIC_VERSION` in the same package. Pin it; SAR versions are immutable
  and that pin keeps the layer bytes stable across deploys.

With the Odigos CDK construct, you do not call SAR yourself — set
`layerSource: OdigosLayerSource.IN_ACCOUNT` and the construct nests the application using those exports.
See [Where the layer comes from](/cloud-connectors/aws/cdk/lambda#where-the-layer-comes-from).
The deploying principal needs `serverlessrepo:CreateCloudFormationTemplate` and
`serverlessrepo:GetApplication`.

> [!NOTE]
>
> Layer versions are retained, never deleted, when a stack update supersedes them
> — a function still pinned to an older version would break at cold start the
> moment it disappeared. Clean up old versions yourself once nothing references
> them.

## Next steps

> **[Instrument Lambda](/cloud-connectors/aws/workloads/lambda/instrument)**
>
> CDK and CloudFormation for every supported Lambda language.

> **[CDK reference](/cloud-connectors/aws/cdk/lambda)**
>
> Every option on the Lambda CDK construct, in depth.


# Enable Cloud Connectors
Source: https://docs.odigos.io/cloud-connectors/enable

Enable the Cloud Connectors feature on Odigos Central with Helm or the CLI, including bundled PostgreSQL configuration.

Cloud Connectors are **off by default**. Until you enable them, Central does not install the connector controller, the connector PostgreSQL store, or the `OdigosCloudConnector` CRD.

Enabling the feature does **not** create per-account connectors. After the subsystem is running, add an [AWS](/cloud-connectors/add-connectors/add-aws-connector) or [GCP](/cloud-connectors/add-connectors/add-gcp-connector) connector from the Central UI.

> [!NOTE]
>
> Cloud Connectors require the same Odigos Enterprise on-prem token (or install
> secret) as the rest of the Central installation.

## What gets installed

When `cloudConnectors.enabled` is `true` and the Central install secret exists, the chart installs:

* The `OdigosCloudConnector` CRD
* The `connector-runtime` controller
* Bundled PostgreSQL (`odigos-connector-postgres`) and a PVC when persistence is enabled

> [!WARNING]
>
> A **StorageClass** must be defined for the Postgres PVC. Bundled Postgres
> uses a PVC for connector state management; without a usable StorageClass the
> PVC cannot bind. Set
> `cloudConnectors.postgres.persistence.storageClassName` explicitly, or leave
> it empty to use the cluster default.

## Enable Cloud Connectors

**CLI**

At install time:

```bash theme={null}
odigos pro central install \
  --set onPremToken=<token> \
  --set cloudConnectors.enabled=true \
  --set cloudConnectors.postgres.password=<strong-password> \
  --set cloudConnectors.postgres.persistence.enabled=true \
  --set cloudConnectors.postgres.persistence.size=20Gi \
  --set cloudConnectors.postgres.persistence.storageClassName=
```

On an existing Central installation:

```bash theme={null}
odigos pro central upgrade \
  --set cloudConnectors.enabled=true \
  --set cloudConnectors.postgres.password=<strong-password> \
  --set cloudConnectors.postgres.persistence.enabled=true \
  --set cloudConnectors.postgres.persistence.size=20Gi \
  --set cloudConnectors.postgres.persistence.storageClassName=
```

> [!TIP]
>
> Leave `storageClassName` empty to use the cluster default, or set it to a
> class your cluster requires (e.g. `gp3`).

**Helm**

Enable with `--set`:

```bash theme={null}
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --reuse-values \
  --set cloudConnectors.enabled=true \
  --set cloudConnectors.postgres.password=<strong-password> \
  --set cloudConnectors.postgres.persistence.enabled=true \
  --set cloudConnectors.postgres.persistence.size=20Gi \
  --set cloudConnectors.postgres.persistence.storageClassName=
```

Or in `values.yaml`:

```yaml theme={null}
cloudConnectors:
  enabled: true
  postgres:
    password: "<strong-password>"
    persistence:
      enabled: true
      size: 20Gi
      storageClassName: "" # set if your cluster requires an explicit class
```

Then apply:

```bash theme={null}
helm upgrade --install odigos-central odigos/odigos-central \
  --namespace odigos-central \
  --create-namespace \
  -f values.yaml
```

## PostgreSQL configuration

Cloud Connectors use a bundled PostgreSQL instance under `cloudConnectors.postgres`. Configure these values when you enable the feature:

**cloudConnectors.postgres.password** `string` (default: odigos)

Database password. Change this for non-development installs.

**cloudConnectors.postgres.username** `string` (default: odigos)

Database username.

**cloudConnectors.postgres.database** `string` (default: odigos\_central)

Database name.

**cloudConnectors.postgres.persistence.enabled** `boolean` (default: true)

Persist connector state on a PVC.

**cloudConnectors.postgres.persistence.size** `string` (default: 20Gi)

PVC size. Your cluster must provide storage.

**cloudConnectors.postgres.persistence.storageClassName** `string`

Storage class. Empty uses the cluster default.

> **Advanced Postgres settings**
>
> | Value | Default | Description |
> | - | - | - |
> | `postgres.persistence.accessModes` | `[ReadWriteOnce]` | PVC access modes |
> | `postgres.resources` | chart defaults | CPU and memory requests/limits for the Postgres pod |
> | `postgres.image` | chart default | Leave at the chart default unless Odigos support or engineering asks you to override it |
>
> > [!WARNING]
> >
> > Do not set a custom Postgres image on your own. Use the image shipped with
> > the chart unless Odigos support or engineering directs otherwise.

> **Optional provider image overrides**
>
> Provider connector images (`cloudConnectors.aws.image`, `cloudConnectors.gcp.image`) default to the versions shipped with the chart.
>
> > [!WARNING]
> >
> > Override these only when Odigos support or engineering asks you to. Custom
> > images can break discovery, instrumentation, or schema compatibility.
>
> ```yaml theme={null}
> cloudConnectors:
>   enabled: true
>   aws:
>     image: "" # empty = chart default; set only if directed by Odigos
>   gcp:
>     image: "" # empty = chart default; set only if directed by Odigos
> ```

## Verify

After enablement, confirm the subsystem is running in the Central namespace (default `odigos-central`):

```bash theme={null}
kubectl get pods -n odigos-central -l 'app in (connector-runtime,odigos-connector-postgres)'
```

You should see `connector-runtime` and `odigos-connector-postgres` in a Running state.

> [!TIP]
>
> Once both pods are Running, you are ready to add a cloud provider connector.

## Next steps

> **[Add AWS connector](/cloud-connectors/add-connectors/add-aws-connector)**
>
> Connect an AWS account.

> **[Add GCP connector](/cloud-connectors/add-connectors/add-gcp-connector)**
>
> Connect a Google Cloud project.

> **[Instrument workloads](/cloud-connectors/instrument-workloads)**
>
> Create Sources after a connector is online.


# Instrument Cloud Workloads
Source: https://docs.odigos.io/cloud-connectors/instrument-workloads

Create Sources from discovered cloud workloads, configure an OTLP destination, and track instrumentation progress.

Once a Cloud Connector is online, browse the workloads it discovered and create **Sources** for the ones you want to observe. Enabling a Source tells the connector to apply provider-specific instrumentation and export telemetry to the OTLP destination(s) on that connector.

## Prerequisites

**1. Healthy connector**

A Cloud Connector with discovery enabled for the workload types you want.

**2. OTLP destination**

At least one **OTLP destination** on that connector. See [Add an OTLP destination](#add-an-otlp-destination). Workload instrumentation is destination-gated; without a usable OTLP endpoint the connector will not attach instrumentation.

**3. Reachable endpoint**

Instrumented workloads export OTLP to that destination from the cloud environment (for example from Lambda or Cloud Run). Point the destination at an endpoint those workloads can reach — for example a collector or backend with appropriate public or VPC access. See [Telemetry export](/cloud-connectors/overview#telemetry-export).

## Add an OTLP destination

Before you create Sources, add at least one destination on the connector so instrumented workloads have somewhere to export telemetry.

Lambda and Cloud Run support OTLP destinations only. For more information, see [Telemetry export](/cloud-connectors/overview#telemetry-export).

Follow the [Destinations overview](/enterprise/backends-overview) for how to add and configure a destination, then apply that setup on the connector in Central. The endpoint must be reachable from the cloud workloads themselves (not only from the Central cluster).

## Browse discovered workloads

Open the connector in Central and review discovered workloads. Each candidate typically includes kind, name, region/location, and whether it is already a Source or instrumented.

## Create Sources

Select the workloads to instrument and create Sources. A Source is the instrumentation intent for one cloud resource. Identity includes:

* **Name**: the provider resource name
* **Kind**: for example `aws.lambda` or `gcp.cloud-run`
* **Region**: the resource's region or location

Enable the Source so the connector applies instrumentation to that workload. That typically means adding the Odigos OTEL SDK and collector components the runtime needs, configuring them for the workload, and configuring where telemetry is sent (the OTLP destination on the connector).

See [Supported languages](/cloud-connectors/overview#supported-languages) for which runtimes each workload type can instrument.

## Check Source status

After you create a Source, it appears under **Sources** in Central. Open it to see details and health.

The Source details typically include:

* **Identity**: kind (e.g. AWS Lambda), name, region, and service name, plus whether the Source is **Enabled**
* **Health**: a status banner such as **Success And Emitting Telemetry** when the Source is instrumented, healthy, and telemetry has been observed
* **Instrumentation**: detected language and version (e.g. `python • 3.13`), and whether the workload is **Instrumented**

If something is wrong (for example missing permissions), use the health message on the Source details to diagnose.

## Uninstrument workloads

To remove instrumentation from one or more Sources, select them with the checkboxes and click **Uninstrument**. The connector removes the Odigos OTEL SDK, collector components, and related configuration it applied earlier, so the workloads stop exporting telemetry through that Source.

## Troubleshooting

> **No instrumentation applied**
>
> Confirm an [OTLP destination](#add-an-otlp-destination) exists on the connector and the Source is enabled. See [Destinations overview](/enterprise/backends-overview) for how to add and configure destinations.

> **GCP: permission denied on update**
>
> Grant `iam.serviceAccounts.actAs` (or `roles/iam.serviceAccountUser` on the runtime service account). See [Add GCP connector](/cloud-connectors/add-connectors/add-gcp-connector).

> **GCP: traces missing intermittently**
>
> Cloud Run CPU throttling or scale-to-zero can delay or drop background OTLP export. Prefer a reachable destination and, if needed, keep CPU allocated or a minimum instance warm on low-traffic services.

> **AWS: test connection passed but instrumentation fails**
>
> Test connection validates discovery permissions. Grant the Lambda instrumentation IAM actions listed on the [Add AWS](/cloud-connectors/add-connectors/add-aws-connector) page.

## Next steps

> **[Manage and remove](/cloud-connectors/manage-and-remove)**
>
> Understand connector health and how deletion policies affect cloud instrumentation.


# Manage and Remove Cloud Connectors
Source: https://docs.odigos.io/cloud-connectors/manage-and-remove

Monitor Cloud Connector health, understand status, and remove connectors with retain or cleanup deletion policies.

Use this page to operate connectors after they are online — health, immutable fields, and safe removal.

## Connector status

The connectors list and detail views show installation and runtime health. The connector phase is one of:

| Status | Meaning |
| - | - |
| **Pending** | The connector resource exists but has not started yet |
| **Starting** | The connector workload is deploying, or the runtime has not finished registering, authenticating, and completing a successful discovery cycle |
| **Connected** | The runtime is healthy: authenticated to the cloud account and discovery is succeeding |
| **Degraded** | The runtime is up but not fully healthy (for example discovery failures, or heartbeats that have gone stale) |
| **Error** | A hard failure that blocks a healthy runtime (for example the connector image cannot be resolved, or `spec.provider` was changed after create) |
| **Disabled** | The connector runtime reported itself as disabled |
| **Deleting** | The connector is being removed (including instrumentation cleanup when the deletion policy is `cleanup`) |

> **Common causes of Degraded or Error**
>
> **Degraded** usually means:
>
> * A discovery cycle failed (missing permissions, cloud API errors, or other detector failures)
> * The runtime stopped refreshing its heartbeat (for example the pod is wedged or lost its Postgres connection), so the last healthy status went stale
>
> Auth that has not succeeded yet, or discovery that has not completed a successful cycle yet, typically stays in **Starting** rather than **Degraded**.
>
> **Error** usually means:
>
> * The connector image could not be resolved for the provider
> * `spec.provider` was changed after the connector was created (provider is immutable)
>
> Schema version mismatches fail connector startup (the pod does not stay healthy), which typically shows as **Starting** or, if heartbeats stop updating, **Degraded** — not a dedicated schema-mismatch phase.
>
> Fix credentials or IAM, confirm [Cloud Connectors are enabled](/cloud-connectors/enable), and check `connector-runtime` / connector pods in the Central namespace.

## What you can change

* You can update display name, credentials Secret contents, discovery interval, and **capabilities** (which types to discover/instrument), depending on what the UI exposes for your version.
* **`provider` is immutable** after the connector is created. To switch providers (for example from AWS to another cloud), create a new connector.

## Delete a connector

Deleting a connector removes the connector runtime from the Central cluster. What happens to cloud-side instrumentation depends on the deletion policy annotation on the `OdigosCloudConnector` resource:

```yaml theme={null}
metadata:
  annotations:
    odigos.io/deletion-policy: retain   # or cleanup
```

**retain (default)**

Deletes the connector from Central (the connector pod and related resources in the Central cluster). It does **not** uninstrument cloud workloads.

Any Odigos instrumentation already applied in the cloud account stays in place (for example Lambda layers or Cloud Run agent injection). To remove that instrumentation, [uninstrument Sources](/cloud-connectors/instrument-workloads#uninstrument-workloads) while the connector is still online, or change those resources outside Odigos after delete.

> [!NOTE]
>
> Creating a connector from the Central UI typically leaves the annotation
> unset, which means **retain**.

**cleanup**

Before removing the connector, attempt to roll back Odigos instrumentation in the provider.

Set the annotation explicitly:

```yaml theme={null}
metadata:
  annotations:
    odigos.io/deletion-policy: cleanup
```

> [!WARNING]
>
> If rollback does not finish within the cleanup timeout (**15 minutes** by
> default), the connector may still be deleted and some resources can remain
> instrumented.

### Sources on delete

When you delete a connector, Sources and discovery data for that account boundary are no longer managed through that connector.

> [!TIP]
>
> Prefer disabling or deleting Sources first if you want a controlled rollback
> while the connector is still running, especially under the default **retain**
> policy.

## Related

> **[Instrument workloads](/cloud-connectors/instrument-workloads)**
>
> Create and disable Sources while the connector is online.

> **[Cloud Connectors overview](/cloud-connectors/overview)**
>
> Back to concepts and supported providers.


# Cloud Connectors Overview
Source: https://docs.odigos.io/cloud-connectors/overview

Use Odigos Central Cloud Connectors to discover and instrument cloud workloads across supported providers from a single control plane.

Cloud Connectors are the third connection type in Odigos Central, alongside [Remote Clusters](/central/adding-connections/remote-clusters) and [VM Agent](/central/adding-connections/vmagent).

Each connector manages **one** cloud account boundary (for example an AWS account, a GCP project, or another supported provider). From Central you discover workloads in that account, create Sources for the ones you want to observe, and track instrumentation progress.

## Prerequisites

**1. Install Odigos Central**

Follow the [Central installation](/central/installation) guide.

**2. Enable Cloud Connectors**

Set `cloudConnectors.enabled=true`. See [Enable Cloud Connectors](/cloud-connectors/enable).

**3. Prepare cloud access**

Create a cloud identity (e.g. IAM role or service account) with only the permissions you want the connector to have. See [You control the access scope](#you-control-the-access-scope).

## Why use Cloud Connectors?

* **Unified management** — Manage serverless and cloud workloads alongside Kubernetes clusters and VM hosts from the same Central UI
* **Automatic discovery** — Surface instrumentable workloads in the account boundary without hunting for each resource by hand
* **Flexible scale** — Instrument one workload or many at once from discovery; nothing is instrumented until you choose
* **Less operational overhead** — Avoid manually attaching layers, agents, or env vars on every Lambda or Cloud Run service
* **Centralized credentials** — Cloud credentials live as Secrets in the Central cluster; one connector owns one account boundary for clear blast radius and ownership

## You control the access scope

Cloud Connectors do **not** get blanket access to your cloud environment. What each connector can discover and instrument is limited by the access you grant.

> [!NOTE]
>
> Scope is defined by your cloud permissions (e.g. IAM policies or service account
> roles), plus the resource types you enable on the connector. Odigos only acts
> within that boundary.

Three layers work together:

1. **Cloud credentials and permissions** — The role or service account you attach decides which APIs and resources the connector can call. Narrow policies mean a narrower discovery and instrumentation footprint.
2. **Connector capabilities** — When you add a connector, you choose which resource types may be discovered and which may be instrumented (per-type capabilities).
3. **Sources** — Even after discovery, nothing is instrumented until you create Sources for the workloads you select.

Provider-specific permission lists are covered on the corresponding Add connector pages (for example [Add AWS](/cloud-connectors/add-connectors/add-aws-connector)).

## Workloads and compute platforms

Connectors organize resources into two domains:

* **Workloads** — resources Odigos discovers and can instrument directly (for example AWS Lambda or GCP Cloud Run).
* **Compute platforms** — managed environments such as ECS clusters, GKE, or Compute Engine. These can be discovered today; installing a separate Odigos agent onto them expands coverage over time.

You choose which resource types to discover and instrument when you add a connector (per-type capabilities).

## Odigos-managed vs. self-managed instrumentation

When adding or editing a connector, choose an instrumentation method for each workload type:

| Discovery | Instrumentation | Mode | Who applies the change |
| - | - | - | - |
| On | On | **Odigos-managed (automatic)** | Odigos attaches instrumentation for you (a Lambda layer + environment, or a sidecar injected into an ECS task definition) and keeps it in place on every reconcile cycle. |
| On | Off | **Self-managed (IaC)** | You apply the change yourself — CloudFormation, CDK, Terraform, or by hand. Odigos only discovers and reports status; it never modifies the resource. |
| Off | — | Not discovered | The type is not surfaced at all. |

Rather than using a dedicated mode toggle, Odigos controls access through per-type instrumentation settings. This ensures self-managed workloads are safe by default—Odigos cannot touch or configure a resource unless its instrumentation capability is active.

> [!TIP]
>
> Pick self-managed when your organization already provisions Lambda functions or ECS
> task definitions through CloudFormation, CDK, or Terraform and you want that
> pipeline to stay the source of truth. Pick Odigos-managed when you want instrumentation
> applied and kept up to date without changing your IaC.

A self-managed workload shows as **Not Instrumented / Waiting for you** until Odigos observes that you've applied its instrumentation, then it flips to **Instrumented** on its own — there is nothing to click on the Odigos side. If a self-managed workload is later disabled as a Source or its Source is deleted while your instrumentation is still applied, it reports an error rather than silently going dark — Odigos never removes instrumentation it did not apply, so taking it back out is yours to do.

> [!WARNING]
>
> Instrument a given workload **one way, not both**. A function or service you
> instrument yourself and also enable as an Odigos-managed Source is owned by
> both sides — every IaC deploy reasserts your configuration, every Odigos
> reconcile reasserts its own, and you get a new function version or task
> revision on every round trip.

For AWS specifically, see [AWS Connector](/cloud-connectors/aws/overview) for how to instrument Lambda and Fargate yourself with CloudFormation or CDK, and how to preload artifacts into your own account first.

## Resource coverage

Current coverage includes the following providers and resource types. Additional providers may be added over time.

| Provider | Discover | Instrument |
| - | - | - |
| AWS Lambda (`aws.lambda`) | Yes | Yes — Odigos-managed or self-managed |
| Amazon ECS Fargate tasks (`aws.fargate-task`) | Yes | Yes — Odigos-managed or self-managed |
| Amazon ECS clusters (`aws.ecs-cluster`, `aws.fargate-cluster`) | Yes | n/a (coverage/grouping only; agent install coming soon) |
| GCP Cloud Run (`gcp.cloud-run`) | Yes | Yes |
| GKE (`gcp.gke`) / Compute Engine (`gcp.gce`) | Yes | Coming soon |

## Supported languages

Direct instrumentation depends on the workload type:

| Workload | Supported languages |
| - | - |
| AWS Lambda (`aws.lambda`) | Java, Python, Node.js, Ruby |
| Amazon ECS Fargate tasks (`aws.fargate-task`) | Java, Python, Node.js, .NET, PHP, Ruby |
| GCP Cloud Run (`gcp.cloud-run`) | Java, Python, Node.js, .NET |

Other Lambda runtimes (for example Go or .NET) can still be discovered; they are not instrumented until support for those runtimes is added. On Fargate, Go and any other eBPF-based language cannot be instrumented at all — Fargate does not allow the privileged containers eBPF needs — and a task with no other instrumentable container is reported unsupported.

AWS Lambda Ruby, and Fargate PHP and Ruby, support OTLP HTTP only. See [Telemetry export](#telemetry-export) for destination protocol limits.

## Telemetry export

Instrumented workloads export OpenTelemetry data to an **OTLP endpoint** you configure on the connector. That endpoint must be reachable from the cloud environment (for example from Lambda or Cloud Run).

Lambda, Fargate, and Cloud Run support **OTLP destinations only**. Fargate tasks export **directly** to the destination — there is no node-local collector on ECS the way there is inside the Lambda layer, so the destination endpoint must be reachable from the task's VPC.

Protocol support for AWS Lambda:

* **Python, Node.js, Java** — OTLP gRPC and OTLP HTTP
* **Ruby** — OTLP HTTP only

Protocol support for Amazon ECS Fargate:

* **Java, Python, Node.js, .NET** — prefer OTLP HTTP but honor OTLP gRPC if that's all the destination offers
* **PHP, Ruby** — OTLP HTTP only; without an OTLP HTTP endpoint on the destination these languages cannot be instrumented at all

GCP Cloud Run supports OTLP gRPC and OTLP HTTP.

See [Add an OTLP destination](/cloud-connectors/instrument-workloads#add-an-otlp-destination). For how to add and configure destinations, see the [Destinations overview](/enterprise/backends-overview).

## Next steps

> **[Enable Cloud Connectors](/cloud-connectors/enable)**
>
> Turn on the feature in Helm or the CLI, including Postgres settings.

> **[Add AWS connector](/cloud-connectors/add-connectors/add-aws-connector)**
>
> Connect an AWS account and discover Lambda and ECS resources.

> **[Add GCP connector](/cloud-connectors/add-connectors/add-gcp-connector)**
>
> Connect a Google Cloud project and discover Cloud Run services.

> **[Instrument workloads](/cloud-connectors/instrument-workloads)**
>
> Create Sources and track instrumentation after a connector is online.

> **[AWS Connector](/cloud-connectors/aws/overview)**
>
> Full AWS connector reference: flows, artifact preloading, and CDK/CloudFormation for every instrumentation type.


# Adding New Observability Destination
Source: https://docs.odigos.io/enterprise/adding-new-dest

There are tens if not hundreds of different observability destinations. Odigos's goal is to provide a seamless and easy way to ship observability data to any one of them.

In this guide, you will learn how to contribute a new destination to Odigos. We will create a new dummy destination called `mydest`.

Creating a new destination involves these steps:

1. Extending the UI for the new destination
2. Adding the collector configuration for the new destination
3. Generating the documentation for the new destination

### User Interface

For our new destination to be visible in the UI, we need to define the destination metadata and fields in YAML format:

**1. Add Logo**

Go to [`destinations/logos/`](https://github.com/odigos-io/odigos/tree/main/destinations/logos) directory and add your logo.

* Must be in SVG format
* Example: `mydest.svg`

```svg mydest.svg theme={null}
<svg fill="#F5F5F5" width="16" height="12" viewBox="0 0 16 12" xmlns="http://www.w3.org/2000/svg">
  <title>My Destination</title>
  <g>
    <path d="M5.17215 9.01884L2.18377 7.69813C1.48159 7.38732 1.0448 6.73677 1.0448 5.99948C1.0448 5.2622 1.48056 4.61165 2.18377 4.30083L6.36172 2.45349L6.45156 2.41425V0H1.66127C1.0417 0 0.538818 0.4781 0.538818 1.06772V10.9312C0.538818 11.5209 1.0417 11.999 1.66127 11.999H5.83303H6.45259V9.5682L5.17319 9.01781H5.17215V9.01884Z"/>
    <path d="M14.3397 0H9.5484V2.4411L10.8041 2.98115L10.801 2.9698L13.8162 4.3029C14.5184 4.61372 14.9552 5.26426 14.9552 6.00155C14.9552 6.73883 14.5194 7.38938 13.8162 7.7002L10.1649 9.31417L9.54736 9.57646V12H14.3387C14.9583 12 15.4611 11.5219 15.4611 10.9323V1.06772C15.4611 0.4781 14.9583 0 14.3387 0H14.3397Z"/>
    <path d="M8.00048 8.47972C9.36972 8.47972 10.4808 7.36966 10.4808 5.99938C10.4808 4.6291 9.37076 3.51904 8.00048 3.51904C6.6302 3.51904 5.52014 4.6291 5.52014 5.99938C5.52014 7.36966 6.6302 8.47972 8.00048 8.47972Z"/>
  </g>
</svg>
```

**2. Add Destination Metadata**

Go to [`destinations/data/`](https://github.com/odigos-io/odigos/tree/main/destinations/data) directory and create a new YAML file.

* Example: `mydest.yaml`

> **Allowed properties**
>
> | Key | Example | Required | Description | Docs |
> | - | - | :-: | - | :-: |
> | metadata.type | `mydest` | ✅ | unique identifier for the destination | ✅ |
> | metadata.displayName | `My Destination` | ✅ | display name of the destination | ✅ |
> | metadata.category | `managed` | ✅ | one-of: `managed` `self hosted` | ✅ |
> | spec.image | `mydest.svg` | ✅ | logo of the destination | ✅ |
> | spec.signals.traces.supported | `true` | ✅ | is tracing supported? | ✅ |
> | spec.signals.metrics.supported | `true` | ✅ | is metrics supported? | ✅ |
> | spec.signals.logs.supported | `false` | ✅ | is logging supported? | ✅ |
> | spec.note.type | `Note` | ❌ | type of [callout box](https://mintlify.com/docs/content/components/callouts) to display one-of: `Note` `Info` `Warning` `Tip` `Check` | ✅ |
> | spec.note.content | `We handle the endpoint internally, so you don't need to provide it.` | ❌ | content of the callout box | ✅ |
> | spec.fields | - | ✅ | list of fields to be configured, [see specification here](#allowed-properties-for-spec-fields) | ✅ |

> **Allowed properties for spec.fields\[]**
>
> | Key | Example | Required | Description | Docs |
> | - | - | :-: | - | :-: |
> | name | `MYDEST_API_KEY` | ✅ | key name of the field | ✅ |
> | displayName | `API Key` | ✅ | display name of the field | ✅ |
> | secret | `true` | ❌ | is the field a secret? (e.g. password, token, etc.) | ❌ |
> | initialValue | - | ❌ | default value for the field | ✅ |
> | componentType | `input` | ✅ | one-of: `input` `textarea` `dropdown` `checkbox` `multiInput` `keyValuePairs` | ✅ |
> | componentProps.type | `password` | ❌ | the type of value (e.g. string, number, password, etc.) | ✅ |
> | componentProps.required | `true` | ❌ | is the field required? | ✅ |
> | componentProps.values | - | ❌ | list of options for dropdown | ❌ |
> | componentProps.placeholder | `glc_eyJvIj...` | ❌ | placeholder text for input | ✅ |
> | componentProps.tooltip | `Obtained from your "My Dest" account` | ❌ | tooltip text for input | ✅ |
> | renderCondition | `["MYDEST_URL", "!=", ""]` | ❌ | hide/show the field in the UI (form only) one-of: `[boolean]` `[string, operator, any]` where `string` is the key name of the dependency field, and `operator` is one-of: `!=` `==` `>=` `<=` `>` `<` | ❌ |
> | hideFromReadData | `["MYDEST_URL", "==", ""]` | ❌ | hide/show the field in the UI (read only) one-of: `[boolean]` `[string, operator, any]` where `string` is the key name of the dependency field, and `operator` is one-of: `!=` `==` `>=` `<=` `>` `<` | ❌ |
> | customReadDataLabels | - | ❌ | custom labels for the UI (read only) | ❌ |

```yaml mydest.yaml theme={null}
apiVersion: internal.odigos.io/v1beta1
kind: Destination
metadata:
  type: mydest
  displayName: My Destination
  category: managed
spec:
  image: mydest.svg
  signals:
    traces:
      supported: true
    metrics:
      supported: true
    logs:
      supported: false
  fields:
    - name: MYDEST_URL
      displayName: URL
      componentType: input
      componentProps:
        required: true
    - name: MYDEST_REGION
      displayName: Region
      componentType: input
      componentProps:
        required: true
    - name: MYDEST_API_KEY
      displayName: API Key
      componentType: input
      componentProps:
        type: password
        required: true
      secret: true
```

### Collector Configuration

Now that our new vendor can be persisted/loaded in the Kubernetes data store, we need to implement the collector configuration.

**1. Add Destination Type**

Go to [`common/dests.go`](https://github.com/odigos-io/odigos/tree/main/common/dests.go) and add your new destination to the `DestinationType` enum.

* Make sure the value is the same as the `type` property in `mydest.yaml`

**2. Add Collector Config**

Go to [`common/config`](https://github.com/odigos-io/odigos/tree/main/common/config) directory and create a new GO file.

* Example: `mydest.go`

> **Explain Config**
>
> * **DestType** - Returns the enum value of the destination added to [`common/dests.go`](https://github.com/odigos-io/odigos/tree/main/common/dests.go) in step 1.
> * **ModifyConfig** - Is called with the `dest` object which holds the data received from the UI and the `currentConfig` object. The `currentConfig` object contains the current configuration of the gateway collector. Modify this object to include the [OpenTelemetry configuration](https://opentelemetry.io/docs/collector/configuration/) needed by your destination. You can assume a basic configuration is already provided in the `currentConfig` object, for details see `getBasicConfig` method in [`common/config/root.go`](https://github.com/odigos-io/odigos/tree/main/common/config/root.go) file.
>   * Make sure to give any exporter or pipeline a unique name in order to avoid conflicts, use the conventions:
>     * `otlp/<dest-name>-<dest.GetID()>` for OTel gRPC exporters
>     * `otlphttp/<dest-name>-<dest.GetID()>` for OTel HTTP exporters
>     * `traces/<dest-name>-<dest.GetID()>` for traces pipelines
>     * `metrics/<dest-name>-<dest.GetID()>` for metrics pipelines
>     * `logs/<dest-name>-<dest.GetID()>` for logs pipelines
>   * You can use the utility methods `isTracingEnabled`, `isMetricsEnabled` and `isLoggingEnabled` to determine which signals are selected by the user for the destination, and configure the collector accordingly.

```go mydest.go theme={null}
package config

import (
  "fmt"

  "github.com/odigos-io/odigos/common"
)

const (
  regionKey = "MYDEST_REGION"
  urlKey    = "MYDEST_URL"
  apiKeyKey = "MYDEST_API_KEY"
)

type MyDest struct{}

func (m *MyDest) DestType() common.DestinationType {
  // DestinationType defined in common/dests.go
  return common.MyDestDestinationType
}

func (m *MyDest) ModifyConfig(dest ExporterConfigurer, currentConfig *Config) ([]string, error) {
  config := dest.GetConfig()
  // To make sure that the exporter and pipeline names are unique, we'll need to define a unique ID
  uniqueUri := "mydest-" + dest.GetID()

  region, exists := config[regionKey]
  if !exists {
    region = "us"
    return errorMissingKey(regionKey)
  }

  endpoint, exists := config[urlKey]
  if !exists {
    endpoint = fmt.Sprintf("https://%s.mydest.com:4317", region)
  }

  // Modify the exporter here
  exporterName := "otlp/" + uniqueUri
  exporterConfig := GenericMap{
    "endpoint": endpoint,
    "headers": GenericMap{
      // The API Key is a secret, so we can't expose it in the config
      "x-mydest-header-apikey": "${MYDEST_API_KEY}",
    },
  }

  currentConfig.Exporters[exporterName] = exporterConfig

  // Modify the pipelines here
  var pipelineNames []string

  if isTracingEnabled(dest) {
    pipeName := "traces/" + uniqueUri
    currentConfig.Service.Pipelines[pipeName] = Pipeline{
      Exporters: []string{exporterName},
    }
    pipelineNames = append(pipelineNames, pipeName)
  }

  if isMetricsEnabled(dest) {
    pipeName := "metrics/" + uniqueUri
    currentConfig.Service.Pipelines[pipeName] = Pipeline{
      Exporters: []string{exporterName},
    }
    pipelineNames = append(pipelineNames, pipeName)
  }

  if isLoggingEnabled(dest) {
    pipeName := "logs/" + uniqueUri
    currentConfig.Service.Pipelines[pipeName] = Pipeline{
      Exporters: []string{exporterName},
    }
    pipelineNames = append(pipelineNames, pipeName)
  }

  return pipelineNames, nil
}
```

**3. Add Available Config**

Go to [`common/config/root.go`](https://github.com/odigos-io/odigos/tree/main/common/config/root.go) and add the new destination to the `availableConfigers` list.

* Example: `&MyDest{}`

```go common/config/root.go theme={null}
var availableConfigers = []Configer{
  &MyDest{},
  /* List of existing destinations...  */
}
```

### Kubernetes-Specific Collector Configuration

If your destination configures Kubernetes settings, for example you have a config field that accepts a value that should be mounted
in the Collector Pod, create a `K8sConfiger` object in [`instrumentor/k8sconfig`](https://github.com/odigos-io/odigos/tree/main/instrumentor/k8sconfig).

This interface provides an additional function, `ModifyGatewayCollectorDeployment`, which accepts a Destination and Collector Deployment.
Your implementation can use this function to modify the Deployment as necessary based on the Destination config.

Note: It is the responsibility of your implementation to check for the validity of this config, such as checking for conflicts or values
that already exist.

The `K8sConfiger` interface also requires a `DestType()` function, similar to common Destinations. This function returns the type
of common Destination that this Kubernetes implementation relies on. This is used by the instrumentor to conditionally apply the `K8sConfiger`
only for the matching Destination.

When it is implemented, add your `K8sConfiger` implementation to the `availableConfigers` list in
[`instrumentor/k8sconfig/root.go`](https://github.com/odigos-io/odigos/tree/main/instrumentor/k8sconfig/root.go)

### Generating Documentation

**1. Install Python**

Make sure you have [Python installed](https://www.python.org/downloads/) on your system.

**2. Generate Documentation**

Go to [`docs`](https://github.com/odigos-io/odigos/tree/main/docs) directory, and run:

```bash theme={null}
python sync-dest-doc.py
# OR
python3 sync-dest-doc.py
```

**3. Edit Documentation**

The documentation has been generated in the [`docs/backends`](https://github.com/odigos-io/odigos/tree/main/docs/backends) directory. Feel free to add any additional content to the generated file.

* Edit only between the `!! START CUSTOM EDIT !!` and `!! END CUSTOM EDIT !!` comments
* Example: `mydest.mdx`

```mdx mydest.mdx theme={null}
{/*
    !! Do not remove this comment, this acts as a key indicator in `docs/sync-dest-doc.py` !!
    !! START CUSTOM EDIT !!
*/}

**Creating Account**<br />
Go to the **[🔗 website](https://odigos.io) > Account** and click **Sign Up**

**Obtaining Access Token**<br />
Go to **⚙️ > Access Tokens** and click **Create New**

{/*
    !! Do not remove this comment, this acts as a key indicator in `docs/sync-dest-doc.py` !!
    !! END CUSTOM EDIT !!
*/}
```

**4. Update README.md**

Go to [`README.md`](https://github.com/odigos-io/odigos/tree/main/README.md) and add your new destination to the list of destinations.

#### That's it! Now you can use your new destination in the UI and send data to it.

**Please submit a PR to the [odigos git repository](https://github.com/odigos-io/odigos).**


# actions.odigos.io/v1alpha1
Source: https://docs.odigos.io/enterprise/api-reference/actions.odigos.io.v1alpha1



## Resource Types

* [AddClusterInfo](#AddClusterInfo)
* [DeleteAttribute](#DeleteAttribute)
* [K8sAttributesResolver](#K8sAttributesResolver)
* [RenameAttribute](#RenameAttribute)

## `AddClusterInfo`&#x20;

AddClusterInfo is the Schema for the addclusterinfo odigos action API
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`AddClusterInfo`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`AddClusterInfoSpec`](#AddClusterInfoSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`AddClusterInfoStatus`](#AddClusterInfoStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttribute`&#x20;

DeleteAttribute is the Schema for the DeleteAttribute odigos action API
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`DeleteAttribute`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`DeleteAttributeSpec`](#DeleteAttributeSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`DeleteAttributeStatus`](#DeleteAttributeStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributesResolver`&#x20;

K8sAttributesResolver allows adding an action to collect k8s attributes.
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`K8sAttributesResolver`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`K8sAttributesSpec`](#K8sAttributesSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`K8sAttributesStatus`](#K8sAttributesStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttribute`&#x20;

RenameAttribute is the Schema for the RenameAttribute odigos action API
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`RenameAttribute`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`RenameAttributeSpec`](#RenameAttributeSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`RenameAttributeStatus`](#RenameAttributeStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `AddClusterInfoConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `clusterAttributes` \[Required]
      </td>

      <td>
        [`[]OtelAttributeWithValue`](#OtelAttributeWithValue)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `overwriteExistingValues` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `AddClusterInfoSpec`&#x20;

**Appears in:**

* [AddClusterInfo](#AddClusterInfo)

AddClusterInfoSpec defines the desired state of AddClusterInfo action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `clusterAttributes` \[Required]
      </td>

      <td>
        [`[]OtelAttributeWithValue`](#OtelAttributeWithValue)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `overwriteExistingValues` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `AddClusterInfoStatus`&#x20;

**Appears in:**

* [AddClusterInfo](#AddClusterInfo)

AddClusterInfoStatus defines the observed state of AddClusterInfo action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a addclusterinfo's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttributeConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `attributeNamesToDelete` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttributeSpec`&#x20;

**Appears in:**

* [DeleteAttribute](#DeleteAttribute)

DeleteAttributeSpec defines the desired state of DeleteAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `attributeNamesToDelete` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttributeStatus`&#x20;

**Appears in:**

* [DeleteAttribute](#DeleteAttribute)

DeleteAttributeStatus defines the observed state of DeleteAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a DeleteAttribute's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `K8sAnnotationAttribute`&#x20;

**Appears in:**

* [K8sAttributesConfig](#K8sAttributesConfig)
* [K8sAttributesSpec](#K8sAttributesSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `annotationKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The annotation name to be extracted.
        e.g. "kubectl.kubernetes.io/restartedAt"
      </td>
    </tr>

    <tr>
      <td>
        `attributeKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The attribute key to be used for the resource attribute created from the annotation.
        e.g. "kubectl.kubernetes.restartedAt"
      </td>
    </tr>

    <tr>
      <td>
        `from` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The source of the annotation.
        e.g. "pod" or "namespace"
        Deprecated: Use FromSources instead for specifying multiple sources with precedence.
      </td>
    </tr>

    <tr>
      <td>
        `fromSources` \[Required]
      </td>

      <td>
        [`[]K8sAttributeSource`](#K8sAttributeSource)
      </td>

      <td>
        The sources from which to extract the annotation, in order of precedence (most specific first).
        When multiple sources are specified, the most specific source (e.g., pod) takes precedence
        over less specific sources (e.g., namespace).
        If an annotation exists in multiple sources, the value from the most specific source will be used.
        Supported sources: "pod", "namespace", "node"
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributeSource`&#x20;

(Alias of `string`)

**Appears in:**

* [K8sAnnotationAttribute](#K8sAnnotationAttribute)
* [K8sLabelAttribute](#K8sLabelAttribute)

## `K8sAttributesConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `collectContainerAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following container related attributes:
        k8s.container.name
        container.id
        container.image.name
        container.image.tag
      </td>
    </tr>

    <tr>
      <td>
        `collectReplicaSetAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        collect replicaset related attributes (when relevant, e.g. for deployments):
        k8s.replicaset.name
        if CollectWorkloadUID is set, also collect:
        k8s.replicaset.uid
        DEPRECATED: ReplicaSet attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectWorkloadUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following workload UID attributes:
        k8s.deployment.uid
        k8s.daemonset.uid
        k8s.statefulset.uid
        DEPRECATED: Workload UID attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectClusterUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the k8s.cluster.uid attribute, which is set to the uid of the namespace "kube-system"
      </td>
    </tr>

    <tr>
      <td>
        `labelsAttributes`
      </td>

      <td>
        [`[]K8sLabelAttribute`](#K8sLabelAttribute)
      </td>

      <td>
        list of labels to be extracted from the pod, and the attribute key to be used for the resource attribute created from each label.
      </td>
    </tr>

    <tr>
      <td>
        `annotationsAttributes`
      </td>

      <td>
        [`[]K8sAnnotationAttribute`](#K8sAnnotationAttribute)
      </td>

      <td>
        list of annotations to be extracted from the pod, and the attribute key to be used for the resource attribute created from each annotation.
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributesSpec`&#x20;

**Appears in:**

* [K8sAttributesResolver](#K8sAttributesResolver)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `collectContainerAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following container related attributes:
        k8s.container.name
        container.id
        container.image.name
        container.image.tag
      </td>
    </tr>

    <tr>
      <td>
        `collectReplicaSetAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        collect replicaset related attributes (when relevant, e.g. for deployments):
        k8s.replicaset.name
        if CollectWorkloadUID is set, also collect:
        k8s.replicaset.uid
        DEPRECATED: ReplicaSet attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectWorkloadUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following workload UID attributes:
        k8s.deployment.uid
        k8s.daemonset.uid
        k8s.statefulset.uid
        DEPRECATED: Workload UID attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectClusterUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the k8s.cluster.uid attribute, which is set to the uid of the namespace "kube-system"
      </td>
    </tr>

    <tr>
      <td>
        `labelsAttributes`
      </td>

      <td>
        [`[]K8sLabelAttribute`](#K8sLabelAttribute)
      </td>

      <td>
        list of labels to be extracted from the pod, and the attribute key to be used for the resource attribute created from each label.
      </td>
    </tr>

    <tr>
      <td>
        `annotationsAttributes`
      </td>

      <td>
        [`[]K8sAnnotationAttribute`](#K8sAnnotationAttribute)
      </td>

      <td>
        list of annotations to be extracted from the pod, and the attribute key to be used for the resource attribute created from each annotation.
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributesStatus`&#x20;

**Appears in:**

* [K8sAttributesResolver](#K8sAttributesResolver)

K8sAttributesStatus defines the observed state of K8sAttributes action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a k8sattributes' current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `K8sLabelAttribute`&#x20;

**Appears in:**

* [K8sAttributesConfig](#K8sAttributesConfig)
* [K8sAttributesSpec](#K8sAttributesSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `labelKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The label name to be extracted.
        e.g. "app.kubernetes.io/name"
      </td>
    </tr>

    <tr>
      <td>
        `attributeKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The attribute key to be used for the resource attribute created from the label.
        e.g. "app.kubernetes.name"
      </td>
    </tr>

    <tr>
      <td>
        `from` \[Required]
      </td>

      <td>
        [`K8sAttributeSource`](#K8sAttributeSource)
      </td>

      <td>
        The source of the label.
        e.g. "pod" or "namespace"
        Deprecated: Use FromSources instead for specifying multiple sources with precedence.
      </td>
    </tr>

    <tr>
      <td>
        `fromSources` \[Required]
      </td>

      <td>
        [`[]K8sAttributeSource`](#K8sAttributeSource)
      </td>

      <td>
        The sources from which to extract the label, in order of precedence (most specific first).
        When multiple sources are specified, the most specific source (e.g., pod) takes precedence
        over less specific sources (e.g., namespace).
        If a label exists in multiple sources, the value from the most specific source will be used.
        Supported sources: "pod", "namespace", "node"
      </td>
    </tr>
  </tbody>
</table>

## `OtelAttributeWithValue`&#x20;

**Appears in:**

* [AddClusterInfoConfig](#AddClusterInfoConfig)
* [AddClusterInfoSpec](#AddClusterInfoSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `attributeName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        the name of the attribute to insert
      </td>
    </tr>

    <tr>
      <td>
        `attributeStringValue` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        if the value is a string, this field should be used.
        empty string is a valid value
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttributeConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `renames` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttributeSpec`&#x20;

**Appears in:**

* [RenameAttribute](#RenameAttribute)

RenameAttributeSpec defines the desired state of RenameAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `renames` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttributeStatus`&#x20;

**Appears in:**

* [RenameAttribute](#RenameAttribute)

RenameAttributeStatus defines the observed state of RenameAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a RenameAttribute's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>


# odigos.io/v1alpha1
Source: https://docs.odigos.io/enterprise/api-reference/odigos.io.v1alpha1



Package v1alpha1 is the v1alpha1 version of the API.

## Resource Types

* [Action](#odigos-io-v1alpha1-Action)
* [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)
* [Destination](#odigos-io-v1alpha1-Destination)
* [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)
* [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)
* [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)
* [Processor](#odigos-io-v1alpha1-Processor)
* [Recommendation](#odigos-io-v1alpha1-Recommendation)
* [Sampling](#odigos-io-v1alpha1-Sampling)
* [Source](#odigos-io-v1alpha1-Source)

## `Action`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Action`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`ActionSpec`](#odigos-io-v1alpha1-ActionSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`ActionStatus`](#odigos-io-v1alpha1-ActionStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroup`&#x20;

CollectorsGroup is the Schema for the collectors API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`CollectorsGroup`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`CollectorsGroupSpec`](#odigos-io-v1alpha1-CollectorsGroupSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`CollectorsGroupStatus`](#odigos-io-v1alpha1-CollectorsGroupStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Destination`&#x20;

Destination is the Schema for the destinations API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Destination`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`DestinationSpec`](#odigos-io-v1alpha1-DestinationSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`DestinationStatus`](#odigos-io-v1alpha1-DestinationStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationConfig`&#x20;

InstrumentationConfig is the Schema for the instrumentationconfig API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`InstrumentationConfig`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`InstrumentationConfigSpec`](#odigos-io-v1alpha1-InstrumentationConfigSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`InstrumentationConfigStatus`](#odigos-io-v1alpha1-InstrumentationConfigStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationInstance`&#x20;

InstrumentationInstance is the Schema for the InstrumentationInstances API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`InstrumentationInstance`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`InstrumentationInstanceSpec`](#odigos-io-v1alpha1-InstrumentationInstanceSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`InstrumentationInstanceStatus`](#odigos-io-v1alpha1-InstrumentationInstanceStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationRule`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`InstrumentationRule`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`InstrumentationRuleSpec`](#odigos-io-v1alpha1-InstrumentationRuleSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`InstrumentationRuleStatus`](#odigos-io-v1alpha1-InstrumentationRuleStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Processor`&#x20;

Processor is the Schema for an Opentelemetry Collector Processor that is added to Odigos pipeline

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Processor`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`ProcessorSpec`](#odigos-io-v1alpha1-ProcessorSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`ProcessorStatus`](#odigos-io-v1alpha1-ProcessorStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Recommendation`&#x20;

Recommendation is the Schema for enabling a catalog recommendation in the cluster.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Recommendation`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`RecommendationSpec`](#odigos-io-v1alpha1-RecommendationSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`RecommendationStatus`](#odigos-io-v1alpha1-RecommendationStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Sampling`&#x20;

Sampling is the Schema for the sampling rules API.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Sampling`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`SamplingSpec`](#odigos-io-v1alpha1-SamplingSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`SamplingStatus`](#odigos-io-v1alpha1-SamplingStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Source`&#x20;

Source configures an application for auto-instrumentation.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Source`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`SourceSpec`](#odigos-io-v1alpha1-SourceSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`SourceStatus`](#odigos-io-v1alpha1-SourceStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `ActionSpec`&#x20;

**Appears in:**

* [Action](#odigos-io-v1alpha1-Action)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        Allows you to attach a meaningful name to the action for convenience. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        A free-form text field that allows you to attach notes regarding the action for convenience. For example: why it was added. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        A boolean field allowing to temporarily disable the action, but keep it around for future use
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        Which signals should this action operate on.
      </td>
    </tr>

    <tr>
      <td>
        `addClusterInfo` \[Required]
      </td>

      <td>
        `actions/v1alpha1.AddClusterInfoConfig`
      </td>

      <td>
        AddClusterInfo is the config for the AddClusterInfo Action.
      </td>
    </tr>

    <tr>
      <td>
        `deleteAttribute` \[Required]
      </td>

      <td>
        `actions/v1alpha1.DeleteAttributeConfig`
      </td>

      <td>
        DeleteAttribute is the config for the DeleteAttribute Action.
      </td>
    </tr>

    <tr>
      <td>
        `renameAttribute` \[Required]
      </td>

      <td>
        `actions/v1alpha1.RenameAttributeConfig`
      </td>

      <td>
        RenameAttribute is the config for the RenameAttribute Action.
      </td>
    </tr>

    <tr>
      <td>
        `piiMasking` \[Required]
      </td>

      <td>
        `actions.PiiMaskingConfig`
      </td>

      <td>
        PiiMasking is the config for the PiiMasking Action.
      </td>
    </tr>

    <tr>
      <td>
        `k8sAttributes` \[Required]
      </td>

      <td>
        `actions/v1alpha1.K8sAttributesConfig`
      </td>

      <td>
        K8sAttributes is the config for the K8sAttributes Action.
      </td>
    </tr>

    <tr>
      <td>
        `urlTemplatization` \[Required]
      </td>

      <td>
        `actions.URLTemplatizationConfig`
      </td>

      <td>
        URLTemplatization is the config for the URLTemplatization Action.
      </td>
    </tr>

    <tr>
      <td>
        `spanRenamer` \[Required]
      </td>

      <td>
        `actions.SpanRenamerConfig`
      </td>

      <td>
        SpanRenamer is the config for the SpanRenamer Action.
      </td>
    </tr>

    <tr>
      <td>
        `extractAttribute` \[Required]
      </td>

      <td>
        `actions.ExtractAttributeConfig`
      </td>

      <td>
        ExtractAttribute is the config for the ExtractAttribute Action.
      </td>
    </tr>

    <tr>
      <td>
        `dbQueryTemplatization` \[Required]
      </td>

      <td>
        `actions.DbQueryTemplatizationConfig`
      </td>

      <td>
        DbQueryTemplatization is the config for the DbQueryTemplatization Action.
      </td>
    </tr>

    <tr>
      <td>
        `inferDbAttributes` \[Required]
      </td>

      <td>
        `actions.InferDbAttributesConfig`
      </td>

      <td>
        InferDbAttributes is the config for the InferDbAttributes Action.
      </td>
    </tr>
  </tbody>
</table>

## `ActionStatus`&#x20;

**Appears in:**

* [Action](#odigos-io-v1alpha1-Action)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a action's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `AgentEnabledReason`&#x20;

(Alias of `string`)

**Appears in:**

* [ContainerAgentConfig](#odigos-io-v1alpha1-ContainerAgentConfig)

## `AgentsTelemetrySettings`&#x20;

**Appears in:**

* [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

## `Attribute`&#x20;

**Appears in:**

* [InstrumentationInstanceStatus](#odigos-io-v1alpha1-InstrumentationInstanceStatus)
* [InstrumentationLibraryStatus](#odigos-io-v1alpha1-InstrumentationLibraryStatus)

Attribute is a key-value pair that describes a component or instrumentation

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `key` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `value` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupMetricsCollectionSettings`&#x20;

**Appears in:**

* [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `spanMetrics` \[Required]
      </td>

      <td>
        [`common.MetricsSourceSpanMetricsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MetricsSourceSpanMetricsConfiguration)
      </td>

      <td>
        if not nil for node collector, it means span to metrics is enabled,
        and the node collector should set it up in the pipeline.
        span to metrics is the ability to calculate metrics like http requests/errors/duration etc
        from the individual spans recorded for relevant operation.
      </td>
    </tr>

    <tr>
      <td>
        `hostMetrics` \[Required]
      </td>

      <td>
        [`common.MetricsSourceHostMetricsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MetricsSourceHostMetricsConfiguration)
      </td>

      <td>
        if not nil for node collector, it means host metrics is enabled,
        and the opentelemetry collector "hostmetrics" receiver should be included in the pipeline.
        host metrics are metrics that are collected from the host node,
        such as cpu, memory, disk, network, etc.
      </td>
    </tr>

    <tr>
      <td>
        `kubeletStats` \[Required]
      </td>

      <td>
        [`common.MetricsSourceKubeletStatsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MetricsSourceKubeletStatsConfiguration)
      </td>

      <td>
        if not nil for node collector, it means kubelet stats is enabled,
        and the opentelemetry collector "kubeletstats" receiver should be included in the pipeline.
        kubelet stats are metrics that are collected from the kubelet point of view,
        such as cpu, memory, disk, network, per pod, node and more.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraph` \[Required]
      </td>

      <td>
        [`ServiceGraphSettings`](#odigos-io-v1alpha1-ServiceGraphSettings)
      </td>

      <td>
        if not nil for cluster collector, it means service graph is enabled,
        and metrics for the "connectivity" between services should be calculated
        to be exported to metrics destinations.
      </td>
    </tr>

    <tr>
      <td>
        `odigosOwnMetrics` \[Required]
      </td>

      <td>
        [`OdigosOwnMetricsSettings`](#odigos-io-v1alpha1-OdigosOwnMetricsSettings)
      </td>

      <td>
        if not nil for node collector, it means that some metric destinations are
        intresseted in collecting metrics about: odigos, the collected data, and the pipeline itself.
        this allows for users to monitor and operate odigos within their existing system,
        create dashboards, alerting, and more.
      </td>
    </tr>

    <tr>
      <td>
        `agentsTelemetry` \[Required]
      </td>

      <td>
        [`AgentsTelemetrySettings`](#odigos-io-v1alpha1-AgentsTelemetrySettings)
      </td>

      <td>
        this part controls the metrics which are received from agents in the otlp receiver.
        it is generally enabled when we want to record metrics, and listed here for completeness.
        any "otlp receiver" specific settings can go here
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupResourcesSettings`&#x20;

**Appears in:**

* [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

The raw values to control the collectors group resources and behavior.
any defaulting, validations and calculations should be done in the controllers
that create this CR.
Values will be used as is without any further processing.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `minReplicas` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        Minumum + Maximum number of replicas for the collector - these relevant only for gateway.
      </td>
    </tr>

    <tr>
      <td>
        `maxReplicas` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `memoryRequestMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        MemoryRequestMiB is the memory resource request to be used on the pod template.
        it will be embedded in the as a resource request of the form `memory: <value>Mi`
      </td>
    </tr>

    <tr>
      <td>
        `memoryLimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        This option sets the limit on the memory usage of the collector.
        since the memory limiter mechanism is heuristic, and operates on fixed intervals,
        while it cannot fully prevent OOMs, it can help in reducing the chances of OOMs in edge cases.
        the settings should prevent the collector from exceeding the memory request,
        so one can set this to the same value as the memory request or higher to allow for some buffer for bursts.
      </td>
    </tr>

    <tr>
      <td>
        `cpuRequestMillicores` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        CPU resource request to be used on the pod template.
        it will be embedded in the as a resource request of the form `cpu: <value>m`
      </td>
    </tr>

    <tr>
      <td>
        `cpuLimitMillicores` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        CPU resource limit to be used on the pod template.
        it will be embedded in the as a resource limit of the form `cpu: <value>m`
      </td>
    </tr>

    <tr>
      <td>
        `memoryLimiterLimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        this parameter sets the "limit\_mib" parameter in the memory limiter configuration for the collector.
        it is the hard limit after which a force garbage collection will be performed.
        this value will end up comparing against the go runtime reported heap Alloc value.
        According to the memory limiter docs:

        > Note that typically the total memory usage of process will be about 50MiB higher than this value
        > a test from nov 2024 showed that fresh odigos collector with no traffic takes 38MiB,
        > thus the 50MiB is a good value to start with.
      </td>
    </tr>

    <tr>
      <td>
        `memoryLimiterSpikeLimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        this parameter sets the "spike\_limit\_mib" parameter in the memory limiter configuration for the collector memory limiter.
        note that this is not the processor soft limit itself, but the diff in Mib between the hard limit and the soft limit.
        according to the memory limiter docs, it is recommended to set this to 20% of the hard limit.
        changing this value allows trade-offs between memory usage and resiliency to bursts.
      </td>
    </tr>

    <tr>
      <td>
        `gomemlimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        the GOMEMLIMIT environment variable value for the collector pod.
        this is when go runtime will start garbage collection.
        it is recommended to be set to 80% of the hard limit of the memory limiter.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupRole`&#x20;

(Alias of `string`)

**Appears in:**

* [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)
* [ProcessorSpec](#odigos-io-v1alpha1-ProcessorSpec)

## `CollectorsGroupSpec`&#x20;

**Appears in:**

* [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)

CollectorsGroupSpec defines the desired state of Collector

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `role` \[Required]
      </td>

      <td>
        [`CollectorsGroupRole`](#odigos-io-v1alpha1-CollectorsGroupRole)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `collectorOwnMetricsPort` \[Required]
      </td>

      <td>
        `int32`
      </td>

      <td>
        The port to use for exposing the collector's own metrics as a prometheus endpoint.
        This can be used to resolve conflicting ports when a collector is using the host network.
      </td>
    </tr>

    <tr>
      <td>
        `resourcesSettings` \[Required]
      </td>

      <td>
        [`CollectorsGroupResourcesSettings`](#odigos-io-v1alpha1-CollectorsGroupResourcesSettings)
      </td>

      <td>
        Resources \[memory/cpu] settings for the collectors group.
        these settings are used to protect the collectors instances from:

        <ul>
          <li>running out of memory and being killed by the k8s OOM killer</li>
          <li>consuming all available memory on the node which can lead to node instability</li>
          <li>pushing back pressure to the instrumented applications</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphDisabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        ServiceGraphEnabled is a feature that allows you to visualize the service graph of your application.
        It is enabled by default and can be disabled by setting the enabled flag to false.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphExtraDimensions` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        ServiceGraphExtraDimensions are additional span attribute names to include as dimensions
        in the service graph metrics, on top of the default service.name dimension.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphVirtualNodePeerAttributes` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        ServiceGraphVirtualNodePeerAttributes is an ordered list of span attributes used to identify
        uninstrumented (virtual) nodes in the service graph.
        The connector picks the first attribute that has a value on the span.
        When nil/empty, the connector uses its built-in defaults: \[peer.service, db.name, db.system].
      </td>
    </tr>

    <tr>
      <td>
        `enableDataCompression` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Deprecated - use OtlpExporterConfiguration instead.
        EnableDataCompression is a feature that allows you to enable data compression before sending data to the Gateway collector.
        It is disabled by default and can be enabled by setting the enabled flag to true.
      </td>
    </tr>

    <tr>
      <td>
        `otlpExporterConfiguration` \[Required]
      </td>

      <td>
        [`common.OtlpExporterConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#OtlpExporterConfiguration)
      </td>

      <td>
        OtlpExporterConfiguration is the configuration for the OTLP exporter from node collector to cluster gateway collector.
      </td>
    </tr>

    <tr>
      <td>
        `resourceDetectors` \[Required]
      </td>

      <td>
        [`common.ResourceDetectorsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ResourceDetectorsConfiguration)
      </td>

      <td>
        ResourceDetectors controls which OpenTelemetry resource detectors are enabled
        on the node collector's resourcedetection processor.
      </td>
    </tr>

    <tr>
      <td>
        `clusterMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        ClusterMetricsEnabled is a feature that allows you to enable the cluster metrics.
        It is disabled by default and can be enabled by setting the enabled flag to true.
      </td>
    </tr>

    <tr>
      <td>
        `httpsProxyAddress` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        for destinations that uses https for exporting data, this value can be used to set the address for an https proxy.
        when unset or empty, no proxy will be used.
      </td>
    </tr>

    <tr>
      <td>
        `metrics` \[Required]
      </td>

      <td>
        [`CollectorsGroupMetricsCollectionSettings`](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)
      </td>

      <td>
        configuration for metrics handling in this collectors group
        if metric collection is disabled, this will be nil
        the content is populated only if relevant to this collectors group
        it's being calculated based on active destinations and their settings,
        and global settings provided in the odigos configuration or instrumentation rules.
        it allows for the collector group reconciler to be simplified,
        and for visibility into the aggregated settings being used to derive configurations deployments and rollouts.
      </td>
    </tr>

    <tr>
      <td>
        `nodeSelector` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        Node selector for the collectors group deployment.
        Use this to force the gateway to run only on nodes with specific labels.
        This is a hard requirement: the pod will be scheduled ONLY on nodes that match all labels.
        If no matching nodes exist, the pod will remain Pending.
      </td>
    </tr>

    <tr>
      <td>
        `deploymentName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        Deployment name for the collectors group deployment.
        Only relevant for cluster gateway collector.
      </td>
    </tr>

    <tr>
      <td>
        `tailSampling` \[Required]
      </td>

      <td>
        `common/api/sampling.TailSamplingConfiguration`
      </td>

      <td>
        Sampling holds the sampling configuration derived from the OdigosConfiguration.
        Currently this is only relevant for the cluster gateway collector.
      </td>
    </tr>

    <tr>
      <td>
        `samplingDryRun` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Set to true to enable dry run mode for sampling.
        When enabled, odigos will invoke the sampling logic, but will not drop any traces.
        This is useful while evaluating sampling rules to check for effectiveness and correctness
        before committing to any changes that might lose data.
      </td>
    </tr>

    <tr>
      <td>
        `spanSamplingAttributes` \[Required]
      </td>

      <td>
        `common/api/sampling.SpanSamplingAttributesConfiguration`
      </td>

      <td>
        Controls whether spans are enhanced with sampling attributes (e.g. category and decisions).
        Capturing these attributes gives visibility into sampling decision-making and effective
        sampling percentages when viewing traces or querying the database with tools.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupStatus`&#x20;

**Appears in:**

* [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)

CollectorsGroupStatus defines the observed state of Collector

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `ready` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `receiverSignals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        Receiver Signals are the signals (trace, metrics, logs) that the collector has setup
        an otlp receiver for, thus it can accept data from an upstream component.
        this is used to determine if a workload should export each signal or not.
        this list is calculated based on the odigos destinations that were configured
      </td>
    </tr>

    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a collectorsroup's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `ContainerAgentConfig`&#x20;

**Appears in:**

* [InstrumentationConfigSpec](#odigos-io-v1alpha1-InstrumentationConfigSpec)

ContainerAgentConfig is a configuration for a specific container in a workload.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the container to which this configuration applies.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        boolean flag to indicate if the agent should be enabled for this container.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnabledReason` \[Required]
      </td>

      <td>
        [`AgentEnabledReason`](#odigos-io-v1alpha1-AgentEnabledReason)
      </td>

      <td>
        An enum reason for the agent injection decision.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnabledMessage` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        free text message to provide more information about the instrumentation decision.
        can be left empty if reason is self-explanatory.
      </td>
    </tr>

    <tr>
      <td>
        `podManifestInjectionOptional` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        set to true if the agent used in this container can be injected without pod manifest changes.
      </td>
    </tr>

    <tr>
      <td>
        `otelDistroName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the otel distribution to use for this container.
        if the name is empty, this container should not be instrumented.
      </td>
    </tr>

    <tr>
      <td>
        `distroParams` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        Additional parameters to the distro that controls how it's being applied.
        Keys are parameter names (like "libc") and values are the value to use for that parameter (glibc / musl)
      </td>
    </tr>

    <tr>
      <td>
        `envInjectionMethod` \[Required]
      </td>

      <td>
        [`common.EnvInjectionDecision`](https://pkg.go.dev/github.com/odigos-io/odigos/common#EnvInjectionDecision)
      </td>

      <td>
        What method to use for injecting the agent environment variables (just those covered by the loader (PYTHONPATH, JAVA\_TOOLS\_OPTIONS, NODE\_OPTIONS))
        Can be either "loader" or "pod-manifest".
        The injection should still check the actual values in the container manifest before injecting.
        Nil means that this container should not have env injection (agent should not be injected, or distro does not specify "loader" injection envs).
      </td>
    </tr>

    <tr>
      <td>
        `traces` \[Required]
      </td>

      <td>
        `common/api/agentsignalconfig.AgentTracesConfig`
      </td>

      <td>
        Each enabled signal must be set with a non-nil value (even if the config content is empty).
        nil means that the signal is disabled and should not be instrumented/collected by the agent.
      </td>
    </tr>

    <tr>
      <td>
        `metrics` \[Required]
      </td>

      <td>
        `common/api/agentsignalconfig.AgentMetricsConfig`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `logs` \[Required]
      </td>

      <td>
        `common/api/agentsignalconfig.AgentLogsConfig`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `agentDiagnostics` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.AgentDiagnostics`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#AgentDiagnostics)
      </td>

      <td>
        Configure the log level for the agent itselg
      </td>
    </tr>
  </tbody>
</table>

## `ContainerOverride`&#x20;

**Appears in:**

* [InstrumentationConfigSpec](#odigos-io-v1alpha1-InstrumentationConfigSpec)
* [SourceSpec](#odigos-io-v1alpha1-SourceSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the container to override.
      </td>
    </tr>

    <tr>
      <td>
        `runtimeInfo` \[Required]
      </td>

      <td>
        [`RuntimeDetailsByContainer`](#odigos-io-v1alpha1-RuntimeDetailsByContainer)
      </td>

      <td>
        RuntimeInfo to use for agent enabling.
        If set for a container, the automatic detection will not be used for this container,
        and the distro to use will be calculated based on this value.
      </td>
    </tr>

    <tr>
      <td>
        `otelDistroName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        select one specific otel distro to use for this container.
        the selected distro must match the programming language and any constraints of the distro.
        if the programmin language is incorrect, the runtime info can be overridden to match
      </td>
    </tr>

    <tr>
      <td>
        `allowConcurrentAgents` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Odigos by default will attempt to detect if another agent is already running in the container,
        and if so, will avoid adding odigos agent to the container.
        This option allows to override this setting for a specific container in a source.
        Can be used for selective setting of which containers should run odigos concurrently with other agents.
        empty means take odigos global config for this setting.
        when not empty, the value will override the global config for this container
      </td>
    </tr>
  </tbody>
</table>

## `CostReductionRule`&#x20;

**Appears in:**

* [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user provided name, for easier identification and reference.
        use short and descriptive name, like "health check", "always keep /transaction in payment service", etc.
        odigos does not use or assume any meaning from this field,
        but it is written as metric attribute, and stored as span attribute on participating spans.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if set to true, the rule will be disabled,
        e.g. will not be taken into account for any sampling decisions.
        disabled rules still participate in metrics calculations,
        allowing enhanced tools and data for troubleshooting and sampling maintenance.
      </td>
    </tr>

    <tr>
      <td>
        `sourceScopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        limit this rule to specific sources (by name, namespace, language, etc.)
        an empty list will match any source.
        if multiple items are set, the operation match if any one matches
        this relates to the "ResourceAttributes" part of a span.
      </td>
    </tr>

    <tr>
      <td>
        `operation` \[Required]
      </td>

      <td>
        `common/api/sampling.TailSamplingOperationMatcher`
      </td>

      <td>
        limit this rule to specific operations.
        for example: specific endpoint or kafka topic.
        this field is optional, and if not set, the rule will be applied to all operations.
      </td>
    </tr>

    <tr>
      <td>
        `percentageAtMost` \[Required]
      </td>

      <td>
        `float64`
      </td>

      <td>
        sampling percentage for cost reduction.
        this field is required.
        the final sampling percentage for traces that match this rule
        will be the highest possible, but at most this value.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        optional free-form text field that allows you to attach notes
        for future context and maintenance.
        users can write why this rule was added, observations, document considerations, etc.
      </td>
    </tr>
  </tbody>
</table>

## `DestinationMetricsSettings`&#x20;

**Appears in:**

* [DestinationSpec](#odigos-io-v1alpha1-DestinationSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `spanMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        following settings enables/disables specific metric sources
        true - collect and send this metric source
        false - do not collect and send this metric source
        nil - use the default setting (from destination manifest, or cluster global setting)
      </td>
    </tr>

    <tr>
      <td>
        `hostMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `kubeletStatsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `odigosOwnMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `agentsTelemetryEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DestinationSpec`&#x20;

**Appears in:**

* [Destination](#odigos-io-v1alpha1-Destination)

DestinationSpec defines the desired state of Destination

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        [`common.DestinationType`](https://pkg.go.dev/github.com/odigos-io/odigos/common#DestinationType)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `destinationName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `data` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `secretRef` \[Required]
      </td>

      <td>
        [`core/v1.LocalObjectReference`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#localobjectreference-v1-core)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `metricsSettings`
      </td>

      <td>
        [`DestinationMetricsSettings`](#odigos-io-v1alpha1-DestinationMetricsSettings)
      </td>

      <td>
        MetricsSettings defines the metrics settings for this destination.
        ignored if destination does not support metrics or metrics are not enabled in signals
      </td>
    </tr>

    <tr>
      <td>
        `sourceSelector`
      </td>

      <td>
        [`SourceSelector`](#odigos-io-v1alpha1-SourceSelector)
      </td>

      <td>
        SourceSelector defines which sources can send data to this destination.
        If not specified, defaults to "all".
      </td>
    </tr>
  </tbody>
</table>

## `DestinationStatus`&#x20;

**Appears in:**

* [Destination](#odigos-io-v1alpha1-Destination)

DestinationStatus defines the observed state of Destination

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a destination's current state.
      </td>
    </tr>
  </tbody>
</table>

## `EnvVar`&#x20;

**Appears in:**

* [RuntimeDetailsByContainer](#odigos-io-v1alpha1-RuntimeDetailsByContainer)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `value` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `HighlyRelevantOperation`&#x20;

**Appears in:**

* [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

define operations (spans) with high observability value.
if found anywhere in the trace, the entire trace will be kept
regaradless of any cost reduction rules.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user provided name, for easier identification and reference.
        use short and descriptive name, like "health check", "always keep /transaction in payment service", etc.
        odigos does not use or assume any meaning from this field,
        but it is written as metric attribute, and stored as span attribute on participating spans.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if set to true, the rule will be disabled,
        e.g. will not be taken into account for any sampling decisions.
        disabled rules still participate in metrics calculations,
        allowing enhanced tools and data for troubleshooting and sampling maintenance.
      </td>
    </tr>

    <tr>
      <td>
        `sourceScopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        limit the operation to specific sources.
        an empty list will match any source.
        if multiple items are set, the operation match if any one matches
        this relates to the "ResourceAttributes" part of a span.
      </td>
    </tr>

    <tr>
      <td>
        `error` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if "Error" is set to true, only spans with SpanStatus set to "Error" are considered
      </td>
    </tr>

    <tr>
      <td>
        `durationAtLeastMs` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        if Duration is set, only operations with duration in milli seconds larger then this value are considered
      </td>
    </tr>

    <tr>
      <td>
        `operation` \[Required]
      </td>

      <td>
        `common/api/sampling.TailSamplingOperationMatcher`
      </td>

      <td>
        optionally, limit this rule to specific operations.
        for example: specific endpoint or kafka topic.
        this field is optional, and if not set, the rule will be applied to all operations.
      </td>
    </tr>

    <tr>
      <td>
        `percentageAtLeast` \[Required]
      </td>

      <td>
        `float64`
      </td>

      <td>
        traces that contains this operation will be sampled by at least this percentage.
        if unset, 100% of such the traces will be sampled.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        optional free-form text field that allows you to attach notes
        for future context and maintenance.
        users can write why this rule was added, observations, document considerations, etc.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationConfigSpec`&#x20;

**Appears in:**

* [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)

Config for the OpenTelemeetry SDKs that should be applied to a workload.
The workload is identified by the owner reference

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `serviceName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        the service.name property is used to populate the `service.name` resource attribute in the telemetry generated by this workload
      </td>
    </tr>

    <tr>
      <td>
        `agentInjectionEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        determines if odigos should inject agents to pods of this workload.
      </td>
    </tr>

    <tr>
      <td>
        `podManifestInjectionOptional` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        true if all containers with agent injection enabled in this source
        can be enabled without pod manifest changes.
        if false, at least one container requires pod manifest changes to enable agent injection.
      </td>
    </tr>

    <tr>
      <td>
        `containers` \[Required]
      </td>

      <td>
        [`[]ContainerAgentConfig`](#odigos-io-v1alpha1-ContainerAgentConfig)
      </td>

      <td>
        configuration for each instrumented container in the workload
      </td>
    </tr>

    <tr>
      <td>
        `workloadCollectorConfig` \[Required]
      </td>

      <td>
        `[]common/api.ContainerCollectorConfig`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `containersOverrides` \[Required]
      </td>

      <td>
        [`[]ContainerOverride`](#odigos-io-v1alpha1-ContainerOverride)
      </td>

      <td>
        will always list all containers of this workload by name,
        and override data in case it is configured on the source.
        this peoperty can be used to know all container names in the workload, even if other controllers did not yet run.
      </td>
    </tr>

    <tr>
      <td>
        `containerOverridesHash` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        An hash of the containers overrides, used to determine if the overrides have changed for event filtering.
        this is updated only when the overrides are changed, and not when some other change in the source occurs.
      </td>
    </tr>

    <tr>
      <td>
        `agentsMetaHash` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        this hash is used to determine the deployment of the agents.
        e.g. when the distro for container changes, or it's compatibility version,
        or something else that requires rollout, the hash change will indicate that.
        if the hash is empty, it means that no agent should be enabled in any pod container.
      </td>
    </tr>

    <tr>
      <td>
        `agentsMetaHashChangedTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        The last time at which the agents meta hash value was changed.
        Pods created before this time may not be in alignment with the AgentsMetaHash.
        e.g. can lack the odigos label, or have a different value.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationConfigStatus`&#x20;

**Appears in:**

* [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `runtimeDetailsByContainer` \[Required]
      </td>

      <td>
        [`[]RuntimeDetailsByContainer`](#odigos-io-v1alpha1-RuntimeDetailsByContainer)
      </td>

      <td>
        Capture Runtime Details for the workloads that this CR applies to.
      </td>
    </tr>

    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a InstrumentationConfig's current state.
      </td>
    </tr>

    <tr>
      <td>
        `workloadRolloutHash` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        This hash is recorded only after the rollout took place.
        it allows us to determine if the workload needs to be rollout based on previous rollout and the current config.
        if this field is different than the spec.AgentsDeploymentHash it means rollout is needed or not yet updated.
      </td>
    </tr>

    <tr>
      <td>
        `rollbackOccurred` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Check if rollback happened to an application
      </td>
    </tr>

    <tr>
      <td>
        `instrumentationTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        This time recorded only after the rollout took place.
        This allows us to determine whether a crashing application should be rolled back or not
      </td>
    </tr>

    <tr>
      <td>
        `podsManifestInjectionStatus` \[Required]
      </td>

      <td>
        [`PodsManifestInjectionStatus`](#odigos-io-v1alpha1-PodsManifestInjectionStatus)
      </td>

      <td>
        Represents the status of odigos MANIFEST injection to existing pods template.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationInstanceSpec`&#x20;

**Appears in:**

* [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        stores the name of the container in the pod where the SDK is running.
        The pod details can be found as the owner reference on the CR.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationInstanceStatus`&#x20;

**Appears in:**

* [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)

InstrumentationInstanceStatus defines the observed state of InstrumentationInstance
If the instrumentation is not active, this CR should be deleted

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `identifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that identify the SDK and are reported as resource attributes in the generated telemetry.
        One can identify if an arbitrary telemetry is generated by this SDK by checking those resource attributes.
      </td>
    </tr>

    <tr>
      <td>
        `nonIdentifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that are not reported as resource attributes but useful to describe characteristics of the SDK.
      </td>
    </tr>

    <tr>
      <td>
        `healthy` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Healthy true means that the odigos agent has started the SDK, and there are no errors. User can expect telemetry to be generated.
        Healthy false means that the agent has stopped and telemetry data is not expected to be generated.
        Healthy nil means that the agent did not report any health status yet (prefer to always report health status).
      </td>
    </tr>

    <tr>
      <td>
        `message` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        message is a human readable message indicating details about the SDK general health.
        can be omitted if healthy is true
      </td>
    </tr>

    <tr>
      <td>
        `reason` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        reason contains a programmatic identifier indicating the reason for the component status.
        Producers of specific condition types may define expected values and meanings for this field,
        and whether the values are considered a guaranteed API.
      </td>
    </tr>

    <tr>
      <td>
        `lastStatusTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `components` \[Required]
      </td>

      <td>
        [`[]InstrumentationLibraryStatus`](#odigos-io-v1alpha1-InstrumentationLibraryStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryConfigTraces`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `enabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Whether the instrumentation library is enabled to record traces.
        When false, it is expected that the instrumentation library does not produce any spans regardless of any other configuration.
        When true, the instrumentation library should produce spans according to the other configuration options.
        If not specified, the default value for this signal should be used (whether to enable libraries by default or not).
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryId`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `libraryName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the instrumentation library

        <ul>
          <li>Node.js: The name of the npm package: `@opentelemetry/instrumentation-<name>`</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        `spanKind` \[Required]
      </td>

      <td>
        [`common.SpanKind`](https://pkg.go.dev/github.com/odigos-io/odigos/common#SpanKind)
      </td>

      <td>
        SpanKind is only supported by Golang and will be ignored for any other SDK language.
        In Go, SpanKind is used because the same instrumentation library can be utilized for different span kinds (e.g., client/server).
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryStatus`&#x20;

**Appears in:**

* [InstrumentationInstanceStatus](#odigos-io-v1alpha1-InstrumentationInstanceStatus)

InstrumentationLibraryStatus defines the observed state of an InstrumentationLibrary.
if a library is not active/disable, it should not be included in the status

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        for example ("net/http", "@opentelemetry/instrumentation-redis")
      </td>
    </tr>

    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        [`InstrumentationLibraryType`](#odigos-io-v1alpha1-InstrumentationLibraryType)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `identifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that identify the component.
        The combination of (Name, Type, IdentifyingAttributes) must be unique.
      </td>
    </tr>

    <tr>
      <td>
        `nonIdentifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that do not necessarily identify the component but help describe
        its characteristics.
      </td>
    </tr>

    <tr>
      <td>
        `healthy` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `message` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        message is a human readable message indicating details about the component health.
        can be omitted if healthy is true
      </td>
    </tr>

    <tr>
      <td>
        `reason` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        reason contains a programmatic identifier indicating the reason for the SDK status.
        Producers of specific condition types may define expected values and meanings for this field,
        and whether the values are considered a guaranteed API.
      </td>
    </tr>

    <tr>
      <td>
        `lastStatusTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryType`&#x20;

(Alias of `string`)

**Appears in:**

* [InstrumentationLibraryStatus](#odigos-io-v1alpha1-InstrumentationLibraryStatus)

## `InstrumentationRuleSpec`&#x20;

**Appears in:**

* [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `ruleName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        Allows you to attach a meaningful name to the rule for convenience. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        A free-form text field that allows you to attach notes regarding the rule for convenience. For example: why it was added. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        A boolean field allowing to temporarily disable the rule, but keep it around for future use
      </td>
    </tr>

    <tr>
      <td>
        `scopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        SourcesScopes lists SourcesScope entries to which the rule should be applied. If unset or empty, the rule applies to all workloads.
      </td>
    </tr>

    <tr>
      <td>
        `instrumentationLibraries` \[Required]
      </td>

      <td>
        [`[]github.com/odigos-io/odigos/api/odigos/v1alpha1.InstrumentationLibraryGlobalId`](#odigos-io-v1alpha1-\[]github-com-odigos-io-odigos-api-odigos-v1alpha1-InstrumentationLibraryGlobalId)
      </td>

      <td>
        For fine grained control, the user can specify the instrumentation library to use.
        One can specify same rule for multiple languages and libraries at the same time.
        If nil, all instrumentation libraries will be used.
        If empty, no instrumentation libraries will be used.
      </td>
    </tr>

    <tr>
      <td>
        `payloadCollection` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.PayloadCollection`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#PayloadCollection)
      </td>

      <td>
        Allows to configure payload collection aspects for different types of payloads.
      </td>
    </tr>

    <tr>
      <td>
        `otelDistros` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.OtelDistros`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#OtelDistros)
      </td>

      <td>
        Set the otel distros to use instead of the defaults.
      </td>
    </tr>

    <tr>
      <td>
        `codeAttributes` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.CodeAttributes`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#CodeAttributes)
      </td>

      <td>
        Configure which code attributes should be recorded as span attributes.
      </td>
    </tr>

    <tr>
      <td>
        `headersCollection` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.HttpHeadersCollection`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#HttpHeadersCollection)
      </td>

      <td>
        Allows to configure the collection of http headers for different types of payloads.
      </td>
    </tr>

    <tr>
      <td>
        `traceConfig` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.TraceConfig`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#TraceConfig)
      </td>

      <td>
        Configure the tracing configuration for the library.
      </td>
    </tr>

    <tr>
      <td>
        `customInstrumentations` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.CustomInstrumentationsWithMetadata`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#CustomInstrumentationsWithMetadata)
      </td>

      <td>
        Add custom instrumentation probes
      </td>
    </tr>

    <tr>
      <td>
        `headSamplingFallbackFraction` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.HeadSamplingFallbackFraction`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#HeadSamplingFallbackFraction)
      </td>

      <td>
        Configure general fraction of traces to record if none of the rules evaluate to true.
        this can help in reducing very noisy and heavy traffic services.
        note that traces will be dropped regardless of thier attributes/errors/importance.
        @deprecated: use odigos config to set this value instead.
      </td>
    </tr>

    <tr>
      <td>
        `ebpfLogCapture` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.EbpfLogCapture`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#EbpfLogCapture)
      </td>

      <td>
        Configure eBPF-based log capture. When enabled, the node collector logs pipeline
        will use only the eBPF receiver instead of the filelog receiver, and odiglet will
        register instrumented processes for eBPF log capture.
      </td>
    </tr>

    <tr>
      <td>
        `networkMetrics` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.NetworkMetricsConfig`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#NetworkMetricsConfig)
      </td>

      <td>
        Configure network flow and TCP stats metrics for scoped workloads.
      </td>
    </tr>

    <tr>
      <td>
        `traceVerbosity` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.TraceVerbosity`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#TraceVerbosity)
      </td>

      <td>
        Configure the verbosity of the traces for the library.
      </td>
    </tr>

    <tr>
      <td>
        `agentDiagnostics` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.AgentDiagnostics`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#AgentDiagnostics)
      </td>

      <td>
        Configure the agent own logging configuration.
      </td>
    </tr>

    <tr>
      <td>
        `kernelTraces` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.KernelTraces`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#KernelTraces)
      </td>

      <td>
        Adds spans for the operating system work of the scoped workloads, from any language, like the
        files they open and the processes they start. A span is a child of the span in progress when the
        work happens. File and process spans start a trace when there is none.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationRuleStatus`&#x20;

**Appears in:**

* [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a instrumentationrule's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `NoisyOperation`&#x20;

**Appears in:**

* [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

endpoints (or other operations) which are considered "noise", and provide no or very little observability value.
these traces should not be collected at all, or dropped aggresevly.
motivation is data sentization and performance improvment (even if cost is not a factor)examples:

<ul>
  <li>health-checks (readiness and liveness probes)</li>
  <li>metrics scrape endpoints (prometheus /metrics endpoint)</li>
  <li>other agents calling home (outgoing http requests to collector.my.vendor.com)</li>
</ul>

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user provided name, for easier identification and reference.
        use short and descriptive name, like "health check", "always keep /transaction in payment service", etc.
        odigos does not use or assume any meaning from this field,
        but it is written as metric attribute, and stored as span attribute on participating spans.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if set to true, the rule will be disabled,
        e.g. will not be taken into account for any sampling decisions.
        disabled rules still participate in metrics calculations,
        allowing enhanced tools and data for troubleshooting and sampling maintenance.
      </td>
    </tr>

    <tr>
      <td>
        `sourceScopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        limit this rule to specific sources (by name, namespace, language, etc.)
        for example: if "other agent" rule for noisty operation is relevant only in java,
        limit this rule by setting source scope to java and prevent other languages from being affected
        if the list is empty - all sources are matched.
      </td>
    </tr>

    <tr>
      <td>
        `operation` \[Required]
      </td>

      <td>
        `common/api/sampling.HeadSamplingOperationMatcher`
      </td>

      <td>
        limit this rule to specific operations.
        for example: specific http server endpoint (GET "/healthz" as an example).
        this field is optional, and if not set, the rule will be applied to all operations.
      </td>
    </tr>

    <tr>
      <td>
        `percentageAtMost` \[Required]
      </td>

      <td>
        `float64`
      </td>

      <td>
        sampling percentage for noisy operations.
        if unset, 0% of such the traces will be collected.
        this percentage has "at most" semantics - the final sampling percentage for traces that match this rule
        will be the highest possible, but at most this value.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        optional free-form text field that allows you to attach notes
        for future context and maintenance.
        users can write why this rule was added, observations, document considerations, etc.
      </td>
    </tr>
  </tbody>
</table>

## `OdigosOwnMetricsSettings`&#x20;

**Appears in:**

* [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

configuration for collecting and exporting odigos own metrics.
e.g. metrics about odigos components, and not about the user's application.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `sendToMetricsDestinations` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if true, odigos will send all the metrics it collects about itself to the metrics pipeline,
        which will make them available to the metrics destinations.
        users can troubleshoot odigos itself by monitoring these metrics in their existing systems,
        and create their own dashboards, alerting, and more.
      </td>
    </tr>

    <tr>
      <td>
        `sendToOdigosMetricsStore` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if true, odigos will send all the metrics it collects about itself to the odigos metrics store,
        which is available to odigos UI.
        it can help in presenting a consistent view of odigos itself, without relying on user system and integrations.
      </td>
    </tr>

    <tr>
      <td>
        `interval` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        time interval for flusing metrics (format: 15s, 1m etc). defaults: 10s
      </td>
    </tr>
  </tbody>
</table>

## `OtherAgent`&#x20;

**Appears in:**

* [RuntimeDetailsByContainer](#odigos-io-v1alpha1-RuntimeDetailsByContainer)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `PodsManifestInjectionStatus`&#x20;

**Appears in:**

* [InstrumentationConfigStatus](#odigos-io-v1alpha1-InstrumentationConfigStatus)

represents the status of odigos MANIFEST injection to existing pods template.
each pod can be in one of 3 states: injected and up-to-date, injected and out-of-date, or not injected.
actual agent in pod container can be injected or not regardless of the pod manifest injection status.
status only shows if pods exists from each category and not count, to avoid frequent (noisy/expensive) updates in steady state

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `hasInjectedUpToDatePods` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if the source has any pods which are using the latest agent deployment setup.
      </td>
    </tr>

    <tr>
      <td>
        `hasInjectedOutOfDatePods` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if the source has any pods which are using an outdated agent deployment setup.
      </td>
    </tr>

    <tr>
      <td>
        `hasUninjectedPods` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if the source has any pods which do not have the odigos agent injected.
      </td>
    </tr>
  </tbody>
</table>

## `ProcessorSpec`&#x20;

**Appears in:**

* [Processor](#odigos-io-v1alpha1-Processor)

ProcessorSpec defines the an OpenTelemetry Collector processor in odigos telemetry pipeline

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        type of the processor (batch, attributes, etc).
        this field is only the type, not it's instance name in the collector configuration yaml
      </td>
    </tr>

    <tr>
      <td>
        `processorName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        this name is solely for the user convenience, to attach a meaningful name to the processor.
        odigos must not assume any semantics from this name.
        odigos cannot assume this name is unique, not empty, exclude spaces or dots, limited in length, etc.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user can attach notes to the processor, to document its purpose, usage, etc.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        disable is a flag to enable or disable the processor.
        if the processor is disabled, it will not be included in the collector configuration yaml.
        this allows the user to keep the processor configuration in the CR, but disable it temporarily.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        signals can be used to control which observability signals are processed by the processor.
      </td>
    </tr>

    <tr>
      <td>
        `collectorRoles` \[Required]
      </td>

      <td>
        [`[]CollectorsGroupRole`](#odigos-io-v1alpha1-CollectorsGroupRole)
      </td>

      <td>
        control which collector roles in odigos pipeline this processor is attached to.
      </td>
    </tr>

    <tr>
      <td>
        `orderHint` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        control the order of processors.
        a processor with lower order hint value will be placed before other processors with higher value.
        if 2 processors have the same value, the order is arbitrary.
        if the value is missing (or 0) the processor can be placed anywhere in the pipeline
      </td>
    </tr>

    <tr>
      <td>
        `processorConfig` \[Required]
      </td>

      <td>
        [`k8s.io/apimachinery/pkg/runtime.RawExtension`](https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime/#RawExtension)
      </td>

      <td>
        this it the configuration of the opentelemetry collector processor component with the type specified in 'type'.
      </td>
    </tr>
  </tbody>
</table>

## `ProcessorStatus`&#x20;

**Appears in:**

* [Processor](#odigos-io-v1alpha1-Processor)

ProcessorStatus defines the observed state of the processor

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a processor's current state.
      </td>
    </tr>
  </tbody>
</table>

## `RecommendationSpec`&#x20;

**Appears in:**

* [Recommendation](#odigos-io-v1alpha1-Recommendation)

RecommendationSpec defines the desired state of Recommendation

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        [`common.RecommendationType`](https://pkg.go.dev/github.com/odigos-io/odigos/common#RecommendationType)
      </td>

      <td>
        Type identifies which recommendation from the catalog this resource enables.
      </td>
    </tr>

    <tr>
      <td>
        `applied`
      </td>

      <td>
        `bool`
      </td>

      <td>
        Applied indicates whether this recommendation is currently applied
        (via the recommendation flow or configured organically).
      </td>
    </tr>

    <tr>
      <td>
        `conditionsMet`
      </td>

      <td>
        `bool`
      </td>

      <td>
        ConditionsMet indicates whether the catalog prerequisites for this recommendation
        currently hold (for example GoEnterpriseSources for AutoGoOffsetUpdater).
      </td>
    </tr>
  </tbody>
</table>

## `RecommendationStatus`&#x20;

**Appears in:**

* [Recommendation](#odigos-io-v1alpha1-Recommendation)

RecommendationStatus defines the observed state of Recommendation

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a recommendation's current state.
      </td>
    </tr>
  </tbody>
</table>

## `RuntimeDetailsByContainer`&#x20;

**Appears in:**

* [ContainerOverride](#odigos-io-v1alpha1-ContainerOverride)
* [InstrumentationConfigStatus](#odigos-io-v1alpha1-InstrumentationConfigStatus)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `language` \[Required]
      </td>

      <td>
        [`common.ProgrammingLanguage`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ProgrammingLanguage)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `runtimeVersion` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `envVars` \[Required]
      </td>

      <td>
        [`[]EnvVar`](#odigos-io-v1alpha1-EnvVar)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `otherAgents` \[Required]
      </td>

      <td>
        [`[]OtherAgent`](#odigos-io-v1alpha1-OtherAgent)
      </td>

      <td>
        OtherAgents lists other instrumentation agents detected in the container
        (a process can carry more than one); empty when none.
      </td>
    </tr>

    <tr>
      <td>
        `libCType` \[Required]
      </td>

      <td>
        [`common.LibCType`](https://pkg.go.dev/github.com/odigos-io/odigos/common#LibCType)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `secureExecutionMode` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Indicates whether the target process is running is secure-execution mode.
        nil means we were unable to determine the secure-execution mode.
      </td>
    </tr>

    <tr>
      <td>
        `criErrorMessage` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        CriErrorMessage is set if the value in EnvFromContainerRuntime was not computed correctly and cannot be used safely.
        Sometimes, even if CRI check failed, it is possible to tell that relevant env vars are not coming from container runtime.
        Thus, this field is set only when there is:

        <ul>
          <li>Actual CRI check failed</li>
          <li>The observed environment variables might come from container runtime</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        `envFromContainerRuntime` \[Required]
      </td>

      <td>
        [`[]EnvVar`](#odigos-io-v1alpha1-EnvVar)
      </td>

      <td>
        Holds the environment variables retrieved from the container runtime.
      </td>
    </tr>
  </tbody>
</table>

## `SamplingSpec`&#x20;

**Appears in:**

* [Sampling](#odigos-io-v1alpha1-Sampling)

define sampling rules.
the rules can be defined as one or multiple objects in kubernetes,
and are all joined together to form the global sampling rules.
odigos users can group rules based on whatever criteria that makes sense for them,
for example - by team, by client, by usecase, admin-policy, etc.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        give these sampling rules a name for display, easier identification and reference.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        a free-form text field that allows you to attach notes regardinag the rule for convenience.
        Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `noisyOperations` \[Required]
      </td>

      <td>
        [`[]NoisyOperation`](#odigos-io-v1alpha1-NoisyOperation)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `highlyRelevantOperations` \[Required]
      </td>

      <td>
        [`[]HighlyRelevantOperation`](#odigos-io-v1alpha1-HighlyRelevantOperation)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `costReductionRules` \[Required]
      </td>

      <td>
        [`[]CostReductionRule`](#odigos-io-v1alpha1-CostReductionRule)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `SamplingStatus`&#x20;

**Appears in:**

* [Sampling](#odigos-io-v1alpha1-Sampling)

SamplingStatus defines the observed state of Sampling.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a Sampling's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `ServiceGraphSettings`&#x20;

**Appears in:**

* [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

## `SourceSelector`&#x20;

**Appears in:**

* [DestinationSpec](#odigos-io-v1alpha1-DestinationSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `namespaces`
      </td>

      <td>
        `[]string`
      </td>

      <td>
        If a namespace is specified, all workloads (sources) within that namespace are allowed to send data.
        Example:
        namespaces: \["default", "production"]
        This means the destination will receive data from all sources in "default" and "production" namespaces.
      </td>
    </tr>

    <tr>
      <td>
        `dataStreams`
      </td>

      <td>
        `[]string`
      </td>

      <td>
        Workloads (sources) are assigned to Datastreams via labels (odigos.io/data-stream: true), allowing a more flexible selection mechanism.
        Example:
        dataStreams: \["backend", "monitoring"]
        This means the destination will receive data only from sources labeled with "backend" or "monitoring".
      </td>
    </tr>
  </tbody>
</table>

## `SourceSpec`&#x20;

**Appears in:**

* [Source](#odigos-io-v1alpha1-Source)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `workload` \[Required]
      </td>

      <td>
        [`k8sconsts.PodWorkload`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#PodWorkload)
      </td>

      <td>
        Workload represents the workload or namespace to be instrumented.
        This field is required upon creation and cannot be modified.
      </td>
    </tr>

    <tr>
      <td>
        `disableInstrumentation` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        DisableInstrumentation excludes this workload from auto-instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `otelServiceName`
      </td>

      <td>
        `string`
      </td>

      <td>
        OtelServiceName determines the "service.name" resource attribute which will be reported by the instrumentations of this source.
        If not set, the workload name will be used.
        It is not valid for namespace sources.
      </td>
    </tr>

    <tr>
      <td>
        `containerOverrides`
      </td>

      <td>
        [`[]ContainerOverride`](#odigos-io-v1alpha1-ContainerOverride)
      </td>

      <td>
        Specify specific override values for containers in a workload source.
        Not valid for namespace sources.
        Can be used to set the runtime info in case the automatic detection fails or produce wrong results.
        Containers are identified by their names.
        All containers not listed will retain their default behavior.
      </td>
    </tr>

    <tr>
      <td>
        `matchWorkloadNameAsRegex`
      </td>

      <td>
        `bool`
      </td>

      <td>
        MatchWorkloadNameAsRegex indicates that the workload name should be matched using regex.
        When true, spec.workload.name is treated as a regular expression pattern.
        This allows matching multiple workloads with a single Source CRD.
        Not valid for namespace sources.
      </td>
    </tr>
  </tbody>
</table>

## `SourceStatus`&#x20;

**Appears in:**

* [Source](#odigos-io-v1alpha1-Source)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a source's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>


# operator.odigos.io/v1alpha1
Source: https://docs.odigos.io/enterprise/api-reference/operator.odigos.io.v1alpha1



## Resource Types

## `Odigos`&#x20;

Odigos is the Schema for the odigos API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`OdigosSpec`](#OdigosSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`OdigosStatus`](#OdigosStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `OdigosSpec`&#x20;

**Appears in:**

* [Odigos](#Odigos)

OdigosSpec defines the desired state of Odigos

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `onPremToken` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        (Optional) OnPremToken is an enterprise token for Odigos Enterprise.
      </td>
    </tr>

    <tr>
      <td>
        `uiMode` \[Required]
      </td>

      <td>
        [`common.UiMode`](https://pkg.go.dev/github.com/odigos-io/odigos/common#UiMode)
      </td>

      <td>
        (Optional) UIMode sets the UI mode to either "normal" or "readonly".
        In "normal" mode the UI is fully interactive, allowing users to view and edit
        Odigos configuration and settings. In "readonly" mode, the UI can only be
        used to view current Odigos configuration and is not interactive.
        Default=default
      </td>
    </tr>

    <tr>
      <td>
        `telemetryEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) TelemetryEnabled records general telemetry regarding Odigos usage.
        Default=false
      </td>
    </tr>

    <tr>
      <td>
        `ignoredNamespaces` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        (Optional) IgnoredNamespaces is a list of namespaces to not show in the Odigos UI.
      </td>
    </tr>

    <tr>
      <td>
        `ignoredContainers` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        (Optional) IgnoredContainers is a list of container names to exclude from instrumentation (useful for ignoring sidecars).
      </td>
    </tr>

    <tr>
      <td>
        `profiles` \[Required]
      </td>

      <td>
        [`[]common.ProfileName`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ProfileName)
      </td>

      <td>
        (Optional) Profiles is a list of preset profiles with a specific configuration.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnvVarsInjectionMethod` \[Required]
      </td>

      <td>
        [`common.EnvInjectionMethod`](https://pkg.go.dev/github.com/odigos-io/odigos/common#EnvInjectionMethod)
      </td>

      <td>
        (Optional) AgentEnvVarsInjectionMethod is the method to inject the Odigos agent env vars into the pod.
        Default=pod-manifest
      </td>
    </tr>

    <tr>
      <td>
        `nodeSelector` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        (Optional) NodeSelector is a map of key-value Kubernetes NodeSelector labels to apply to all Odigos components.
        Note that Odigos will only be able to instrument applications on the same node.
      </td>
    </tr>

    <tr>
      <td>
        `skipWebhookIssuerCreation` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) SkipWebhookIssuerCreation skips creating the Issuer and Certificate for the Instrumentor pod webhook if cert-manager is installed.
        Default=false
      </td>
    </tr>

    <tr>
      <td>
        `podSecurityPolicy` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) PodSecurityPolicy allows Odigos pods to use a privileged pod security policy.
        Default=false
      </td>
    </tr>

    <tr>
      <td>
        `imagePrefix` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        (Optional) ImagePrefix is a prefix for all container images.
        This should only be used if you are pulling Odigos images from the non-default registry.
        Default (community): docker.io/keyval
        Default (enterprise): registry.odigos.io
        Default (OpenShift): registry.connect.redhat.com/odigos
      </td>
    </tr>

    <tr>
      <td>
        `logLevel` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        (Optional) LogLevel sets the default log level for all Odigos components (e.g. instrumentor, odiglet, collector).
        Accepted values: debug, info, warn, error. Default=info when unset.
      </td>
    </tr>

    <tr>
      <td>
        `mountMethod` \[Required]
      </td>

      <td>
        [`common.MountMethod`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MountMethod)
      </td>

      <td>
        (Optional) MountMethod optionally defines the mechanism for mounting Odigos files into instrumented pods.
        One of "k8s-virtual-device" (default) or "k8s-host-path".
      </td>
    </tr>

    <tr>
      <td>
        `openshiftEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) OpenShiftEnabled configures selinux on OpenShift nodes.
        DEPRECATED: OpenShift clusters are auto-detected and this API field will be removed in a future release.
      </td>
    </tr>
  </tbody>
</table>

## `OdigosStatus`&#x20;

**Appears in:**

* [Odigos](#Odigos)

OdigosStatus defines the observed state of Odigos

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Conditions store the status conditions of the Odigos instances
      </td>
    </tr>
  </tbody>
</table>


# Overview
Source: https://docs.odigos.io/enterprise/backends-overview

Odigos makes it simple to add and configure destinations, allowing you to select the specific signals (`traces`,`metrics`,`logs`) that you want to send to each destination.

> [!TIP]
>
> Can't find your backend in Odigos? Please tell us! We are constantly adding new integrations.\\
>
> You can also follow our quick [add new destination](/adding-new-dest) guide and submit a PR.

| | | | Traces | Metrics | Logs | Profiles |
| - | - | - | :-: | :-: | :-: | :-: |
| ![alibabacloud](https://d15jtxgb40qetw.cloudfront.net/alibabacloud.svg) | [Alibaba Cloud](/backends/alibabacloud) | Managed | ✅ | | | |
| ![appdynamics](https://d15jtxgb40qetw.cloudfront.net/appdynamics.svg) | [AppDynamics](/backends/appdynamics) | Managed | ✅ | ✅ | ✅ | |
| ![cloudwatch](https://d15jtxgb40qetw.cloudfront.net/awscloudwatch.svg) | [AWS CloudWatch](/backends/awscloudwatch) | Managed | | ✅ | ✅ | |
| ![s3](https://d15jtxgb40qetw.cloudfront.net/awss3.svg) | [AWS S3](/backends/awss3) | Managed | ✅ | ✅ | ✅ | |
| ![xray](https://d15jtxgb40qetw.cloudfront.net/awsxray.svg) | [AWS X-Ray](/backends/awsxray) | Managed | ✅ | | | |
| ![axiom](https://d15jtxgb40qetw.cloudfront.net/axiom.svg) | [Axiom](/backends/axiom) | Managed | ✅ | | ✅ | |
| ![azureblob](https://d15jtxgb40qetw.cloudfront.net/blobstorage.svg) | [Azure Blob Storage](/backends/azureblob) | Managed | ✅ | | ✅ | |
| ![azuremonitor](https://d15jtxgb40qetw.cloudfront.net/azuremonitor.svg) | [Azure Monitor](/backends/azuremonitor) | Managed | ✅ | ✅ | ✅ | |
| ![betterstack](https://d15jtxgb40qetw.cloudfront.net/betterstack.svg) | [Better Stack](/backends/betterstack) | Managed | | ✅ | ✅ | |
| ![bonree](https://d15jtxgb40qetw.cloudfront.net/bonree.svg) | [Bonree ONE](/backends/bonree) | Managed | ✅ | ✅ | | |
| ![causely](https://d15jtxgb40qetw.cloudfront.net/causely.svg) | [Causely](/backends/causely) | Managed | ✅ | ✅ | | |
| ![checkly](https://d15jtxgb40qetw.cloudfront.net/checkly.svg) | [Checkly](/backends/checkly) | Managed | ✅ | | | |
| ![chronosphere](https://d15jtxgb40qetw.cloudfront.net/chronosphere.svg) | [Chronosphere](/backends/chronosphere) | Managed | ✅ | ✅ | | |
| ![clickhouse](https://d15jtxgb40qetw.cloudfront.net/clickhouse.svg) | [Clickhouse](/backends/clickhouse) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![coralogix](https://d15jtxgb40qetw.cloudfront.net/coralogix.svg) | [Coralogix](/backends/coralogix) | Managed | ✅ | ✅ | ✅ | |
| ![dash0](https://d15jtxgb40qetw.cloudfront.net/dash0.svg) | [Dash0](/backends/dash0) | Managed | ✅ | ✅ | ✅ | |
| ![datadog](https://d15jtxgb40qetw.cloudfront.net/datadog.svg) | [Datadog](/backends/datadog) | Managed | ✅ | ✅ | ✅ | |
| ![dynamic](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg) | [Dynamic Destination](/backends/dynamic) | Self-Hosted | ✅ | ✅ | ✅ | ✅ |
| ![dynatrace](https://d15jtxgb40qetw.cloudfront.net/dynatrace.svg) | [Dynatrace](/backends/dynatrace) | Managed | ✅ | ✅ | ✅ | |
| ![elasticapm](https://d15jtxgb40qetw.cloudfront.net/elasticapm.svg) | [Elastic APM](/backends/elasticapm) | Managed | ✅ | ✅ | ✅ | |
| ![elasticsearch](https://d15jtxgb40qetw.cloudfront.net/elasticsearch.svg) | [Elasticsearch](/backends/elasticsearch) | Self-Hosted | ✅ | | ✅ | |
| ![qryn](https://d15jtxgb40qetw.cloudfront.net/gigapipe.svg) | [Gigapipe](/backends/gigapipe) | Managed | ✅ | ✅ | ✅ | |
| ![googlecloud](https://d15jtxgb40qetw.cloudfront.net/gcp.svg) | [Google Cloud Monitoring](/backends/googlecloudmonitoring) | Managed | ✅ | | ✅ | |
| ![googlecloudotlp](https://d15jtxgb40qetw.cloudfront.net/gcp.svg) | [Google Cloud (OTLP)](/backends/googlecloudotlp) | Managed | ✅ | | | |
| ![grafanacloudloki](https://d15jtxgb40qetw.cloudfront.net/grafana.svg) | [Grafana Cloud Loki](/backends/grafanacloudloki) | Managed | | | ✅ | |
| ![grafanacloudprometheus](https://d15jtxgb40qetw.cloudfront.net/grafana.svg) | [Grafana Cloud Prometheus](/backends/grafanacloudprometheus) | Managed | | ✅ | | |
| ![grafanacloudtempo](https://d15jtxgb40qetw.cloudfront.net/grafana.svg) | [Grafana Cloud Tempo](/backends/grafanacloudtempo) | Managed | ✅ | | | |
| ![greptime](https://d15jtxgb40qetw.cloudfront.net/greptime.svg) | [GreptimeDB](/backends/greptime) | Managed | | ✅ | | |
| ![groundcover](https://d15jtxgb40qetw.cloudfront.net/groundcover.svg) | [Groundcover inCloud](/backends/groundcover) | Managed | ✅ | ✅ | ✅ | |
| ![honeycomb](https://d15jtxgb40qetw.cloudfront.net/honeycomb.svg) | [Honeycomb](/backends/honeycomb) | Managed | ✅ | ✅ | ✅ | |
| ![hyperdx](https://d15jtxgb40qetw.cloudfront.net/hyperdx.svg) | [HyperDX](/backends/hyperdx) | Managed | ✅ | ✅ | ✅ | |
| ![instana](https://d15jtxgb40qetw.cloudfront.net/instana.svg) | [IBM Instana](/backends/instana) | Managed | ✅ | ✅ | ✅ | |
| ![jaeger](https://d15jtxgb40qetw.cloudfront.net/jaeger.svg) | [Jaeger](/backends/jaeger) | Self-Hosted | ✅ | | | |
| ![kafka](https://d15jtxgb40qetw.cloudfront.net/kafka.svg) | [Kafka](/backends/kafka) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![kloudmate](https://d15jtxgb40qetw.cloudfront.net/kloudmate.svg) | [KloudMate](/backends/kloudmate) | Managed | ✅ | ✅ | ✅ | |
| ![last9](https://d15jtxgb40qetw.cloudfront.net/last9.svg) | [Last9](/backends/last9) | Managed | ✅ | ✅ | ✅ | |
| ![lightstep](https://d15jtxgb40qetw.cloudfront.net/lightstep.svg) | [Lightstep](/backends/lightstep) | Managed | ✅ | | | |
| ![logzio](https://d15jtxgb40qetw.cloudfront.net/logzio.svg) | [Logz.io](/backends/logzio) | Managed | ✅ | ✅ | ✅ | |
| ![loki](https://d15jtxgb40qetw.cloudfront.net/loki.svg) | [Loki](/backends/loki) | Self-Hosted | | | ✅ | |
| ![lumigo](https://d15jtxgb40qetw.cloudfront.net/lumigo.svg) | [Lumigo](/backends/lumigo) | Managed | ✅ | ✅ | ✅ | |
| ![middleware](https://d15jtxgb40qetw.cloudfront.net/middleware.svg) | [Middleware](/backends/middleware) | Managed | ✅ | ✅ | ✅ | |
| ![newrelic](https://d15jtxgb40qetw.cloudfront.net/newrelic.svg) | [New Relic](/backends/newrelic) | Managed | ✅ | ✅ | ✅ | |
| ![observe](https://d15jtxgb40qetw.cloudfront.net/observe.svg) | [Observe](/backends/observe) | Managed | ✅ | ✅ | ✅ | |
| ![oneuptime](https://d15jtxgb40qetw.cloudfront.net/oneuptime.svg) | [OneUptime](/backends/oneuptime) | Managed | ✅ | ✅ | ✅ | |
| ![openobserve](https://d15jtxgb40qetw.cloudfront.net/openobserve.svg) | [OpenObserve](/backends/openobserve) | Managed | ✅ | | ✅ | |
| ![oracle](https://d15jtxgb40qetw.cloudfront.net/oracle.svg) | [Oracle Cloud](/backends/oracle) | Managed | ✅ | ✅ | | |
| ![otlp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg) | [OTLP gRPC](/backends/otlp) | Self-Hosted | ✅ | ✅ | ✅ | ✅ |
| ![otlphttp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg) | [OTLP http](/backends/otlphttp) | Self-Hosted | ✅ | ✅ | ✅ | ✅ |
| ![prometheus](https://d15jtxgb40qetw.cloudfront.net/prometheus.svg) | [Prometheus](/backends/prometheus) | Self-Hosted | | ✅ | | |
| ![pyroscope](https://d15jtxgb40qetw.cloudfront.net/pyroscope.svg) | [Pyroscope](/backends/pyroscope) | Self-Hosted | | | | ✅ |
| ![qryn-oss](https://d15jtxgb40qetw.cloudfront.net/qryn.svg) | [qryn](/backends/qryn) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![quickwit](https://d15jtxgb40qetw.cloudfront.net/quickwit.svg) | [Quickwit](/backends/quickwit) | Self-Hosted | ✅ | | ✅ | |
| ![seq](https://d15jtxgb40qetw.cloudfront.net/seq.svg) | [Seq](/backends/seq) | Self-Hosted | ✅ | | ✅ | |
| ![signalfx](https://d15jtxgb40qetw.cloudfront.net/signalfx.svg) | [SignalFx](/backends/signalfx) | Managed | ✅ | ✅ | | |
| ![signoz](https://d15jtxgb40qetw.cloudfront.net/signoz.svg) | [SigNoz](/backends/signoz) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![splunk](https://d15jtxgb40qetw.cloudfront.net/splunk.svg) | [Splunk (Legacy) (Deprecated)](/backends/splunk) | Managed | ✅ | | | |
| ![splunkotlp](https://d15jtxgb40qetw.cloudfront.net/splunk.svg) | [Splunk (OTLP)](/backends/splunkotlp) | Managed | ✅ | | | |
| ![sumologic](https://d15jtxgb40qetw.cloudfront.net/sumologic.svg) | [Sumo Logic](/backends/sumologic) | Managed | ✅ | ✅ | ✅ | |
| ![telemetryhub](https://d15jtxgb40qetw.cloudfront.net/telemetryhub.svg) | [TelemetryHub](/backends/telemetryhub) | Managed | ✅ | ✅ | ✅ | |
| ![tempo](https://d15jtxgb40qetw.cloudfront.net/tempo.svg) | [Tempo](/backends/tempo) | Self-Hosted | ✅ | | | |
| ![tingyun](https://d15jtxgb40qetw.cloudfront.net/tingyun.svg) | [Tingyun 基调听云](/backends/tingyun) | Managed | ✅ | ✅ | | |
| ![traceloop](https://d15jtxgb40qetw.cloudfront.net/traceloop.svg) | [Traceloop](/backends/traceloop) | Managed | ✅ | ✅ | | |
| ![uptrace](https://d15jtxgb40qetw.cloudfront.net/uptrace.svg) | [Uptrace](/backends/uptrace) | Managed | ✅ | ✅ | ✅ | |
| ![victoriametricscloud](https://d15jtxgb40qetw.cloudfront.net/victoriametrics.svg) | [VictoriaMetrics Cloud](/backends/victoriametricscloud) | Managed | | ✅ | | |


# Alibaba Cloud
Source: https://docs.odigos.io/enterprise/backends/alibabacloud

Configuring the Alibaba Cloud backend (Managed)

### Getting Started

![alibabacloud](https://d15jtxgb40qetw.cloudfront.net/alibabacloud.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **ALIBABA\_ENDPOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is required. Do not add the prefix `http://`.
  * This field is required
* **ALIBABA\_TOKEN** `string` : Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Alibaba Cloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `alibabacloud.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: alibabacloud-example
  namespace: odigos-system
spec:
  data:
    ALIBABA_ENDPOINT: <OTLP gRPC Endpoint>
  destinationName: alibabacloud
  secretRef:
    name: alibabacloud-secret
  signals:
  - TRACES
  type: alibabacloud

---

apiVersion: v1
data:
  ALIBABA_TOKEN: <Base64 Token>
kind: Secret
metadata:
  name: alibabacloud-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f alibabacloud.yaml
```


# AppDynamics
Source: https://docs.odigos.io/enterprise/backends/appdynamics

Configuring the AppDynamics backend (Managed)

### Getting Started

![appdynamics](https://d15jtxgb40qetw.cloudfront.net/appdynamics.svg)

1. [Register](https://accounts.appdynamics.com/trial)/[Login](https://login.appdynamics.com/sso/authenticate) to AppDynamics.
2. Navigate to the OpenTelemetry Configuration page
3. Click on `Processors`, you'll find 3 attributes, locate `appdynamics.controller.account` and copy its value, this is your `Account Name`.
4. Click on `Exporters`, copy the `Endpoint`, and generate an `API Key`.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **APPDYNAMICS\_APPLICATION\_NAME** `string` : Application Name. Will define a namespace in AppDynamics
  * This field is optional
* **APPDYNAMICS\_ACCOUNT\_NAME** `string` : Account Name.
  * This field is required
* **APPDYNAMICS\_ENDPOINT\_URL** `string` : Endpoint URL.
  * This field is required
* **APPDYNAMICS\_API\_KEY** `string` : API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AppDynamics` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `appdynamics.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: appdynamics-example
  namespace: odigos-system
spec:
  data:
    APPDYNAMICS_ACCOUNT_NAME: <Account Name>
    APPDYNAMICS_ENDPOINT_URL: <Endpoint URL>
    # Note: The commented fields below are optional.
    # APPDYNAMICS_APPLICATION_NAME: <Application Name>
  destinationName: appdynamics
  secretRef:
    name: appdynamics-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: appdynamics

---

apiVersion: v1
data:
  APPDYNAMICS_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: appdynamics-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f appdynamics.yaml
```


# AWS CloudWatch
Source: https://docs.odigos.io/enterprise/backends/awscloudwatch

Configuring the AWS CloudWatch backend (Managed)

### Getting Started

![cloudwatch](https://d15jtxgb40qetw.cloudfront.net/awscloudwatch.svg)

> [!WARNING]
>
> Make sure that you have the necessary `AWS Credentials` and permissions set up.\\
>
> AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PushLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:<Region>:<AccountId>:log-group:<LogGroupName>:log-stream:<LogStreamName>"
    },
    {
      "Sid": "CreateLogGroup",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:<Region>:<AccountId>:log-group:<LogGroupName>"
    },
    {
      "Sid": "DescribeLogGroups",
      "Effect": "Allow",
      "Action": "logs:DescribeLogGroups",
      "Resource": "arn:aws:logs:<Region>:<AccountId>:log-group:<LogGroupName>"
    }
  ]
}
```

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **AWS\_CLOUDWATCH\_LOG\_GROUP\_NAME** `string` : Log Group Name. The group name of the CloudWatch Logs. If it does not exist it will be created automatically.
  * This field is required
* **AWS\_CLOUDWATCH\_LOG\_STREAM\_NAME** `string` : Log Stream Name. The stream name of the CloudWatch Logs. If it does not exist it will be created automatically.
  * This field is required
* **AWS\_CLOUDWATCH\_REGION** `string` : Region. The AWS region where the log stream is in. Region must be specified if it is not already set in the default credential chain.
  * This field is optional
* **AWS\_CLOUDWATCH\_ENDPOINT** `string` : Endpoint. The CloudWatch Logs service endpoint which the requests are forwarded to.
  * This field is optional
* **AWS\_CLOUDWATCH\_LOG\_RETENTION** `string` : Log Retention. LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. The value is in days, 0 = Never Expire. This value can also be changed later-on via the AWS console (UI) or AWS API, it's not a permanent decision once set. Also, the value here might not reflect the actual retention value in the destination if it was modified or some automatic policy is applied.
  * This field is optional and defaults to `0`
* **AWS\_CLOUDWATCH\_TAGS** `{ key: string; value: string; }[]` : Tags. Tags is the option to set tags for the CloudWatch Log Group. If specified, please add at most 50 tags. Keys must be between 1-128 characters and follow the pattern: (alphanumerics, whitespace, and \_.:/=+-!). Values must be between 1-256 characters and follow the pattern: (alphanumerics, whitespace, and \_.:/=+-!).
  * This field is optional
* **AWS\_CLOUDWATCH\_RAW\_LOG** `boolean` : Raw Log. If set to true, only the log message will be exported to CloudWatch Logs. This needs to be set to true for EMF logs.
  * This field is optional and defaults to `False`
* **AWS\_CLOUDWATCH\_METRICS\_NAMESPACE** `string` : Metrics Namespace. Customized CloudWatch metrics namespace. Metrics in different namespaces are isolated from each other. Please note this is not a cluster namespace.
  * This field is optional and defaults to `Odigos`
* **AWS\_CLOUDWATCH\_METRICS\_DIMENSION\_ROLLUP** `string` : Metrics Dimension Rollup. The option for metrics dimension rollup.
  * This field is optional and defaults to `NoDimensionRollup`
* **AWS\_CLOUDWATCH\_METRICS\_DETAILED** `boolean` : Detailed Metrics. Retain detailed datapoint values in exported metrics (e.g instead of exporting a quantile as a statistical value, preserve the quantile's population).
  * This field is optional and defaults to `False`
* **AWS\_CLOUDWATCH\_RETAIN\_INITIAL\_VALUE\_OF\_DELTA\_METRIC** `boolean` : Retain Initial Value of Delta Metric. This option specifies how the first value of a metric is handled. AWS EMF expects metric values to only contain deltas to the previous value. In the default case the first received value is therefor not sent to AWS but only used as a baseline for follow up changes to this metric. This is fine for high throughput metrics with stable labels. In this case it does not matter if the first value of this metric is discarded. However when your metric describes infrequent events or events with high label cardinality, then the exporter in default configuration would still drop the first occurrence of this metric. With this configuration value set to true the first value of all metrics will instead be send to AWS.
  * This field is optional and defaults to `False`

> [!NOTE]
>
> * If the log group and/or log stream are specified in an EMF log, that EMF log will be exported to that log group and/or log stream (i.e. ignores the log group and log stream defined in the configuration).
>   * The log group and log stream will also be created automatically if they do not already exist.
>   * Example of an EMF log with log group and log stream:
>   ```json theme={null}
>   {
>     "_aws": {
>       "Timestamp": 1574109732004,
>       "LogGroupName": "Foo",
>       "LogStreamName": "Bar",
>       "CloudWatchMetrics": [
>         {
>           "Namespace": "MyApp",
>           "Dimensions": [
>             [
>               "Operation"
>             ]
>           ],
>           "Metrics": [
>             {
>               "Name": "ProcessingLatency",
>               "Unit": "Milliseconds",
>               "StorageResolution": 60
>             }
>           ]
>         }
>       ]
>     },
>     "Operation": "Aggregator",
>     "ProcessingLatency": 100
>   }
>   ```
> * Resource ARNs (Amazon Resource Name (ARN) of the AWS resource running the collector) are currently not supported with the CloudWatch Logs Exporter.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AWS CloudWatch` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `cloudwatch.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: cloudwatch-example
  namespace: odigos-system
spec:
  data:
    AWS_CLOUDWATCH_LOG_GROUP_NAME: <Log Group Name>
    AWS_CLOUDWATCH_LOG_STREAM_NAME: <Log Stream Name>
    # Note: The commented fields below are optional.
    # AWS_CLOUDWATCH_REGION: <Region (options: [af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ap-southeast-3, ap-southeast-4, ap-southeast-5, ap-southeast-7, ca-central-1, ca-west-1, eu-central-1, eu-central-2, eu-north-1, eu-south-1, eu-south-2, eu-west-1, eu-west-2, eu-west-3, il-central-1, me-central-1, me-south-1, mx-central-1, sa-east-1, us-east-1, us-east-2, us-gov-east-1, us-gov-west-1, us-west-1, us-west-2])>
    # AWS_CLOUDWATCH_ENDPOINT: <Endpoint (options: [logs.af-south-1.amazonaws.com, logs.af-south-1.api.aws, logs.ap-east-1.amazonaws.com, logs.ap-east-1.api.aws, logs.ap-northeast-1.amazonaws.com, logs.ap-northeast-1.api.aws, logs.ap-northeast-2.amazonaws.com, logs.ap-northeast-2.api.aws, logs.ap-northeast-3.amazonaws.com, logs.ap-northeast-3.api.aws, logs.ap-south-1.amazonaws.com, logs.ap-south-1.api.aws, logs.ap-south-2.amazonaws.com, logs.ap-south-2.api.aws, logs.ap-southeast-1.amazonaws.com, logs.ap-southeast-1.api.aws, logs.ap-southeast-2.amazonaws.com, logs.ap-southeast-2.api.aws, logs.ap-southeast-3.amazonaws.com, logs.ap-southeast-3.api.aws, logs.ap-southeast-4.amazonaws.com, logs.ap-southeast-4.api.aws, logs.ap-southeast-5.amazonaws.com, logs.ap-southeast-5.api.aws, logs.ap-southeast-7.amazonaws.com, logs.ap-southeast-7.api.aws, logs.ca-central-1.amazonaws.com, logs.ca-central-1.api.aws, logs-fips.ca-central-1.amazonaws.com, logs.ca-west-1.amazonaws.com, logs.ca-west-1.api.aws, logs-fips.ca-west-1.amazonaws.com, logs.eu-central-1.amazonaws.com, logs.eu-central-1.api.aws, logs.eu-central-2.amazonaws.com, logs.eu-central-2.api.aws, logs.eu-north-1.amazonaws.com, logs.eu-north-1.api.aws, logs.eu-south-1.amazonaws.com, logs.eu-south-1.api.aws, logs.eu-south-2.amazonaws.com, logs.eu-south-2.api.aws, logs.eu-west-1.amazonaws.com, logs.eu-west-1.api.aws, logs.eu-west-2.amazonaws.com, logs.eu-west-2.api.aws, logs.eu-west-3.amazonaws.com, logs.eu-west-3.api.aws, logs.il-central-1.amazonaws.com, logs.il-central-1.api.aws, logs.me-central-1.amazonaws.com, logs.me-central-1.api.aws, logs.me-south-1.amazonaws.com, logs.me-south-1.api.aws, logs.mx-central-1.amazonaws.com, logs.mx-central-1.api.aws, logs.sa-east-1.amazonaws.com, logs.sa-east-1.api.aws, logs.us-east-1.amazonaws.com, logs-fips.us-east-1.amazonaws.com, logs.us-east-1.api.aws, logs.us-east-2.amazonaws.com, logs.us-east-2.api.aws, logs-fips.us-east-2.amazonaws.com, logs.us-gov-east-1.amazonaws.com, logs.us-gov-west-1.amazonaws.com, logs.us-west-1.amazonaws.com, logs.us-west-1.api.aws, logs-fips.us-west-1.amazonaws.com, logs.us-west-2.amazonaws.com, logs-fips.us-west-2.amazonaws.com, logs.us-west-2.api.aws])>
    # AWS_CLOUDWATCH_LOG_RETENTION: <Log Retention (default: 0) (options: [0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, 3653])>
    # AWS_CLOUDWATCH_TAGS: <Tags>
    # AWS_CLOUDWATCH_RAW_LOG: <Raw Log>
    # AWS_CLOUDWATCH_METRICS_NAMESPACE: <Metrics Namespace (default: Odigos)>
    # AWS_CLOUDWATCH_METRICS_DIMENSION_ROLLUP: <Metrics Dimension Rollup (default: NoDimensionRollup) (options: [NoDimensionRollup, SingleDimensionRollupOnly, ZeroAndSingleDimensionRollup])>
    # AWS_CLOUDWATCH_METRICS_DETAILED: <Detailed Metrics>
    # AWS_CLOUDWATCH_RETAIN_INITIAL_VALUE_OF_DELTA_METRIC: <Retain Initial Value of Delta Metric>
  destinationName: cloudwatch
  signals:
  - METRICS
  - LOGS
  type: cloudwatch
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f cloudwatch.yaml
```


# AWS S3
Source: https://docs.odigos.io/enterprise/backends/awss3

Configuring the AWS S3 backend (Managed)

### Getting Started

![s3](https://d15jtxgb40qetw.cloudfront.net/awss3.svg)

**Setting up AWS S3 Bucket**

If you haven't already, you need to create an S3 bucket to store your data. You can do it using the [AWS Management Console](https://console.aws.amazon.com/s3/home) or the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html).

For example, to create a bucket named `otel-data` in region `us-east-1` using the AWS CLI, you can run the following command:

```bash theme={null}
aws s3api create-bucket --bucket otel-data --region us-east-1
```

> [!WARNING]
>
> Make sure that you have the necessary `AWS Credentials` and permissions set up.\\
>
> AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<BucketName>", "arn:aws:s3:::<BucketName>/*"]
    }
  ]
}
```

You can do it with the aws cli.

* Replace `otel-data` with the name of your bucket if you used a different name.
* The policy name is `odigos-s3` in this example, you can use it or choose a different name. Take note of the policy ARN, you will need it in the next step.

```bash theme={null}
aws iam create-policy
--policy-name odigos-s3
--policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::otel-data", "arn:aws:s3:::otel-data/*"] } ] }'
```

Or use the aws management console to create the policy.

The last step is to configure this IAM policy to be assumed by the EKS cluster. Use the following cli command as template and adjust it according to your setup:

* make sure to replace the `--attach-policy-arn` value with the ARN of the policy you created in the previous step.
* use your EKS cluster name and region.

```bash theme={null}
eksctl create iamserviceaccount \
--name default \
--namespace odigos-system \
--cluster otel-cluster \
--region=us-east-1 \
--attach-policy-arn arn:aws:iam::123456789012:policy/odigos-s3 \
--override-existing-serviceaccounts \
--approve
```

**Format**

The aws s3 exporter supports two formats: `otlp_json` and `otlp_proto`.

Each batch of logs / metrics / spans will be written as a single s3 object in the specified bucket.
The content of the object will be the serialized data in the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto),
represented as either JSON or Protocol Buffers.

For example, the following JSON string represents a single span in `otlp_json` format, from Odigos demo application:

```json theme={null}
{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"service.name","value":{"stringValue":"inventory"}},{"key":"odigos.device","value":{"stringValue":"python"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"k8s.namespace.name","value":{"stringValue":"default"}},{"key":"k8s.deployment.name","value":{"stringValue":"inventory"}},{"key":"k8s.pod.name","value":{"stringValue":"inventory-7b87f7566c-2d6fq"}},{"key":"container.name","value":{"stringValue":"inventory"}},{"key":"k8s.node.name","value":{"stringValue":"ip-192-168-4-178.ec2.internal"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9ee5a0de5802d3f1e7f93440f892509c","spanId":"88a4bfb5bfcaf297","parentSpanId":"f9d7bb9e9243df06","name":"/buy","kind":2,"startTimeUnixNano":"1707600957298456744","endTimeUnixNano":"1707600958303042894","attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"8080"}},{"key":"http.host","value":{"stringValue":"inventory:8080"}},{"key":"http.target","value":{"stringValue":"/buy?id=10"}},{"key":"net.peer.ip","value":{"stringValue":"192.168.13.194"}},{"key":"http.user_agent","value":{"stringValue":"Java/17.0.9"}},{"key":"net.peer.port","value":{"intValue":"49420"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/buy"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}
```

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **S3\_BUCKET** `string` : Bucket Name. The name of the bucket where the data will be stored
  * This field is required
* **S3\_REGION** `string` : Bucket Region. The AWS region where the bucket is located
  * This field is required
* **S3\_PARTITION** `string` : Time granularity of S3 Bucket (Deprecated, use Partition Format). Deprecated field for time granularity. Use S3\_PARTITION\_FORMAT for custom formats.
  * This field is optional and defaults to `minute`
* **S3\_PARTITION\_FORMAT** `string` : S3 Partition Format. Custom partition format using strftime syntax (e.g., '%Y/%m/%d/%H/%M' for minute-level, '%Y/%m/%d/%H' for hour-level). If not specified, S3\_PARTITION will be used.
  * This field is optional
  * Example: `%Y/%m/%d/%H/%M`
* **S3\_MARSHALER** `string` : Marshaller (Data Format). The format in which the data will be encoded. It can be either `otlp_json` or `otlp_proto`. Default is `otlp_json`
  * This field is required and defaults to `otlp_json`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AWS S3` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `s3.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: s3-example
  namespace: odigos-system
spec:
  data:
    S3_BUCKET: <Bucket Name>
    S3_MARSHALER: '<Marshaller (Data Format) (default: otlp_json) (options: [otlp_json,
      otlp_proto])>'
    S3_REGION: '<Bucket Region (options: [af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2,
      ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ap-southeast-3,
      ap-southeast-4, ap-southeast-5, ap-southeast-7, ca-central-1, ca-west-1, eu-central-1,
      eu-central-2, eu-north-1, eu-south-1, eu-south-2, eu-west-1, eu-west-2, eu-west-3,
      il-central-1, me-central-1, me-south-1, mx-central-1, sa-east-1, us-east-1,
      us-east-2, us-gov-east-1, us-gov-west-1, us-west-1, us-west-2])>'
    # Note: The commented fields below are optional.
    # S3_PARTITION: <Time granularity of S3 Bucket (Deprecated, use Partition Format) (default: minute) (options: [minute, hour])>
    # S3_PARTITION_FORMAT: <S3 Partition Format>
  destinationName: s3
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: s3
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f s3.yaml
```


# AWS X-Ray
Source: https://docs.odigos.io/enterprise/backends/awsxray

Configuring the AWS X-Ray backend (Managed)

### Getting Started

![xray](https://d15jtxgb40qetw.cloudfront.net/awsxray.svg)

> [!WARNING]
>
> Make sure that you have the necessary `AWS Credentials` and permissions set up.\\
>
> AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "xray:PutTraceSegments",
      "xray:PutTelemetryRecords"
    ],
    "Resource": [
      "*"
    ]
  }
}
```

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **AWS\_XRAY\_REGION** `string` : Region. Send segments to AWS X-Ray service in a specific region.
  * This field is optional
* **AWS\_XRAY\_ENDPOINT** `string` : Endpoint. Optionally override the default X-Ray service endpoint.
  * This field is optional
* **AWS\_XRAY\_PROXY\_ADDRESS** `string` : Proxy Address. Upload segments to AWS X-Ray through a proxy.
  * This field is optional
* **AWS\_XRAY\_RESOURCE\_ARN** `string` : Resource ARN. Amazon Resource Name (ARN) of the AWS resource running the collector.
  * This field is optional
* **AWS\_XRAY\_ROLE\_ARN** `string` : Role ARN. IAM role to upload segments to a different account.
  * This field is optional
* **AWS\_XRAY\_EXTERNAL\_ID** `string` : External ID. Shared identitier used when assuming an IAM role in an external AWS account.
  * This field is optional
* **AWS\_XRAY\_LOG\_GROUPS** `string[]` : Log Groups. List of log group names for CloudWatch.
  * This field is optional
* **AWS\_XRAY\_DISABLE\_SSL** `boolean` : Disable SSL Verification. Enable or disable TLS certificate verification.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_LOCAL\_MODE** `boolean` : Local Mode. Local mode to skip EC2 instance metadata check.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_TELEMETRY\_ENABLED** `boolean` : Telemetry - Enabled. Whether telemetry collection is enabled at all.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_TELEMETRY\_INCLUDE\_METADATA** `boolean` : Telemetry - Include Metadata. Whether to include metadata in the telemetry (InstanceID, Hostname, ResourceARN)
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_TELEMETRY\_HOSTNAME** `string` : Telemetry - Hostname. Sets the Hostname included in the telemetry.
  * This field is optional
* **AWS\_XRAY\_TELEMETRY\_INSTANCE\_ID** `string` : Telemetry - Instance ID. Sets the InstanceID included in the telemetry.
  * This field is optional
* **AWS\_XRAY\_TELEMETRY\_RESOURCE\_ARN** `string` : Telemetry - Resource ARN. Sets the Amazon Resource Name (ARN) included in the telemetry.
  * This field is optional
* **AWS\_XRAY\_TELEMETRY\_CONTRIBUTORS** `string[]` : Telemetry - Contributors. List of X-Ray component IDs contributing to the telemetry (ex. for multiple X-Ray receivers: awsxray/1, awsxray/2)
  * This field is optional
* **AWS\_XRAY\_INDEX\_ALL\_ATTRIBUTES** `boolean` : Index All Attributes. Enable or disable conversion of all OpenTelemetry attributes to X-Ray annotations.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_INDEXED\_ATTRIBUTES** `string[]` : Indexed Attributes. List of attribute names to be converted to X-Ray annotations.
  * This field is optional

> [!NOTE]
>
> **Traces and logs correlation**
>
> AWS X-Ray can be integrated with CloudWatch Logs to correlate traces with logs.
>
> For this integration to work, the X-Ray segments must have the AWS Property `cloudwatch_logs` set. This property is set using the AWS X-Ray exporter with the following values that are evaluated in this order:
>
> 1. `aws.log.group.arns` resource attribute.
> 2. `aws.log.group.names` resource attribute.
> 3. `aws_log_groups` configuration property.
>
> `aws.log.group.arns` and `aws.log.group.names` are slice resource attributes, alternatively those resource attributes can be set using the [OTEL\_RESOURCE\_ATTRIBUTES environment variable](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md#specifying-resource-information-via-an-environment-variable).
>
> To set multiple log group names / log group arns, you can use `&` to separate them.\\
>
> For example 3 log groups: `log-group1`, `log-group2`, and `log-group3` are set in the following command:
>
> ```bash theme={null}
> export OTEL_RESOURCE_ATTRIBUTES="aws.log.group.names=log-group1&log-group2&log-group3"
> ```

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AWS X-Ray` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `xray.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: xray-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: xray
  signals:
  - TRACES
  type: xray
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f xray.yaml
```


# Axiom
Source: https://docs.odigos.io/enterprise/backends/axiom

Configuring the Axiom backend (Managed)

### Getting Started

![axiom](https://d15jtxgb40qetw.cloudfront.net/axiom.svg)

1. [Register](https://app.axiom.co/register)/[Login](https://app.axiom.co/login) to Axiom.
2. Navigate to the [Axiom Datasets](https://app.axiom.co/settings/datasets) page, create a dataset and copy its name.
3. Navigate to the [Axiom API Tokens](https://app.axiom.co/settings/api-tokens) page, and generate a new API Token.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **AXIOM\_DATASET** `string` : Dataset.
  * This field is required
* **AXIOM\_API\_TOKEN** `string` : Axiom API token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Axiom` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `axiom.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: axiom-example
  namespace: odigos-system
spec:
  data:
    AXIOM_DATASET: <Dataset>
  destinationName: axiom
  secretRef:
    name: axiom-secret
  signals:
  - TRACES
  - LOGS
  type: axiom

---

apiVersion: v1
data:
  AXIOM_API_TOKEN: <Base64 Axiom API token>
kind: Secret
metadata:
  name: axiom-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f axiom.yaml
```


# Azure Blob Storage
Source: https://docs.odigos.io/enterprise/backends/azureblob

Configuring the Azure Blob Storage backend (Managed)

### Getting Started

![azureblob](https://d15jtxgb40qetw.cloudfront.net/blobstorage.svg)

Odigos exports data to Azure Blob Storage in [OTLP format](https://opentelemetry.io/docs/specs/otlp/). Data can be exported either in JSON format or in binary format (protobuf).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **AZURE\_BLOB\_ACCOUNT\_NAME** `string` : Account Name.
  * This field is required
* **AZURE\_BLOB\_CONTAINER\_NAME** `string` : Container Name.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Azure Blob Storage` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `azureblob.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: azureblob-example
  namespace: odigos-system
spec:
  data:
    AZURE_BLOB_ACCOUNT_NAME: <Account Name>
    AZURE_BLOB_CONTAINER_NAME: <Container Name>
  destinationName: azureblob
  signals:
  - TRACES
  - LOGS
  type: azureblob
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f azureblob.yaml
```


# Azure Monitor
Source: https://docs.odigos.io/enterprise/backends/azuremonitor

Configuring the Azure Monitor backend (Managed)

### Getting Started

![azuremonitor](https://d15jtxgb40qetw.cloudfront.net/azuremonitor.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **AZURE\_MONITOR\_CONNECTION\_STRING** `string` : Connection String. The Connection String is the recommended method for configuring the exporter, aligning with Azure Monitor's best practices. If you don't have a Connection String, you can create one in the Azure Portal (after creating an Application Insights resource).
  * This field is optional
* **AZURE\_MONITOR\_INSTRUMENTATION\_KEY** `string` : Instrumentation Key. Application Insights instrumentation key, which can be found in the Application Insights resource in the Azure Portal. While it is currently supported, its use is discouraged and it is slated for deprecation.
  * This field is optional
* **AZURE\_MONITOR\_ENDPOINT** `string` : Endpoint. The endpoint URL where data will be submitted. While this option remains available, it is important to note that the use of the Connection String is recommended, as it encompasses the endpoint information. The direct configuration of the endpoint is considered to be on a deprecation path.
  * This field is optional

> [!WARNING]
>
> Do not configure the `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable. It will override this configuration!

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Azure Monitor` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `azuremonitor.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: azuremonitor-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: azuremonitor
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: azuremonitor
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f azuremonitor.yaml
```


# Better Stack
Source: https://docs.odigos.io/enterprise/backends/betterstack

Configuring the Better Stack backend (Managed)

### Getting Started

![betterstack](https://d15jtxgb40qetw.cloudfront.net/betterstack.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **BETTERSTACK\_TOKEN** `string` : Better Stack Source Token.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint for metrics is `https://in-otel.logs.betterstack.com/metrics`
> * The endpoint for logs is `https://in-otel.logs.betterstack.com:443`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Better Stack` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `betterstack.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: betterstack-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: betterstack
  secretRef:
    name: betterstack-secret
  signals:
  - METRICS
  - LOGS
  type: betterstack

---

apiVersion: v1
data:
  BETTERSTACK_TOKEN: <Base64 Better Stack Source Token>
kind: Secret
metadata:
  name: betterstack-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f betterstack.yaml
```


# Bonree ONE
Source: https://docs.odigos.io/enterprise/backends/bonree

Configuring the Bonree ONE backend (Managed)

### Getting Started

![bonree](https://d15jtxgb40qetw.cloudfront.net/bonree.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **BONREE\_ENDPOINT** `string` : OTLP HTTP Endpoint. APM controller address, obtained from the probe deployment page.
  * This field is required
* **BONREE\_ACCOUNT\_ID** `string` : Account ID. Obtained from the probe deployment page.
  * This field is required
* **BONREE\_ENVIRONMENT\_ID** `string` : Environment ID. The default environment does not need to be configured.
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Bonree ONE` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `bonree.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: bonree-example
  namespace: odigos-system
spec:
  data:
    BONREE_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: bonree
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: bonree-secret
  signals:
  - TRACES
  - METRICS
  type: bonree

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   BONREE_ACCOUNT_ID: <Base64 Account ID>
#   BONREE_ENVIRONMENT_ID: <Base64 Environment ID>
# kind: Secret
# metadata:
#   name: bonree-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f bonree.yaml
```


# Causely
Source: https://docs.odigos.io/enterprise/backends/causely

Configuring the Causely backend (Managed)

### Getting Started

![causely](https://d15jtxgb40qetw.cloudfront.net/causely.svg)

This destination is for the Causely Mediator Service, so you will need to have a Causely instance running and accessible from the k8s cluster running Odigos.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **CAUSELY\_URL** `string` : Endpoint. The endpoint URL is the combined `<protocol>://<hostname>:<port>` to access your Causely Mediator service. Protocol should be `http`; using `https` or omitting it will automatically be converted to `http`. Hostname should typically follow the format: `mediator.<namespace>`. Namespace is the k8s namespace where the Causely Mediator service is deployed. Port is optional and defaults to the default OTLP gRPC port `4317`
  * This field is required
  * Example: `http://mediator.causely:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Causely` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `causely.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: causely-example
  namespace: odigos-system
spec:
  data:
    CAUSELY_URL: <Endpoint>
  destinationName: causely
  signals:
  - TRACES
  - METRICS
  type: causely
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f causely.yaml
```


# Checkly
Source: https://docs.odigos.io/enterprise/backends/checkly

Configuring the Checkly backend (Managed)

### Getting Started

![checkly](https://d15jtxgb40qetw.cloudfront.net/checkly.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **CHECKLY\_ENDOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`.
  * This field is required
* **CHECKLY\_API\_KEY** `string` : API Key.
  * This field is required

> [!WARNING]
>
> Odigos will only filter and export traces to Checkly if their `trace_state` includes the tag `checkly="true"`, which is added by a Checkly SDK.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Checkly` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `checkly.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: checkly-example
  namespace: odigos-system
spec:
  data:
    CHECKLY_ENDOINT: <OTLP gRPC Endpoint>
  destinationName: checkly
  secretRef:
    name: checkly-secret
  signals:
  - TRACES
  type: checkly

---

apiVersion: v1
data:
  CHECKLY_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: checkly-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f checkly.yaml
```


# Chronosphere
Source: https://docs.odigos.io/enterprise/backends/chronosphere

Configuring the Chronosphere backend (Managed)

### Getting Started

![chronosphere](https://d15jtxgb40qetw.cloudfront.net/chronosphere.svg)

Chronosphere is a monitoring and observability platform that provides a scalable, reliable, and cost-effective way to collect, store, and analyze metrics and traces.
To send data from Odigos to Chronosphere, [you need to have an account with Chronosphere](https://chronosphere.io/demo-request/) and configure the Chronosphere destination in Odigos with the following details:

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **CHRONOSPHERE\_DOMAIN** `string` : Chronosphere Company Domain. Company domain in Chronosphere, can be found in the URL of the Chronosphere UI. For example, if your URL is `https://demo-sandbox.chronosphere.io/`, then your company domain is `demo-sandbox`
  * This field is required
* **CHRONOSPHERE\_API\_TOKEN** `string` : API Token. API token generated from your Chronosphere service account in the Chronosphere UI, you should login to your Chronosphere admin account click the `Platform` menu on the left side of the screen, then click `Service Accounts` and create a new service account
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Chronosphere` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `chronosphere.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: chronosphere-example
  namespace: odigos-system
spec:
  data:
    CHRONOSPHERE_DOMAIN: <Chronosphere Company Domain>
  destinationName: chronosphere
  secretRef:
    name: chronosphere-secret
  signals:
  - TRACES
  - METRICS
  type: chronosphere

---

apiVersion: v1
data:
  CHRONOSPHERE_API_TOKEN: <Base64 API Token>
kind: Secret
metadata:
  name: chronosphere-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f chronosphere.yaml
```


# ClickHouse
Source: https://docs.odigos.io/enterprise/backends/clickhouse

Configuring the ClickHouse backend (Self-Hosted)

### Getting Started

![clickhouse](https://d15jtxgb40qetw.cloudfront.net/clickhouse.svg)

ClickHouse is very popular database for storing telemetry data and running efficient queries on it.

**Use Cases**

* ClickHouse can handle all 3 signals (metrics, logs, traces) in one single database. less overhead to manage multiple databases.
* ClickHouse is a columnar database, which is optimized for time-series immutable data like OpenTelemetry telemetry data.
* ClickHouse is a distributed database, which can scale horizontally and handle large amounts of data with efficient storage and query performance.
* ClickHouse is open-source and has a large community.

**Prerequisites**

* To use ClickHouse as a destination in Odigos, you need to have a ClickHouse deployment running somewhere and accessible from cluster where Odigos is running.
* You should know the service endpoint where ClickHouse listens for incoming client connections.
* If you haven't already, create a database in ClickHouse where you want to store the telemetry data. the default database name is `otel` (configurable). To create it, run the following SQL command: `CREATE DATABASE otel;`
* Understanding of how to maintain, scale, and optimize your self-hosted ClickHouse deployment, as well as how to fine tune setting based on your queries and use case.

**Schema**

When using the ClickHouse destination with Odigos, logs metrics and traces are going to be "INSERT INTO" the ClickHouse database.

It is important to understand what schema is used, e.g. what table names are used, column names, data types, etc.
Then you can run queries on this data, or modify and optimize it for your specific use case.

There are 2 modes of operation for ClickHouse destination in Odigos:

**Create Schema**

In this option, the schema will be automatically created by odigos with reasonable defaults.

The benefit of this option is that you can see the value fast, without needing to apply and manage any schema yourself.
The downside is that the schema may not be optimized for your specific use case, and may make changes more complicated.

To use it:

* Odigos UI - When adding a new ClickHouse destination, select the `Create Scheme` checkbox field.
* Destination K8s Manifest - Set the `CLICKHOUSE_CREATE_SCHEME` setting to value `true`.

The schema which will be used by default can be found [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/a80fdb94cecb1cd3c1fbd7fbe3000396cd588970/exporter/clickhouseexporter/example/default_ddl).

* Indexes - The default schema includes indexes for the data, including trace\_id, resource and scope attributes, span/metric/log attributes etc.
* TTL is set to 180 days. This means data will be kept in the database for this period of time and then deleted.
* Partitioning - The default schema includes partitioning by day, which means data is stored in separate partitions for each day.
* Order By - Optimized for trace queries on service\_name + span\_name + time, for logs on service\_name + time, and for metrics on service\_name + metric\_name + attributes + time.

This option is not recommended for production workloads:

* You may want to adjust the settings to better fit your use case, scale performance requirements, and costs.
* Each new exporter will attempt to create the schema, which is less robust and harder to manage than a pre-created schema.

**Self Managed Schema**

With this option, you are responsible for creating and managing the schema yourself.

To use it:

* Odigos UI - Unselect the `Create Scheme` checkbox field.
* Destination K8s Manifest - Set the `CLICKHOUSE_CREATE_SCHEME` setting to value `false`.

The benefit of this option is that you have full control over the schema, and can optimize it for your specific use case.

How it works?

* Browse to the [default DDL example](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/a80fdb94cecb1cd3c1fbd7fbe3000396cd588970/exporter/clickhouseexporter/example/default_ddl).
* Copy the `sql` files to your local machine.
* Make any changes you need to the schema.
* Run the SQL files in your ClickHouse database to create the schema.

This option is recommended for production workloads:

* You can optimize the schema for your specific use case, scale performance requirements, and costs.
* You can manage the schema in a version control system, and apply changes in a controlled way.
* Applying changes to the schema is more robust and easier to manage than attempting to create it on the fly with each new connection.

Important Settings:

* Indexes - You may want to add or remove indexes based on your queries, to optimize performance and costs.
* TTL - You may want to adjust the TTL based on your retention policy. If you need traces for auditing purposes, you may extend it to 365 days. For high-throughput, low-latency systems, you might want to reduce it to 90 days or even less.
* Partitioning - partitioning by day is a good default, but in high throughput systems you may want to partition by hour or even minute. You may also consider partitioning by service\_name, or other attributes.
* Order By - You may want to adjust the order by clause based on your queries, so that common columns are used first in the query, to optimize performance.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **CLICKHOUSE\_ENDPOINT** `string` : Endpoint. The ClickHouse endpoint is the URL where the ClickHouse server is listening for incoming connections.
  * This field is required
  * Example: `http://host:port`
* **CLICKHOUSE\_USERNAME** `string` : Username. If Clickhouse Authentication is used, provide the username
  * This field is optional
* **CLICKHOUSE\_PASSWORD** `string` : Password. If Clickhouse Authentication is used, provide the password
  * This field is optional
* **CLICKHOUSE\_CREATE\_SCHEME** `boolean` : Create Schema. Should the destination create the schema for you? Set to `false` if you manage your own schema, or `true` to have Odigos create the schema for you
  * This field is required and defaults to `True`
* **CLICKHOUSE\_DATABASE\_NAME** `string` : Database Name. The name of the Clickhouse Database where the telemetry data will be stored. The Database will not be created when not exists, so make sure you have created it before
  * This field is required and defaults to `otel`
* **CLICKHOUSE\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **CLICKHOUSE\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`
* **CLICKHOUSE\_USE\_CUSTOM\_CA** `boolean` : Use Custom CA Certificate. Use a custom CA certificate instead of the system root CA
  * This field is optional and defaults to `False`
* **CLICKHOUSE\_CA\_PEM** `string` : Certificate Authority (CA PEM). CA certificate in PEM format to verify the server
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **CLICKHOUSE\_TRACES\_TABLE** `string` : Traces Table. Name of the ClickHouse Table to use for storing trace spans. This name should be used in span queries
  * This field is required and defaults to `otel_traces`
* **CLICKHOUSE\_LOGS\_TABLE** `string` : Logs Table. Name of the ClickHouse Table to use for storing logs
  * This field is required and defaults to `otel_logs`
* **CLICKHOUSE\_METRICS\_TABLE\_EXP\_HISTOGRAM** `string` : Metrics Table - Exp. Histogram. Name of the table for storing exponential histogram metrics
  * This field is required and defaults to `otel_metrics_exponential_histogram`
* **CLICKHOUSE\_METRICS\_TABLE\_GAUGE** `string` : Metrics Table - Gauge. Name of the table for storing gauge metrics
  * This field is required and defaults to `otel_metrics_gauge`
* **CLICKHOUSE\_METRICS\_TABLE\_HISTOGRAM** `string` : Metrics Table - Histogram. Name of the table for storing histogram metrics
  * This field is required and defaults to `otel_metrics_histogram`
* **CLICKHOUSE\_METRICS\_TABLE\_SUM** `string` : Metrics Table - Sum. Name of the table for storing sum metrics
  * This field is required and defaults to `otel_metrics_sum`
* **CLICKHOUSE\_METRICS\_TABLE\_SUMMARY** `string` : Metrics Table - Summary. Name of the table for storing summary metrics
  * This field is required and defaults to `otel_metrics_summary`
* **HYPERDX\_LOG\_NORMALIZER** `boolean` : Enable Log Normalizer. Enable HyperDX log normalization processor. This parses JSON from log bodies, infers severity levels, and normalizes log attributes for better querying.
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Clickhouse` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `clickhouse.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: clickhouse-example
  namespace: odigos-system
spec:
  data:
    CLICKHOUSE_CREATE_SCHEME: '<Create Schema (default: True)>'
    CLICKHOUSE_DATABASE_NAME: '<Database Name (default: otel)>'
    CLICKHOUSE_ENDPOINT: <Endpoint>
    CLICKHOUSE_LOGS_TABLE: '<Logs Table (default: otel_logs)>'
    CLICKHOUSE_METRICS_TABLE_EXP_HISTOGRAM: '<Metrics Table - Exp. Histogram (default:
      otel_metrics_exponential_histogram)>'
    CLICKHOUSE_METRICS_TABLE_GAUGE: '<Metrics Table - Gauge (default: otel_metrics_gauge)>'
    CLICKHOUSE_METRICS_TABLE_HISTOGRAM: '<Metrics Table - Histogram (default: otel_metrics_histogram)>'
    CLICKHOUSE_METRICS_TABLE_SUM: '<Metrics Table - Sum (default: otel_metrics_sum)>'
    CLICKHOUSE_METRICS_TABLE_SUMMARY: '<Metrics Table - Summary (default: otel_metrics_summary)>'
    CLICKHOUSE_TRACES_TABLE: '<Traces Table (default: otel_traces)>'
    # Note: The commented fields below are optional.
    # CLICKHOUSE_USERNAME: <Username>
    # CLICKHOUSE_TLS_ENABLED: <Enable TLS>
    # CLICKHOUSE_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
    # CLICKHOUSE_USE_CUSTOM_CA: <Use Custom CA Certificate>
    # HYPERDX_LOG_NORMALIZER: <Enable Log Normalizer>
  destinationName: clickhouse
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: clickhouse-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: clickhouse

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   CLICKHOUSE_CA_PEM: <Base64 Certificate Authority (CA PEM)>
#   CLICKHOUSE_PASSWORD: <Base64 Password>
# kind: Secret
# metadata:
#   name: clickhouse-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f clickhouse.yaml
```


# Coralogix
Source: https://docs.odigos.io/enterprise/backends/coralogix

Configuring the Coralogix backend (Managed)

### Getting Started

![coralogix](https://d15jtxgb40qetw.cloudfront.net/coralogix.svg)

1. Get an [API Key](https://coralogix.com/docs/send-your-data-api-key/).
2. Obtain the [Domain](https://coralogix.com/docs/coralogix-domain/) determined by your Coralogix Account URL.
3. Follow the [Application and Subsystem Names guide](https://coralogix.com/docs/application-and-subsystem-names/) to learn how configure the application and subsystem names for your Coralogix account.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **CORALOGIX\_PRIVATE\_KEY** `string` : Send-Your-Data API Key. The Send-Your-Data API Key is used to authenticate the data sent to Coralogix.
  * This field is required
* **CORALOGIX\_DOMAIN** `string` : Domain. The Coralogix domain to which you want to send the data.
  * This field is required
* **CORALOGIX\_APPLICATION\_NAME** `string` : Application Name. The name of the application that sends the data to Coralogix.
  * This field is required
* **CORALOGIX\_SUBSYSTEM\_NAME** `string` : Subsystem Name. The name of the subsystem that sends the data to Coralogix.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Coralogix` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `coralogix.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: coralogix-example
  namespace: odigos-system
spec:
  data:
    CORALOGIX_APPLICATION_NAME: <Application Name>
    CORALOGIX_DOMAIN: '<Domain (options: [coralogix.com, eu2.coralogix.com, coralogix.us,
      cx498.coralogix.com, coralogix.in, coralogixsg.com])>'
    CORALOGIX_SUBSYSTEM_NAME: <Subsystem Name>
  destinationName: coralogix
  secretRef:
    name: coralogix-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: coralogix

---

apiVersion: v1
data:
  CORALOGIX_PRIVATE_KEY: <Base64 Send-Your-Data API Key>
kind: Secret
metadata:
  name: coralogix-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f coralogix.yaml
```


# Dash0
Source: https://docs.odigos.io/enterprise/backends/dash0

Configuring the Dash0 backend (Managed)

### Getting Started

![dash0](https://d15jtxgb40qetw.cloudfront.net/dash0.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **DASH0\_ENDPOINT** `string` : Dash0 OTLP gRPC Endpoint. The format is `host:port`. Host is required, located in Dash0 UI (OpenTelemetry Collector). Port is optional and defaults to the default OTLP gRPC port `4317`.
  * This field is required
* **DASH0\_TOKEN** `string` : Dash0 Bearer Token. Located in Dash0 UI (OpenTelemetry Collector).
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Dash0` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `dash0.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: dash0-example
  namespace: odigos-system
spec:
  data:
    DASH0_ENDPOINT: <Dash0 OTLP gRPC Endpoint>
  destinationName: dash0
  secretRef:
    name: dash0-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: dash0

---

apiVersion: v1
data:
  DASH0_TOKEN: <Base64 Dash0 Bearer Token>
kind: Secret
metadata:
  name: dash0-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f dash0.yaml
```


# Datadog
Source: https://docs.odigos.io/enterprise/backends/datadog

Configuring the Datadog backend (Managed)

### Getting Started

![datadog](https://d15jtxgb40qetw.cloudfront.net/datadog.svg)

On the left sidebar, click on your username and then click on **Organization Settings**

![Organization settings menu](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/datadog1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1fed9ecd072cd3bc44a331e79fef38e0)

Click on **API Keys** and then click on **New Key** and give it a name.

![New API key screen](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/datadog2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=f8d85799fc464f727f77fb89ad458638)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **DATADOG\_API\_KEY** `string` : API Key. The API key for your Datadog account.
  * This field is required
* **DATADOG\_SITE** `string` : Site. The datadog site you are using (the url of the datadog site you are using).
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Datadog` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `datadog.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: datadog-example
  namespace: odigos-system
spec:
  data:
    DATADOG_SITE: '<Site (options: [us3.datadoghq.com, datadoghq.com, us5.datadoghq.com,
      datadoghq.eu, ddog-gov.com, ap1.datadoghq.com])>'
  destinationName: datadog
  secretRef:
    name: datadog-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: datadog

---

apiVersion: v1
data:
  DATADOG_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: datadog-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f datadog.yaml
```


# Dynamic Destination
Source: https://docs.odigos.io/enterprise/backends/dynamic

Configuring the Dynamic Destination backend (Self-Hosted)

### Getting Started

![dynamic](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg)

Use Dynamic Destination when you need a collector exporter configuration that is not covered by a built-in destination type (for example, OTLP with client mTLS via freeform YAML).

> [!NOTE]
>
> Environment variable placeholders such as `${OTLP_CA_PEM}` in `DYNAMIC_CONFIGURATION_DATA` are expanded by the gateway collector from pod environment variables. To inject those values, create a Secret in the **same namespace** as the Destination (the Odigos system namespace) whose keys match the placeholder names, and set `spec.secretRef.name` on the Destination. Without `secretRef`, the instrumentor will not mount the Secret into the gateway and the placeholders remain unset. Use unique env var names across dynamic destinations — they are mounted into the shared gateway without a destination prefix (built-in destinations are automatically scoped).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ✅ Profiles

* **DYNAMIC\_DESTINATION\_TYPE** `string` : Destination Type. The type of OpenTelemetry Collector Exporter to use for the destination.
  * This field is required
  * Example: `otlp`
* **DYNAMIC\_CONFIGURATION\_DATA** `string` : Config. The raw YAML config for the Collector Exporter for this destination. To inject secrets (e.g. TLS PEMs), use \$ placeholders and set Destination.spec.secretRef to a Secret in the same namespace whose keys match those variable names. Use unique env var names across dynamic destinations — they are mounted into the shared gateway without a destination prefix.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Dynamic Destination` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `dynamic.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: dynamic-example
  namespace: odigos-system
spec:
  data:
    DYNAMIC_CONFIGURATION_DATA: <Config>
    DYNAMIC_DESTINATION_TYPE: <Destination Type>
  destinationName: dynamic
  signals:
  - TRACES
  - METRICS
  - LOGS
  - PROFILES
  type: dynamic
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f dynamic.yaml
```


# Dynatrace
Source: https://docs.odigos.io/enterprise/backends/dynatrace

Configuring the Dynatrace backend (Managed)

### Getting Started

![dynatrace](https://d15jtxgb40qetw.cloudfront.net/dynatrace.svg)

On the left sidebar click on `Apps`, then under `Manage` select `Access Tokens`:

![Obtaining access token](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/dynatrace1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=9a53c177bf76f5c7bb929c0c0dfdc009)

Make sure to add the required capabilities for the access tokens (e.g `Ingest OpenTelemetry traces` for traces).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **DYNATRACE\_URL** `string` : Tenant URL.
  * This field is required
  * Example: `https://{your-environment-id}.live.dynatrace.com`
* **DYNATRACE\_API\_TOKEN** `string` : API Access Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Dynatrace` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `dynatrace.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: dynatrace-example
  namespace: odigos-system
spec:
  data:
    DYNATRACE_URL: <Tenant URL>
  destinationName: dynatrace
  secretRef:
    name: dynatrace-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: dynatrace

---

apiVersion: v1
data:
  DYNATRACE_API_TOKEN: <Base64 API Access Token>
kind: Secret
metadata:
  name: dynatrace-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f dynatrace.yaml
```


# Elastic APM
Source: https://docs.odigos.io/enterprise/backends/elasticapm

Configuring the Elastic APM backend (Managed)

### Getting Started

![elasticapm](https://d15jtxgb40qetw.cloudfront.net/elasticapm.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **ELASTIC\_APM\_SERVER\_ENDPOINT** `string` : elastic APM server endpoint.
  * This field is required
* **ELASTIC\_APM\_SECRET\_TOKEN** `string` : elastic APM server token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Elastic APM` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `elasticapm.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: elasticapm-example
  namespace: odigos-system
spec:
  data:
    ELASTIC_APM_SERVER_ENDPOINT: <elastic APM server endpoint>
  destinationName: elasticapm
  secretRef:
    name: elasticapm-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: elasticapm

---

apiVersion: v1
data:
  ELASTIC_APM_SECRET_TOKEN: <Base64 elastic APM server token>
kind: Secret
metadata:
  name: elasticapm-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f elasticapm.yaml
```


# Elasticsearch
Source: https://docs.odigos.io/enterprise/backends/elasticsearch

Configuring the Elasticsearch backend (Self-Hosted)

### Getting Started

![elasticsearch](https://d15jtxgb40qetw.cloudfront.net/elasticsearch.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **ELASTICSEARCH\_URL** `string` : Elasticsearch URL. Elasticsearch endpoint. Port defaults to `9200` if not specified
  * This field is required
  * Example: `http://host:port`
* **ES\_TRACES\_INDEX** `string` : Traces Index. The name of the index where traces will be stored
  * This field is optional and defaults to `trace_index`
* **ES\_LOGS\_INDEX** `string` : Logs Index. The name of the index where logs will be stored.
  * This field is optional and defaults to `log_index`
* **ELASTICSEARCH\_BASIC\_AUTH\_ENABLED** `boolean` : Enable HTTP Basic Authentication.
  * This field is optional and defaults to `False`
* **ELASTICSEARCH\_USERNAME** `string` : Username. Username used for HTTP Basic Authentication
  * This field is optional
* **ELASTICSEARCH\_PASSWORD** `string` : Password. Password used for HTTP Basic Authentication
  * This field is optional
* **ELASTICSEARCH\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection (Transport Layer Security)
  * This field is optional and defaults to `False`
* **ELASTICSEARCH\_CA\_PEM** `string` : CA Certificate. When using TLS, provide the CA certificate to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Elasticsearch` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `elasticsearch.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: elasticsearch-example
  namespace: odigos-system
spec:
  data:
    ELASTICSEARCH_URL: <Elasticsearch URL>
    # Note: The commented fields below are optional.
    # ES_TRACES_INDEX: <Traces Index (default: trace_index)>
    # ES_LOGS_INDEX: <Logs Index (default: log_index)>
    # ELASTICSEARCH_BASIC_AUTH_ENABLED: <Enable HTTP Basic Authentication>
    # ELASTICSEARCH_USERNAME: <Username>
    # ELASTICSEARCH_TLS_ENABLED: <Enable TLS>
    # ELASTICSEARCH_CA_PEM: <CA Certificate>
  destinationName: elasticsearch
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: elasticsearch-secret
  signals:
  - TRACES
  - LOGS
  type: elasticsearch

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   ELASTICSEARCH_PASSWORD: <Base64 Password>
# kind: Secret
# metadata:
#   name: elasticsearch-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f elasticsearch.yaml
```


# Gigapipe
Source: https://docs.odigos.io/enterprise/backends/gigapipe

Configuring the Gigapipe backend (Managed)

### Getting Started

![qryn](https://d15jtxgb40qetw.cloudfront.net/gigapipe.svg)

Access your Gigapipe account and generate a **Scoped Token** for Odigos

![Gigapipe Token](https://i.imgur.com/zzmufhk.png)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **QRYN\_API\_SECRET** `string` : API Secret. The API Secret for your Gigapipe Account
  * This field is required
* **QRYN\_API\_KEY** `string` : API Key. The API Token for your Gigapipe Account
  * This field is required
* **QRYN\_URL** `string` : API Url. The API Endpoint for your Gigapipe Account
  * This field is required
  * Example: `https://your-account.gigapipe.com`
* **QRYN\_RESOURCE\_TO\_TELEMETRY\_CONVERSION** `boolean` : Convert container attributes to labels.
  * This field is optional and defaults to `True`
* **QRYN\_ADD\_EXPORTER\_NAME** `boolean` : Add exporter name to labels.
  * This field is optional and defaults to `True`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Gigapipe` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `qryn.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: qryn-example
  namespace: odigos-system
spec:
  data:
    QRYN_API_KEY: <API Key>
    QRYN_URL: <API Url>
    # Note: The commented fields below are optional.
    # QRYN_RESOURCE_TO_TELEMETRY_CONVERSION: <Convert container attributes to labels (default: True)>
    # QRYN_ADD_EXPORTER_NAME: <Add exporter name to labels (default: True)>
  destinationName: qryn
  secretRef:
    name: qryn-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: qryn

---

apiVersion: v1
data:
  QRYN_API_SECRET: <Base64 API Secret>
kind: Secret
metadata:
  name: qryn-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f qryn.yaml
```


# Google Cloud Monitoring
Source: https://docs.odigos.io/enterprise/backends/googlecloudmonitoring

Configuring the Google Cloud Monitoring backend (Managed)

### Getting Started

![googlecloud](https://d15jtxgb40qetw.cloudfront.net/gcp.svg)

### Google Cloud Trace API

This destination uses the [Google Cloud Trace API](https://cloud.google.com/trace/docs/reference).
It is different from the [Google CLoud Telemetry (OTLP) API destination](https://docs.odigos.io/backends/googlecloudotlp).

On GCP, Odigos currently supports Standard (non-Autopilot) GKE clusters.

### Authenticating with Google Cloud

When running the Google Cloud Exporter on GKE, by default the OpenTelemetry Collector will attempt to use
[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
to authenticate to the Google Cloud Observability backend. This means that for most use cases running on GCP,
there is not authentication or service account token required as your workloads are automatically authenticated.

However, to authenticate with Cloud Observability when running off-GKE (for example, on a local machine for another
cloud provider) you must provide a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console)
for the OpenTelemetry Collector Exporter. In Odigos, this can be provided in the Application Credentials field of the
Google Cloud destination.

When the Application Credentials field is set in the UI, the Odigos UI will save the contents of that field as a Kubernetes Secret
which is mounted as a volume into the Collector Pod. The location of that volume is then set as the `GOOGLE_APPLICATION_CREDENTIALS`
environment variable, instructing the Google Cloud Exporter where to look for its credentials.

When creating the destination as a YAML manifest off-GKE, the SecretRef must point to a Secret containing the Service Account token.
The token must be set as the `GCP_APPLICATION_CREDENTIALS` key in the referenced Secret.

The `GCP_APPLICATION_CREDENTIALS` field does not need to be set on the YAML manifest destination CRD, only in the referenced Secret.
This field is used internally by Odigos when creating GCP destinations from the UI.

Currently, only one Google Cloud destination can be configured with Application Credentials.

See the [Google Cloud OpenTelemetry Collector Exporter docs](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/tree/1ea43a4534606df1d91ee1754e6c5bc31ee8664d/exporter/collector#prerequisite-authenticating)
for more information on authentication.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **GCP\_TIMEOUT** `string` : Timeout. The request timeout in seconds for the Google Cloud Monitoring API. Defaults to 10 seconds. Increasing this value can help with deadline exceeded errors.
  * This field is optional
  * Example: `10s`
* **GCP\_PROJECT\_ID** `string` : Project ID. The project ID for the Google Cloud Monitoring API to send data to. Required if running Odigos outside of Google Cloud. Defaults to the detected project ID when running on GCP.
  * This field is optional
  * Example: `project-id`
* **GCP\_APPLICATION\_CREDENTIALS** `string` : Application Credentials. The JSON GCP Application Credentials file for the GCP project. Either a Workload Identity Federation config file or Service Account Key. Required if running Odigos outside of Google Cloud.
  * This field is optional
  * Example: `Credentials config... (optional)`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Google Cloud Monitoring` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `googlecloud.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: googlecloud-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: googlecloud
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: googlecloud-secret
  signals:
  - TRACES
  - LOGS
  type: googlecloud

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   GCP_APPLICATION_CREDENTIALS: <Base64 Application Credentials>
# kind: Secret
# metadata:
#   name: googlecloud-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f googlecloud.yaml
```


# Google Cloud (OTLP)
Source: https://docs.odigos.io/enterprise/backends/googlecloudotlp

Configuring the Google Cloud (OTLP) backend (Managed)

### Getting Started

![googlecloudotlp](https://d15jtxgb40qetw.cloudfront.net/gcp.svg)

### Google Cloud Telemetry API

This destination uses the [Google Cloud Telemetry (OTLP) API](https://cloud.google.com/stackdriver/docs/reference/telemetry/overview)
to send data as OTLP to GCP Observability using an OTLP/HTTP exporter. It is different from the
[Google Cloud Monitoring destination](https://docs.odigos.io/backends/googlecloudmonitoring)
which uses the [googlecloud exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/d9769f7c1a0f3bea82c711af551d9b5b44d69e6f/exporter/googlecloudexporter).

Ong GCP, Odigos currently supports Standard (non-Autopilot) GKE clusters.

### Authenticating with Google Cloud

When running the Google Cloud Exporter on GKE, by default the OpenTelemetry Collector will attempt to use
[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
to authenticate to the Google Cloud Observability backend. This means that for most use cases running on GCP,
there is not authentication or service account token required as your workloads are automatically authenticated.

However, to authenticate with Cloud Observability when running off-GKE (for example, on a local machine for another
cloud provider) you must provide a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console)
for the OpenTelemetry Collector Exporter. In Odigos, this can be provided in the Application Credentials field of the
Google Cloud destination.

When the Application Credentials field is set in the UI, the Odigos UI will save the contents of that field as a Kubernetes Secret
which is mounted as a volume into the Collector Pod. The location of that volume is then set as the `GOOGLE_APPLICATION_CREDENTIALS`
environment variable, instructing the Google Cloud Exporter where to look for its credentials.

When creating the destination as a YAML manifest off-GKE, the SecretRef must point to a Secret containing the Service Account token.
The token must be set as the `GCP_APPLICATION_CREDENTIALS` key in the referenced Secret.

The `GCP_APPLICATION_CREDENTIALS` field does not need to be set on the YAML manifest destination CRD, only in the referenced Secret.
This field is used internally by Odigos when creating GCP destinations from the UI.

Currently, only one Google Cloud destination can be configured with Application Credentials.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **GCP\_PROJECT\_ID** `string` : Project ID. The project ID for the Google Cloud Monitoring API to send data to. This defaults to the Application Credentials project ID (which is automatically detected when running on GCP). Setting this can override the Application Credentials project ID. Required if running Odigos outside of Google Cloud.
  * This field is optional
  * Example: `project-id`
* **GCP\_APPLICATION\_CREDENTIALS** `string` : Application Credentials. The JSON GCP Application Credentials file for the GCP project. Either a Workload Identity Federation config file or Service Account Key. Required if running Odigos outside of Google Cloud.
  * This field is optional
  * Example: `Credentials config... (optional)`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Google Cloud (OTLP)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `googlecloudotlp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: googlecloudotlp-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: googlecloudotlp
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: googlecloudotlp-secret
  signals:
  - TRACES
  type: googlecloudotlp

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   GCP_APPLICATION_CREDENTIALS: <Base64 Application Credentials>
# kind: Secret
# metadata:
#   name: googlecloudotlp-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f googlecloudotlp.yaml
```


# Google Cloud Storage
Source: https://docs.odigos.io/enterprise/backends/googlecloudstorage

Configuring Google Cloud Storage

Odigos exports data to Google Cloud Storage in [OTLP format](https://opentelemetry.io/docs/specs/otlp/).
Data can be exported either in JSON format or in binary format (protobuf).

Authentication to Google Cloud Storage is done using [Google Application Default Credentials](https://cloud.google.com/docs/authentication/production).

## Configuring Google Cloud Storage Backend

The only required field is the **bucket name**. The bucket must exists and the odigos gateway pod must have write access to it.


# Grafana Cloud Loki
Source: https://docs.odigos.io/enterprise/backends/grafanacloudloki

Configuring the Grafana Cloud Loki backend (Managed)

### Getting Started

![grafanacloudloki](https://d15jtxgb40qetw.cloudfront.net/grafana.svg)

> [!NOTE]
>
> These instructions are for the Grafana Cloud **managed** Loki service. If you run a self hosted Loki instance, please follow the instructions on the [Loki](../backends/loki) page.

You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Loki as a **logs** backend.

All the connection details can be found at [grafana.com](https://grafana.com) under **My Account**

![Grafana labs website](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1e83df72b7b02521187f810d913bbd5f)

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Loki" card and click "Send Logs"

![Loki managed service configuration](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1c379a219e7e4e9ad698f8fb2f70d815)

**Loki Labels**

Read about setting grafana loki labels in the [Loki](../backends/loki#loki-labels) page.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **GRAFANA\_CLOUD\_LOKI\_ENDPOINT** `string` : Endpoint. The endpoint of your Grafana Cloud Loki instance
  * This field is required
  * Example: `https://logs-prod-{REGION/SHARD}.grafana.net/otlp`
* **GRAFANA\_CLOUD\_LOKI\_USERNAME** `string` : Username. You can find the loki username in the "Grafana Data Source settings" section as "User" value. The username is a number.
  * This field is required
  * Example: `1234567`
* **GRAFANA\_CLOUD\_LOKI\_PASSWORD** `string` : Password (Api Token). This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `logs:write` scope
  * This field is required
  * Example: `glc_eyJvIj...`
* **GRAFANA\_CLOUD\_LOKI\_LABELS** `string[]` : Labels. Use these OpenTelemetry resource attributes as Loki labels for each log record
  * This field is optional and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]`

> [!NOTE]
>
> The supported Loki version is V3. Please make sure you are using the correct version, and that you have enabled native OTLP support. As part of this version, the path `/loki/api/v1/push` has been deprecated and replaced with `/otlp`.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Grafana Cloud Loki` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `grafanacloudloki.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: grafanacloudloki-example
  namespace: odigos-system
spec:
  data:
    GRAFANA_CLOUD_LOKI_ENDPOINT: <Endpoint>
    GRAFANA_CLOUD_LOKI_USERNAME: <Username>
    # Note: The commented fields below are optional.
    # GRAFANA_CLOUD_LOKI_LABELS: <Labels (default: ["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"])>
  destinationName: grafanacloudloki
  secretRef:
    name: grafanacloudloki-secret
  signals:
  - LOGS
  type: grafanacloudloki

---

apiVersion: v1
data:
  GRAFANA_CLOUD_LOKI_PASSWORD: <Base64 Password (Api Token)>
kind: Secret
metadata:
  name: grafanacloudloki-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f grafanacloudloki.yaml
```


# Grafana Cloud Prometheus
Source: https://docs.odigos.io/enterprise/backends/grafanacloudprometheus

Configuring the Grafana Cloud Prometheus backend (Managed)

### Getting Started

![grafanacloudprometheus](https://d15jtxgb40qetw.cloudfront.net/grafana.svg)

> [!NOTE]
>
> These instructions are for the Grafana Cloud **managed** Prometheus service. If you run a self hosted Prometheus instance, please follow the instructions on the [Prometheus](../backends/prometheus) page.

You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Prometheus as a **metrics** backend.

All the connection details can be found at [grafana.com](https://grafana.com) under **My Account**

![Grafana labs website](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1e83df72b7b02521187f810d913bbd5f)

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Prometheus" card and click "Send Metrics"

![Prometheus managed service configuration](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud3.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=74e9827c06a656b0895412c1b4d05e00)

**Resource Attributes as Metrics Labels**

To use [resource attributes as labels in your data points](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter#setting-resource-attributes-as-metric-labels), you can set the resource attribute names in the `Resource Attributes as Labels` multi-input config option in the destination page in the Odigos UI.

**Cluster Static Labels**

You can set static labels for each metric data pointsent exported to Grafana Cloud Prometheus from the cluster by setting the `External Labels` configuration with lable keys and values.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **GRAFANA\_CLOUD\_PROMETHEUS\_RW\_ENDPOINT** `string` : Prometheus Remote Write Endpoint. This is the URL of the Prometheus service. From the grafana cloud UI, Prometheus page, make sure you copy the URL from “Remote Write Endpoint” section, and not the first Endpoint which is used for queries
  * This field is required
  * Example: `https://{hostname}.grafana.net/api/prom/push`
* **GRAFANA\_CLOUD\_PROMETHEUS\_USERNAME** `string` : Username / Instance ID. You can find the Username / Instance ID on the Prometheus page. Value is a number.
  * This field is required
  * Example: `12345678`
* **GRAFANA\_CLOUD\_PROMETHEUS\_PASSWORD** `string` : Password / Api Token. This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `metrics:write` scope.
  * This field is required
  * Example: `glc_eyJvIj...`
* **PROMETHEUS\_RESOURCE\_ATTRIBUTES\_LABELS** `string[]` : Resource Attributes Labels. use these OpenTelemetry resource attributes as prometheus labels for each metric data point
  * This field is optional and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]`
* **PROMETHEUS\_RESOURCE\_EXTERNAL\_LABELS** `{ key: string; value: string; }[]` : External Labels. map of labels names and values to be attached to each metric data point
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Grafana Cloud Prometheus` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `grafanacloudprometheus.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: grafanacloudprometheus-example
  namespace: odigos-system
spec:
  data:
    GRAFANA_CLOUD_PROMETHEUS_RW_ENDPOINT: <Prometheus Remote Write Endpoint>
    GRAFANA_CLOUD_PROMETHEUS_USERNAME: <Username / Instance ID>
    # Note: The commented fields below are optional.
    # PROMETHEUS_RESOURCE_ATTRIBUTES_LABELS: <Resource Attributes Labels (default: ["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"])>
    # PROMETHEUS_RESOURCE_EXTERNAL_LABELS: <External Labels>
  destinationName: grafanacloudprometheus
  secretRef:
    name: grafanacloudprometheus-secret
  signals:
  - METRICS
  type: grafanacloudprometheus

---

apiVersion: v1
data:
  GRAFANA_CLOUD_PROMETHEUS_PASSWORD: <Base64 Password / Api Token>
kind: Secret
metadata:
  name: grafanacloudprometheus-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f grafanacloudprometheus.yaml
```


# Grafana Cloud Tempo
Source: https://docs.odigos.io/enterprise/backends/grafanacloudtempo

Configuring the Grafana Cloud Tempo backend (Managed)

### Getting Started

![grafanacloudtempo](https://d15jtxgb40qetw.cloudfront.net/grafana.svg)

> [!NOTE]
>
> These instructions are for the Grafana Cloud **managed** Tempo service. If you run a self hosted Tempo instance, please follow the instructions on the [Tempo](../backends/tempo) page.

You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Tempo as a **trace** backend.

All the connection details can be found at [grafana.com](https://grafana.com) under **My Account**

![Grafana labs website](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1e83df72b7b02521187f810d913bbd5f)

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Tempo" card and click "Send Traces"

![Tempo managed service configuration](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud4.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1fd3c17c628b2bead50e17554215305a)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **GRAFANA\_CLOUD\_TEMPO\_ENDPOINT** `string` : Endpoint. This is the URL of the Tempo service, in the form of `host:port`. You can find the endpoint under the "Sending data to Tempo" section.
  * This field is required
  * Example: `tempo-us-central1.grafana.net:443`
* **GRAFANA\_CLOUD\_TEMPO\_USERNAME** `string` : Username. You can find the username in the "Sending data to Tempo" section under the "basic\_auth" yaml field. The username is a number.
  * This field is required
* **GRAFANA\_CLOUD\_TEMPO\_PASSWORD** `string` : Password (Api Token). This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `traces:write` scope.
  * This field is required
  * Example: `glc_eyJvIj...`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Grafana Cloud Tempo` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `grafanacloudtempo.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: grafanacloudtempo-example
  namespace: odigos-system
spec:
  data:
    GRAFANA_CLOUD_TEMPO_ENDPOINT: <Endpoint>
    GRAFANA_CLOUD_TEMPO_USERNAME: <Username>
  destinationName: grafanacloudtempo
  secretRef:
    name: grafanacloudtempo-secret
  signals:
  - TRACES
  type: grafanacloudtempo

---

apiVersion: v1
data:
  GRAFANA_CLOUD_TEMPO_PASSWORD: <Base64 Password (Api Token)>
kind: Secret
metadata:
  name: grafanacloudtempo-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f grafanacloudtempo.yaml
```


# GreptimeDB
Source: https://docs.odigos.io/enterprise/backends/greptime

Configuring the GreptimeDB backend (Managed)

### Getting Started

![greptime](https://d15jtxgb40qetw.cloudfront.net/greptime.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **GREPTIME\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
  * Example: `https://<unique_id>.<aws_region>.aws.greptime.cloud`
* **GREPTIME\_DB\_NAME** `string` : Database Name.
  * This field is required
* **GREPTIME\_BASIC\_USERNAME** `string` : Basic Auth Username. The left part (from ":") of the Basic Auth Token.
  * This field is required
* **GREPTIME\_BASIC\_PASSWORD** `string` : Basic Auth Password. The right part (from ":") of the Basic Auth Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `GreptimeDB` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `greptime.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: greptime-example
  namespace: odigos-system
spec:
  data:
    GREPTIME_BASIC_USERNAME: <Basic Auth Username>
    GREPTIME_DB_NAME: <Database Name>
    GREPTIME_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: greptime
  secretRef:
    name: greptime-secret
  signals:
  - METRICS
  type: greptime

---

apiVersion: v1
data:
  GREPTIME_BASIC_PASSWORD: <Base64 Basic Auth Password>
kind: Secret
metadata:
  name: greptime-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f greptime.yaml
```


# Groundcover inCloud
Source: https://docs.odigos.io/enterprise/backends/groundcover

Configuring the Groundcover inCloud backend (Managed)

### Getting Started

![groundcover](https://d15jtxgb40qetw.cloudfront.net/groundcover.svg)

To get an API Key, refer to [these docs](https://docs.groundcover.com/architecture/byoc/ingestion-endpoints#creating-an-ingestion-key).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **GROUNDCOVER\_ENDPOINT** `string` : Groundcover inCloud Site. The format is `host:port`. Host is required, also known as your `inCloud_Site`, it is part of the configuration provided to you by Groundcover when setting up inCloud Managed. Port is optional, and defaults to the default OpenTelemetry gRPC port `4317`.
  * This field is required
* **GROUNDCOVER\_API\_KEY** `string` : Groundcover API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Groundcover inCloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `groundcover.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: groundcover-example
  namespace: odigos-system
spec:
  data:
    GROUNDCOVER_ENDPOINT: <Groundcover inCloud Site>
  destinationName: groundcover
  secretRef:
    name: groundcover-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: groundcover

---

apiVersion: v1
data:
  GROUNDCOVER_API_KEY: <Base64 Groundcover API Key>
kind: Secret
metadata:
  name: groundcover-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f groundcover.yaml
```


# Honeycomb
Source: https://docs.odigos.io/enterprise/backends/honeycomb

Configuring the Honeycomb backend (Managed)

### Getting Started

![honeycomb](https://d15jtxgb40qetw.cloudfront.net/honeycomb.svg)

**Where to find the data**

* Each Odigos destination send collected telemetry to a **single environment** in Honeycomb.
* The collected telemetry will go into Honeycomb [datasets](https://docs.honeycomb.io/configure/datasets/manage/) based on the `service.name` resource attribute. odigos will set it by default to the k8s workload (deployment) name (or as set in source configuration).

**Login to Honeycomb**\

Go to the **[🔗 website](https://ui.honeycomb.io)** and login/create your honeycomb account.
You can start with the Free plan to test sending traces, logs and metrics without needing to provide a credit card.

**Select Environment**\

Honeycomb uses [environments](https://docs.honeycomb.io/configure/environments/manage/) to partition and organize data.

Common environments includes `staging`, `production`, `development`, `integration`, or you might want to include the cluster/unit name
to group and isolate telemetry that is unrelated.

Each Odigos Honeycomb destination always sends the data to a **single environment**, based on the "API Key" it is configured with.
Before creating your API Key, make sure to select in ui the environment you want your odigos collected telemetry to show up in.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombenv.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=cd8f7b766c1d92d8fb4c18e103f29002)

**Obtaining Honeycomb API Key**

From the top left corner click on the ENVIRONMENT, and click the **Settings** configuration gear icon to go to the settings page.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombenvsettings.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=b072d71a4d9b1202eab8363962b80fb9)

From the Settings page, click on the **API Keys** tab, make sure you are on "Ingest" sub-tab and click on the **Create Ingest API Key** blue button on the right.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombcreateapikey.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=bc15baaf1bada7ba22baf6d8114a9ba6)

Give your API Key a name (include "odigos" in the name to identify it easily later on), make sure the checkbox for "Can create services/datasets" is checked, and click the **Create** button on the modal bottom right.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombcreateapikeymodal.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=2edbf9646142a7956f985b8d367ea03b)

Copy the ingest key to keyboard and paste it in your Honeycomb Odigos destination configuration (ui or kubernetes manifest).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **HONEYCOMB\_API\_KEY** `string` : API Key.
  * This field is required
* **HONEYCOMB\_ENDPOINT** `string` : Endpoint. Choose one of the endpoints in the dropdown (`api.honeycomb.io` is the US instance, `api.eu1.honeycomb.io` is EU instance)
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Honeycomb` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `honeycomb.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: honeycomb-example
  namespace: odigos-system
spec:
  data:
    HONEYCOMB_ENDPOINT: '<Endpoint (options: [api.honeycomb.io, api.eu1.honeycomb.io])>'
  destinationName: honeycomb
  secretRef:
    name: honeycomb-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: honeycomb

---

apiVersion: v1
data:
  HONEYCOMB_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: honeycomb-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f honeycomb.yaml
```


# HyperDX
Source: https://docs.odigos.io/enterprise/backends/hyperdx

Configuring the HyperDX backend (Managed)

### Getting Started

![hyperdx](https://d15jtxgb40qetw.cloudfront.net/hyperdx.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **HYPERDX\_API\_KEY** `string` : HyperDX API Key.
  * This field is required
* **HYPERDX\_LOG\_NORMALIZER** `boolean` : Enable Log Normalizer. Enable HyperDX log normalization processor. This parses JSON from log bodies, infers severity levels, and normalizes log attributes for better querying.
  * This field is optional and defaults to `True`

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `in-otel.hyperdx.io:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `HyperDX` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `hyperdx.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: hyperdx-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: hyperdx
  secretRef:
    name: hyperdx-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: hyperdx

---

apiVersion: v1
data:
  HYPERDX_API_KEY: <Base64 HyperDX API Key>
kind: Secret
metadata:
  name: hyperdx-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f hyperdx.yaml
```


# IBM Instana
Source: https://docs.odigos.io/enterprise/backends/instana

Configuring the IBM Instana backend (Managed)

### Getting Started

![instana](https://d15jtxgb40qetw.cloudfront.net/instana.svg)

**Creating Account**\

Go to the **[🔗 Website](https://www.ibm.com/products/instana?mhsrc=ibmsearch_a\&mhq=instana) > Account** and click **Log In**

**Obtaining Endpoint**\

Go to the **[📖 Docs](https://www.ibm.com/docs/en/instana-observability/current?topic=instana-backend#endpoints-of-the-instana-backend-otlp-acceptor) > Endpoints of the Instana backend otlp-acceptor** and find your **Endpoint**

**Obtaining Agent Key**\

Go to your **📩 Email > Instana Login Details** and copy **Agent-Key**

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **INSTANA\_ENDPOINT** `string` : OTLP gRPC Endpoint. Domain name of the otlp-acceptor component of the Instana backend.
  * This field is required
  * Example: `otlp-{color}-saas.instana.io`
* **INSTANA\_AGENT\_KEY** `string` : Agent Key. The agent key of the Instana for targeting the Instana backend.
  * This field is required

> [!WARNING]
>
> For ingesting logs via OpenTelemetry, an add-on to the Instana license is required. It’s advisable to contact your IBM Sales representative to include this add-on in your account.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `IBM Instana` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `instana.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: instana-example
  namespace: odigos-system
spec:
  data:
    INSTANA_ENDPOINT: <OTLP gRPC Endpoint>
  destinationName: instana
  secretRef:
    name: instana-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: instana

---

apiVersion: v1
data:
  INSTANA_AGENT_KEY: <Base64 Agent Key>
kind: Secret
metadata:
  name: instana-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f instana.yaml
```


# Jaeger
Source: https://docs.odigos.io/enterprise/backends/jaeger

Configuring the Jaeger backend (Self-Hosted)

### Getting Started

![jaeger](https://d15jtxgb40qetw.cloudfront.net/jaeger.svg)

Version v1.35 of Jaeger introduced the ability to receive OpenTelemetry trace data via the OpenTelemetry Protocol (OTLP). This allows to create a new Jaeger backend by simply specifying the Jaeger OTLP gRPC unencrypted URL.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **JAEGER\_URL** `string` : Jaeger OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`
  * This field is required
* **JAEGER\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **JAEGER\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Jaeger` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `jaeger.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: jaeger-example
  namespace: odigos-system
spec:
  data:
    JAEGER_URL: <Jaeger OTLP gRPC Endpoint>
    # Note: The commented fields below are optional.
    # JAEGER_TLS_ENABLED: <Enable TLS>
    # JAEGER_CA_PEM: <Certificate Authority>
  destinationName: jaeger
  signals:
  - TRACES
  type: jaeger
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f jaeger.yaml
```


# Kafka
Source: https://docs.odigos.io/enterprise/backends/kafka

Configuring the Kafka backend (Self-Hosted)

### Getting Started

![kafka](https://d15jtxgb40qetw.cloudfront.net/kafka.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **KAFKA\_PROTOCOL\_VERSION** `string` : Protocol Version. Kafka protocol version.
  * This field is required
  * Example: `2.0.0`
* **KAFKA\_BROKERS** `string[]` : Brokers. The list of kafka brokers
  * This field is optional and defaults to `["localhost:9092"]`
* **KAFKA\_RESOLVE\_CANONICAL\_BOOTSTRAP\_SERVERS\_ONLY** `boolean` : Resolve Canonical Bootstrap Servers Only. Whether to resolve then reverse-lookup broker IPs during startup.
  * This field is optional and defaults to `False`
* **KAFKA\_CLIENT\_ID** `string` : Client ID. The client ID to configure the Sarama Kafka client with. The client ID will be used for all produce requests.
  * This field is optional and defaults to `sarama`
* **KAFKA\_TOPIC** `string` : Topic. The name of the default kafka topic to export to (default = otlp\_spans for traces, otlp\_metrics for metrics, otlp\_logs for logs).
  * This field is optional
* **KAFKA\_TOPIC\_FROM\_ATTRIBUTE** `string` : Topic from Attribute. Specify the resource attribute whose value should be used as the message's topic.
  * This field is optional
* **KAFKA\_ENCODING** `string` : Encoding. The encoding of the traces sent to kafka.
  * This field is optional and defaults to `otlp_proto`
* **KAFKA\_PARTITION\_TRACES\_BY\_ID** `boolean` : Partition Traces by ID. Configures the exporter to include the trace ID as the message key in trace messages sent to kafka. Please note: this setting does not have any effect on Jaeger encoding exporters since Jaeger exporters include trace ID as the message key by default.
  * This field is optional and defaults to `False`
* **KAFKA\_PARTITION\_METRICS\_BY\_RESOURCE\_ATTRIBUTES** `boolean` : Partition Metrics by Resource Attributes. Configures the exporter to include the hash of sorted resource attributes as the message partitioning key in metric messages sent to kafka.
  * This field is optional and defaults to `False`
* **KAFKA\_PARTITION\_LOGS\_BY\_RESOURCE\_ATTRIBUTES** `boolean` : Partition Logs by Resource Attributes. Configures the exporter to include the hash of sorted resource attributes as the message partitioning key in log messages sent to kafka.
  * This field is optional and defaults to `False`
* **KAFKA\_AUTH\_METHOD** `string` : Auth Method. The auth method to use.
  * This field is required and defaults to `none`
* **KAFKA\_USERNAME** `string` : Username. The username to use.
  * This field is optional
* **KAFKA\_PASSWORD** `string` : Password. The password to use.
  * This field is optional
* **KAFKA\_METADATA\_FULL** `boolean` : Metadata Full. Whether to maintain a full set of metadata. When disabled, the client does not make the initial request to broker at the startup.
  * This field is optional and defaults to `False`
* **KAFKA\_METADATA\_MAX\_RETRY** `string` : Metadata Max Retry. The number of retries to get metadata.
  * This field is optional and defaults to `3`
* **KAFKA\_METADATA\_BACKOFF\_RETRY** `string` : Metadata Backoff Retry. How long to wait between metadata retries.
  * This field is optional and defaults to `250ms`
* **KAFKA\_TIMEOUT** `string` : Timeout. Is the timeout for every attempt to send data to the backend.
  * This field is optional and defaults to `5s`
* **KAFKA\_RETRY\_ON\_FAILURE\_ENABLED** `boolean` : Enable Retry on Failure.
  * This field is optional and defaults to `True`
* **KAFKA\_RETRY\_ON\_FAILURE\_INITIAL\_INTERVAL** `string` : Initial Interval. Time to wait after the first failure before retrying; ignored if `enabled` is `false`.
  * This field is optional and defaults to `5s`
* **KAFKA\_RETRY\_ON\_FAILURE\_MAX\_INTERVAL** `string` : Max Interval. Is the upper bound on backoff; ignored if `enabled` is `false`.
  * This field is optional and defaults to `30s`
* **KAFKA\_RETRY\_ON\_FAILURE\_MAX\_ELAPSED\_TIME** `string` : Max Elapsed Time. Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false`.
  * This field is optional and defaults to `120s`
* **KAFKA\_PRODUCER\_MAX\_MESSAGE\_BYTES** `string` : Producer Max Message Bytes. The maximum permitted size of a message in bytes.
  * This field is optional and defaults to `1000000`
* **KAFKA\_PRODUCER\_REQUIRED\_ACKS** `string` : Producer Required Acks. Controls when a message is regarded as transmitted.
  * This field is optional and defaults to `1`
* **KAFKA\_PRODUCER\_COMPRESSION** `string` : Producer Compression. The compression used when producing messages to kafka.
  * This field is optional and defaults to `none`
* **KAFKA\_PRODUCER\_FLUSH\_MAX\_MESSAGES** `string` : Producer Flush Max Messages. The maximum number of messages the producer will send in a single broker request.
  * This field is optional and defaults to `0`

> [!NOTE]
>
> The destination topic can be defined in a few different ways and takes priority in the following order:
>
> 1. When `topic_from_attribute` is configured, and the corresponding attribute is found on the ingested data, the value of this attribute is used.
> 2. If a prior component in the collector pipeline sets the topic on the context via the `topic.WithTopic` function (from the [github.com/open-telemetry/opentelemetry-collector-contrib/pkg/kafka/topic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/kafka/topic) package), the value set in the context is used.
> 3. Finally, the topic configuration is used as a default/fallback destination.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Kafka` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `kafka.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: kafka-example
  namespace: odigos-system
spec:
  data:
    KAFKA_AUTH_METHOD: '<Auth Method (default: none) (options: [none, plain_text])>'
    KAFKA_PROTOCOL_VERSION: <Protocol Version>
    # Note: The commented fields below are optional.
    # KAFKA_BROKERS: <Brokers (default: ["localhost:9092"])>
    # KAFKA_RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY: <Resolve Canonical Bootstrap Servers Only>
    # KAFKA_CLIENT_ID: <Client ID (default: sarama)>
    # KAFKA_TOPIC: <Topic>
    # KAFKA_TOPIC_FROM_ATTRIBUTE: <Topic from Attribute>
    # KAFKA_ENCODING: <Encoding (default: otlp_proto) (options: [otlp_proto, otlp_json])>
    # KAFKA_PARTITION_TRACES_BY_ID: <Partition Traces by ID>
    # KAFKA_PARTITION_METRICS_BY_RESOURCE_ATTRIBUTES: <Partition Metrics by Resource Attributes>
    # KAFKA_PARTITION_LOGS_BY_RESOURCE_ATTRIBUTES: <Partition Logs by Resource Attributes>
    # KAFKA_USERNAME: <Username>
    # KAFKA_METADATA_FULL: <Metadata Full>
    # KAFKA_METADATA_MAX_RETRY: <Metadata Max Retry (default: 3)>
    # KAFKA_METADATA_BACKOFF_RETRY: <Metadata Backoff Retry (default: 250ms)>
    # KAFKA_TIMEOUT: <Timeout (default: 5s)>
    # KAFKA_RETRY_ON_FAILURE_ENABLED: <Enable Retry on Failure (default: True)>
    # KAFKA_RETRY_ON_FAILURE_INITIAL_INTERVAL: <Initial Interval (default: 5s)>
    # KAFKA_RETRY_ON_FAILURE_MAX_INTERVAL: <Max Interval (default: 30s)>
    # KAFKA_RETRY_ON_FAILURE_MAX_ELAPSED_TIME: <Max Elapsed Time (default: 120s)>
    # KAFKA_PRODUCER_MAX_MESSAGE_BYTES: <Producer Max Message Bytes (default: 1000000)>
    # KAFKA_PRODUCER_REQUIRED_ACKS: <Producer Required Acks (default: 1)>
    # KAFKA_PRODUCER_COMPRESSION: <Producer Compression (default: none) (options: [none, gzip, snappy, lz4, zstd])>
    # KAFKA_PRODUCER_FLUSH_MAX_MESSAGES: <Producer Flush Max Messages (default: 0)>
  destinationName: kafka
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: kafka-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: kafka

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   KAFKA_PASSWORD: <Base64 Password>
# kind: Secret
# metadata:
#   name: kafka-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f kafka.yaml
```


# KloudMate
Source: https://docs.odigos.io/enterprise/backends/kloudmate

Configuring the KloudMate backend (Managed)

### Getting Started

![kloudmate](https://d15jtxgb40qetw.cloudfront.net/kloudmate.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **KLOUDMATE\_API\_KEY** `string` : KloudMate API Key.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `https://otel.kloudmate.com:4318`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `KloudMate` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `kloudmate.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: kloudmate-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: kloudmate
  secretRef:
    name: kloudmate-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: kloudmate

---

apiVersion: v1
data:
  KLOUDMATE_API_KEY: <Base64 KloudMate API Key>
kind: Secret
metadata:
  name: kloudmate-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f kloudmate.yaml
```


# Last9
Source: https://docs.odigos.io/enterprise/backends/last9

Configuring the Last9 backend (Managed)

### Getting Started

![last9](https://d15jtxgb40qetw.cloudfront.net/last9.svg)

[Click here](https://app.last9.io/integrations?category=all\&integration=OpenTelemetry) to visit the Last9 OpenTelemetry integration page.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **LAST9\_OTLP\_ENDPOINT** `string` : Last9 OpenTelemetry Endpoint.
  * This field is required
* **LAST9\_OTLP\_BASIC\_AUTH\_HEADER** `string` : Basic Auth Header.
  * This field is required
  * Example: `Basic ...`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Last9` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `last9.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: last9-example
  namespace: odigos-system
spec:
  data:
    LAST9_OTLP_ENDPOINT: <Last9 OpenTelemetry Endpoint>
  destinationName: last9
  secretRef:
    name: last9-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: last9

---

apiVersion: v1
data:
  LAST9_OTLP_BASIC_AUTH_HEADER: <Base64 Basic Auth Header>
kind: Secret
metadata:
  name: last9-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f last9.yaml
```


# Lightstep
Source: https://docs.odigos.io/enterprise/backends/lightstep

Configuring the Lightstep backend (Managed)

### Getting Started

![lightstep](https://d15jtxgb40qetw.cloudfront.net/lightstep.svg)

**Obtaining Lightstep Access Token**\

Go to **⚙️ > Access Tokens** and click **Create New**

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **LIGHTSTEP\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Lightstep` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `lightstep.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: lightstep-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: lightstep
  secretRef:
    name: lightstep-secret
  signals:
  - TRACES
  type: lightstep

---

apiVersion: v1
data:
  LIGHTSTEP_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: lightstep-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f lightstep.yaml
```


# Logz.io
Source: https://docs.odigos.io/enterprise/backends/logzio

Configuring the Logz.io backend (Managed)

### Getting Started

![logzio](https://d15jtxgb40qetw.cloudfront.net/logzio.svg)

**Finding Logz.io Region**

You can find your account’s region by selecting **⚙️ > Settings > General** from the top menu. Your two-letter region code is at the start of the region designation.
Another option is to sign in to your Logz.io account and look at the URL in the address bar.

If you see `app.logz.io`, your account is hosted in the US East (Northern Virginia). All other regions have a two-letter region code. For example, if you see `app-eu.logz.io`, then your account is in the Europe (Frankfurt) region.

**Obtaining Tokens**

From your account, go to the **Manage Tokens > Data shipping** tokens tab of your Operations workspace.
It can be reached by selecting **⚙️ > Settings > Manage tokens** in the ADMIN ZONE section of the side navigation menu.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **LOGZIO\_REGION** `string` : Region. The region of your Logz.io account
  * This field is required
* **LOGZIO\_TRACING\_TOKEN** `string` : Tracing token. The token you obtained from Logz.io
  * This field is optional
* **LOGZIO\_METRICS\_TOKEN** `string` : Metrics token. The token you obtained from Logz.io
  * This field is optional
* **LOGZIO\_LOGS\_TOKEN** `string` : Logs token. The token you obtained from Logz.io
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Logz.io` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `logzio.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: logzio-example
  namespace: odigos-system
spec:
  data:
    LOGZIO_REGION: '<Region (options: [listener.logz.io, listener-au.logz.io, listener-ca.logz.io,
      listener-eu.logz.io, listener-nl.logz.io, listener-uk.logz.io, listener-wa.logz.io])>'
  destinationName: logzio
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: logzio-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: logzio

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   LOGZIO_LOGS_TOKEN: <Base64 Logs token>
#   LOGZIO_METRICS_TOKEN: <Base64 Metrics token>
#   LOGZIO_TRACING_TOKEN: <Base64 Tracing token>
# kind: Secret
# metadata:
#   name: logzio-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f logzio.yaml
```


# Loki
Source: https://docs.odigos.io/enterprise/backends/loki

Configuring the Loki backend (Self-Hosted)

### Getting Started

![loki](https://d15jtxgb40qetw.cloudfront.net/loki.svg)

> [!NOTE]
>
> These instructions are for the **self hosted** Loki instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Loki](../backends/grafanacloudloki) page.

This destination is for self hosted Loki instances.

**Loki Labels**

Loki labels are used to "index" logs and allow efficient filtering in the Grafana UI. You can read more about them [here](https://grafana.com/docs/loki/latest/get-started/labels/).

You can configure your grafana cloud loki labels in the `Labels` multi-input config option in the destination page in the Odigos UI.

* Labels should be names of OpenTelemetry resource attribute or log record attribute keys that you want to index, for example: `"k8s.namespace.name"`.
* To avoid performance issues, it is recommended to limit the number of labels to a reasonable amount.
* [Use only low cardinality values](https://grafana.com/docs/loki/latest/get-started/labels/#cardinality). e.g. use only OpenTelemetry attributes for which you expect a small number of unique values like `http.response.status_code` but **not** `network.peer.address`.
* If the label is not present in a log record, it will be ignored.
* The default labels if not set are: `k8s.container.name, k8s.pod.name, k8s.namespace.name`

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **LOKI\_URL** `string` : Endpoint. If scheme is not provided, it defaults to `http`. If port is not provided, it defaults to `3100`. If path is not provided, it defaults to `/otlp`.
  * This field is required
  * Example: `http://<loki-host>:<loki-port>/otlp`
* **LOKI\_SCOPE\_ORG\_ID** `string` : X-Scope-OrgID. use this X-Scope-OrgID if you need to scope the data to a specific org. Multiple orgs can be chained using `|`
  * This field is optional
  * Example: `org1|org2|org3`
* **LOKI\_USERNAME** `string` : Basic Auth Username. username for basic authentication
  * This field is optional
  * Example: `<loki-username>`
* **LOKI\_PASSWORD** `string` : Basic Auth Password. password for basic authentication
  * This field is optional
  * Example: `<loki-password>`
* **LOKI\_LABELS** `string[]` : Labels. use these OpenTelemetry resource attributes as loki labels for each log record
  * This field is required and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]`

> [!NOTE]
>
> The supported Loki version is V3. Please make sure you are using the correct version, and that you have enabled native OTLP support. As part of this version, the path `/loki/api/v1/push` has been deprecated and replaced with `/otlp`.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Loki` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `loki.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: loki-example
  namespace: odigos-system
spec:
  data:
    LOKI_LABELS: '<Labels (default: ["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"])>'
    LOKI_URL: <Endpoint>
    # Note: The commented fields below are optional.
    # LOKI_SCOPE_ORG_ID: <X-Scope-OrgID>
    # LOKI_USERNAME: <Basic Auth Username>
  destinationName: loki
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: loki-secret
  signals:
  - LOGS
  type: loki

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   LOKI_PASSWORD: <Base64 Basic Auth Password>
# kind: Secret
# metadata:
#   name: loki-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f loki.yaml
```


# Lumigo
Source: https://docs.odigos.io/enterprise/backends/lumigo

Configuring the Lumigo backend (Managed)

### Getting Started

![lumigo](https://d15jtxgb40qetw.cloudfront.net/lumigo.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **LUMIGO\_ENDPOINT** `string` : Lumigo OTLP HTTP Endpoint.
  * This field is required and defaults to `https://ga-otlp.lumigo-tracer-edge.golumigo.com`
* **LUMIGO\_TOKEN** `string` : Lumigo Authorization Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Lumigo` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `lumigo.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: lumigo-example
  namespace: odigos-system
spec:
  data:
    LUMIGO_ENDPOINT: '<Lumigo OTLP HTTP Endpoint (default: https://ga-otlp.lumigo-tracer-edge.golumigo.com)>'
  destinationName: lumigo
  secretRef:
    name: lumigo-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: lumigo

---

apiVersion: v1
data:
  LUMIGO_TOKEN: <Base64 Lumigo Authorization Token>
kind: Secret
metadata:
  name: lumigo-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f lumigo.yaml
```


# Middleware
Source: https://docs.odigos.io/enterprise/backends/middleware

Configuring the Middleware backend (Managed)

### Getting Started

![middleware](https://d15jtxgb40qetw.cloudfront.net/middleware.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **MW\_TARGET** `string` : Endpoint.
  * This field is required
* **MW\_API\_KEY** `string` : API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Middleware` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `middleware.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: middleware-example
  namespace: odigos-system
spec:
  data:
    MW_TARGET: <Endpoint>
  destinationName: middleware
  secretRef:
    name: middleware-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: middleware

---

apiVersion: v1
data:
  MW_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: middleware-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f middleware.yaml
```


# New Relic
Source: https://docs.odigos.io/enterprise/backends/newrelic

Configuring the New Relic backend (Managed)

### Getting Started

![newrelic](https://d15jtxgb40qetw.cloudfront.net/newrelic.svg)

**Obtaining a License Key**\

On the left sidebar, click on your account name and select **API Keys** and then click on **Create a key**.

![New Relic API keys](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/newrelic1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=11888f0e65ddb12403e887acd2b8f763)

For key type select **Ingest - License**, give a name to your key and press **Create a key**.

![Create an API key screen in New Relic](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/newrelic2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=4e8c022b81cd8287902d373ac2f2e1b4)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **NEWRELIC\_API\_KEY** `string` : License Key.
  * This field is required
* **NEWRELIC\_ENDPOINT** `string` : Endpoint.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `New Relic` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `newrelic.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: newrelic-example
  namespace: odigos-system
spec:
  data:
    NEWRELIC_ENDPOINT: '<Endpoint (options: [https://otlp.nr-data.net, https://otlp.eu01.nr-data.net])>'
  destinationName: newrelic
  secretRef:
    name: newrelic-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: newrelic

---

apiVersion: v1
data:
  NEWRELIC_API_KEY: <Base64 License Key>
kind: Secret
metadata:
  name: newrelic-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f newrelic.yaml
```


# Observe
Source: https://docs.odigos.io/enterprise/backends/observe

Configuring the Observe backend (Managed)

### Getting Started

![observe](https://d15jtxgb40qetw.cloudfront.net/observe.svg)

**Creating an ingest token**\

Data is sent to Observe's OTLP collection endpoint, which requires a **Datastream ingest token** (a user/API token will be rejected with `401`). Create one in the Observe UI under **Datastreams**, following [Configure your own OTel collector without Kubernetes](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment).

**Collection endpoint**\

By default the endpoint is `https://<customer_id>.collect.observeinc.com/v2/otel`. For regional tenants or other overrides, set the optional `OBSERVE_ENDPOINT` field. See [Configure your own OTel collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) for the endpoint format.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **OBSERVE\_CUSTOMER\_ID** `string` : Customer ID. The Customer ID is the first part of your Observe URL. For example, if your Observe URL is `https://<customer_id>.observe.com`, then the Customer ID is `<customer_id>`.
  * This field is required
* **OBSERVE\_TOKEN** `string` : Token. A Datastream ingest token (not a user/API token). Create one in the Observe UI under "Datastreams". See [https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment) for how to create an ingest token.
  * This field is required
* **OBSERVE\_ENDPOINT** `string` : Collection Endpoint. Optional. The full OTLP/HTTP collection endpoint. Set this to override the default, for example for a regional tenant such as `https://<customer_id>.collect.eu1.observeinc.com/v2/otel`. When empty, it is derived from the Customer ID. See [https://docs.observeinc.com/docs/configure-your-own-otel-collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) for the endpoint format.
  * This field is optional
  * Example: `https://<customer_id>.collect.observeinc.com/v2/otel`

> [!TIP]
>
> By default the endpoint is derived from your Customer ID as `https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/v2/otel`.
> To send to a different endpoint (for example a regional tenant), set the optional `OBSERVE_ENDPOINT` field.
> For details on the collection endpoint and creating an ingest token, see Observe's guides for [configuring your own OTel collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) and [configuring it without Kubernetes](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment).

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Observe` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `observe.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: observe-example
  namespace: odigos-system
spec:
  data:
    OBSERVE_CUSTOMER_ID: <Customer ID>
    # Note: The commented fields below are optional.
    # OBSERVE_ENDPOINT: <Collection Endpoint>
  destinationName: observe
  secretRef:
    name: observe-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: observe

---

apiVersion: v1
data:
  OBSERVE_TOKEN: <Base64 Token>
kind: Secret
metadata:
  name: observe-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f observe.yaml
```


# OneUptime
Source: https://docs.odigos.io/enterprise/backends/oneuptime

Configuring the OneUptime backend (Managed)

### Getting Started

![oneuptime](https://d15jtxgb40qetw.cloudfront.net/oneuptime.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **ONEUPTIME\_INGESTION\_KEY** `string` : Telemetry Ingestion Key. The Telemetry Ingestion Key is used to authenticate your data. You can find it in the OneUptime dashboard under Project Settings > Ingestion Keys.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `https://oneuptime.com/otlp`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OneUptime` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `oneuptime.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: oneuptime-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: oneuptime
  secretRef:
    name: oneuptime-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: oneuptime

---

apiVersion: v1
data:
  ONEUPTIME_INGESTION_KEY: <Base64 Telemetry Ingestion Key>
kind: Secret
metadata:
  name: oneuptime-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f oneuptime.yaml
```


# OpenObserve
Source: https://docs.odigos.io/enterprise/backends/openobserve

Configuring the OpenObserve backend (Managed)

### Getting Started

![openobserve](https://d15jtxgb40qetw.cloudfront.net/openobserve.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **OPEN\_OBSERVE\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
  * Example: `https://api.openobserve.ai/api/<organization_id>`
* **OPEN\_OBSERVE\_API\_KEY** `string` : API Key. Do not include the `Basic` prefix.
  * This field is required
* **OPEN\_OBSERVE\_STREAM\_NAME** `string` : Stream Name.
  * This field is required and defaults to `default`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OpenObserve` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `openobserve.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: openobserve-example
  namespace: odigos-system
spec:
  data:
    OPEN_OBSERVE_ENDPOINT: <OTLP HTTP Endpoint>
    OPEN_OBSERVE_STREAM_NAME: '<Stream Name (default: default)>'
  destinationName: openobserve
  secretRef:
    name: openobserve-secret
  signals:
  - TRACES
  - LOGS
  type: openobserve

---

apiVersion: v1
data:
  OPEN_OBSERVE_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: openobserve-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f openobserve.yaml
```


# Oracle Cloud
Source: https://docs.odigos.io/enterprise/backends/oracle

Configuring the Oracle Cloud backend (Managed)

### Getting Started

![oracle](https://d15jtxgb40qetw.cloudfront.net/oracle.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **ORACLE\_ENDPOINT** `string` : OTLP HTTP Endpoint. The data upload endpoint of the Oracle APM domain.
  * This field is required
* **ORACLE\_DATA\_KEY** `string` : Data Key.
  * This field is required
* **ORACLE\_DATA\_KEY\_TYPE** `string` : Data Key Type. The type of the data key, relevant for Traces. Metrics are always authenticated with a private data key.
  * This field is required and defaults to `private`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Oracle Cloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `oracle.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: oracle-example
  namespace: odigos-system
spec:
  data:
    ORACLE_DATA_KEY_TYPE: '<Data Key Type (default: private) (options: [private, public])>'
    ORACLE_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: oracle
  secretRef:
    name: oracle-secret
  signals:
  - TRACES
  - METRICS
  type: oracle

---

apiVersion: v1
data:
  ORACLE_DATA_KEY: <Base64 Data Key>
kind: Secret
metadata:
  name: oracle-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f oracle.yaml
```


# OTLP gRPC
Source: https://docs.odigos.io/enterprise/backends/otlp

Configuring the OTLP gRPC backend (Self-Hosted)

### Getting Started

![otlp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg)

For advanced users trying to implement complex observability pipelines, Odigos support sending data to any OTLP gRPC unencrypted endpoint.

> [!NOTE]
>
> Note that if your backend expects [OTLP over http](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) you should use the [OTLP http](../backends/otlphttp) destination instead.

> [!TIP]
>
> If your backend is supported natively in Odigos, it's recommended to use the native integration.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ✅ Profiles

* **OTLP\_GRPC\_ENDPOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`.
  * This field is required
  * Example: `host:port`
* **OTLP\_GRPC\_COMPRESSION** `string` : Destination Compression Type. Compression type to use for the destination. The default is `none`. The compression type must be supported by the destination.
  * This field is optional and defaults to `none`
* **OTLP\_GRPC\_HEADERS** `{ key: string; value: string; }[]` : Headers. Headers is the option to set custom GRPC headers for OTLP GRPC destination. If specified, please provide each header in the format: key:value. Multiple headers can be added. Keys must be non-empty strings and follow standard HTTP header conventions. Values must be non-empty strings and may include alphanumerics, whitespace, and standard punctuation.
  * This field is optional
* **OTLP\_GRPC\_OAUTH2\_ENABLED** `boolean` : Enable OAuth2. Enable OAuth2 client credentials authentication
  * This field is optional and defaults to `False`
* **OTLP\_GRPC\_OAUTH2\_CLIENT\_ID** `string` : OAuth2 Client ID. OAuth2 client identifier for client credentials flow
  * This field is optional
  * Example: `your-client-id`
* **OTLP\_GRPC\_OAUTH2\_CLIENT\_SECRET** `string` : OAuth2 Client Secret. OAuth2 client secret for client credentials flow
  * This field is optional
  * Example: `your-client-secret`
* **OTLP\_GRPC\_OAUTH2\_TOKEN\_URL** `string` : OAuth2 Token URL. OAuth2 token endpoint URL for obtaining access tokens
  * This field is optional
  * Example: `https://example.com/oauth2/token`
* **OTLP\_GRPC\_OAUTH2\_SCOPES** `string` : OAuth2 Scopes. Comma-separated list of OAuth2 scopes to request (e.g., "api.metrics,api.traces")
  * This field is optional
  * Example: `api.metrics,api.traces`
* **OTLP\_GRPC\_OAUTH2\_AUDIENCE** `string` : OAuth2 Audience. OAuth2 audience parameter for token requests
  * This field is optional
  * Example: `api.example.com`
* **OTLP\_GRPC\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **OTLP\_GRPC\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_GRPC\_MTLS\_ENABLED** `boolean` : Enable mTLS. Use a client certificate and key for mutual TLS authentication with the destination
  * This field is optional and defaults to `False`
* **OTLP\_GRPC\_CLIENT\_CERT\_PEM** `string` : Client Certificate. Client certificate in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_GRPC\_CLIENT\_KEY\_PEM** `string` : Client Key. Client private key in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN PRIVATE KEY-----`
* **OTLP\_GRPC\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OTLP gRPC` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `otlp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: otlp-example
  namespace: odigos-system
spec:
  data:
    OTLP_GRPC_ENDPOINT: <OTLP gRPC Endpoint>
    # Note: The commented fields below are optional.
    # OTLP_GRPC_COMPRESSION: <Destination Compression Type (default: none) (options: [none, gzip, snappy, lz4, zlib, deflate, zstd])>
    # OTLP_GRPC_HEADERS: <Headers>
    # OTLP_GRPC_OAUTH2_ENABLED: <Enable OAuth2>
    # OTLP_GRPC_OAUTH2_CLIENT_ID: <OAuth2 Client ID>
    # OTLP_GRPC_OAUTH2_TOKEN_URL: <OAuth2 Token URL>
    # OTLP_GRPC_OAUTH2_SCOPES: <OAuth2 Scopes>
    # OTLP_GRPC_OAUTH2_AUDIENCE: <OAuth2 Audience>
    # OTLP_GRPC_TLS_ENABLED: <Enable TLS>
    # OTLP_GRPC_CA_PEM: <Certificate Authority>
    # OTLP_GRPC_MTLS_ENABLED: <Enable mTLS>
    # OTLP_GRPC_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
  destinationName: otlp
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: otlp-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  - PROFILES
  type: otlp

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   OTLP_GRPC_CLIENT_CERT_PEM: <Base64 Client Certificate>
#   OTLP_GRPC_CLIENT_KEY_PEM: <Base64 Client Key>
#   OTLP_GRPC_OAUTH2_CLIENT_SECRET: <Base64 OAuth2 Client Secret>
# kind: Secret
# metadata:
#   name: otlp-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f otlp.yaml
```


# OTLP http
Source: https://docs.odigos.io/enterprise/backends/otlphttp

Configuring the OTLP http backend (Self-Hosted)

### Getting Started

![otlphttp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg)

For advanced users trying to implement complex observability pipelines, Odigos support sending data to any OTLP http endpoint.

> [!NOTE]
>
> Note that if your backend expects [OTLP over gRPC](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) you should use the [OTLP gRPC](../backends/otlp) destination instead.

> [!TIP]
>
> If your backend is supported natively in Odigos, it's recommended to use the native integration.

#### Using Basic Authentication

This section is relevant if your OTLP http endpoint requires [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) (username and password).

To configure basic authentication, use the optional config options `Basic Auth Username` and `Basic Auth Password`.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ✅ Profiles

* **OTLP\_HTTP\_ENDPOINT** `string` : OTLP http Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`.
  * This field is required
  * Example: `http://host:port`
* **OTLP\_HTTP\_BASIC\_AUTH\_USERNAME** `string` : Basic Auth Username. in case the otlp receiver requires basic auth, this is the username
  * This field is optional
  * Example: `username`
* **OTLP\_HTTP\_BASIC\_AUTH\_PASSWORD** `string` : Basic Auth Password. in case the otlp receiver requires basic auth, this is the password
  * This field is optional
  * Example: `password`
* **OTLP\_HTTP\_OAUTH2\_ENABLED** `boolean` : Enable OAuth2. Enable OAuth2 client credentials authentication
  * This field is optional and defaults to `False`
* **OTLP\_HTTP\_OAUTH2\_CLIENT\_ID** `string` : OAuth2 Client ID. OAuth2 client identifier for client credentials flow
  * This field is optional
  * Example: `your-client-id`
* **OTLP\_HTTP\_OAUTH2\_CLIENT\_SECRET** `string` : OAuth2 Client Secret. OAuth2 client secret for client credentials flow
  * This field is optional
  * Example: `your-client-secret`
* **OTLP\_HTTP\_OAUTH2\_TOKEN\_URL** `string` : OAuth2 Token URL. OAuth2 token endpoint URL for obtaining access tokens
  * This field is optional
  * Example: `https://example.com/oauth2/token`
* **OTLP\_HTTP\_OAUTH2\_SCOPES** `string` : OAuth2 Scopes. Comma-separated list of OAuth2 scopes to request (e.g., "api.metrics,api.traces")
  * This field is optional
  * Example: `api.metrics,api.traces`
* **OTLP\_HTTP\_OAUTH2\_AUDIENCE** `string` : OAuth2 Audience. OAuth2 audience parameter for token requests
  * This field is optional
  * Example: `api.example.com`
* **OTLP\_HTTP\_COMPRESSION** `string` : Destination Compression Type. Compression type to use for the destination. The default is `none`. The compression type must be supported by the destination.
  * This field is optional and defaults to `none`
* **OTLP\_HTTP\_HEADERS** `{ key: string; value: string; }[]` : Headers. Headers is the option to set custom HTTP headers for OTLP HTTP destination. If specified, please provide each header in the format: key:value. Multiple headers can be added. Keys must be non-empty strings and follow standard HTTP header conventions. Values must be non-empty strings and may include alphanumerics, whitespace, and standard punctuation.
  * This field is optional
* **OTLP\_HTTP\_TRACES\_ENDPOINT** `string` : OTLP http traces Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/traces` will be used.
  * This field is optional
  * Example: `http://host:port/v1/traces`
* **OTLP\_HTTP\_METRICS\_ENDPOINT** `string` : OTLP http metrics Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/metrics` will be used.
  * This field is optional
  * Example: `http://host:port/v1/metrics`
* **OTLP\_HTTP\_LOGS\_ENDPOINT** `string` : OTLP http logs Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/logs` will be used.
  * This field is optional
  * Example: `http://host:port/v1/logs`
* **OTLP\_HTTP\_PROFILES\_ENDPOINT** `string` : OTLP http profiles Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/profiles` will be used.
  * This field is optional
  * Example: `http://host:port/v1/profiles`
* **OTLP\_HTTP\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **OTLP\_HTTP\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_HTTP\_MTLS\_ENABLED** `boolean` : Enable mTLS. Use a client certificate and key for mutual TLS authentication with the destination
  * This field is optional and defaults to `False`
* **OTLP\_HTTP\_CLIENT\_CERT\_PEM** `string` : Client Certificate. Client certificate in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_HTTP\_CLIENT\_KEY\_PEM** `string` : Client Key. Client private key in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN PRIVATE KEY-----`
* **OTLP\_HTTP\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OTLP http` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `otlphttp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: otlphttp-example
  namespace: odigos-system
spec:
  data:
    OTLP_HTTP_ENDPOINT: <OTLP http Endpoint>
    # Note: The commented fields below are optional.
    # OTLP_HTTP_BASIC_AUTH_USERNAME: <Basic Auth Username>
    # OTLP_HTTP_OAUTH2_ENABLED: <Enable OAuth2>
    # OTLP_HTTP_OAUTH2_CLIENT_ID: <OAuth2 Client ID>
    # OTLP_HTTP_OAUTH2_TOKEN_URL: <OAuth2 Token URL>
    # OTLP_HTTP_OAUTH2_SCOPES: <OAuth2 Scopes>
    # OTLP_HTTP_OAUTH2_AUDIENCE: <OAuth2 Audience>
    # OTLP_HTTP_COMPRESSION: <Destination Compression Type (default: none) (options: [none, gzip, snappy, lz4, zlib, deflate, zstd])>
    # OTLP_HTTP_HEADERS: <Headers>
    # OTLP_HTTP_TRACES_ENDPOINT: <OTLP http traces Endpoint>
    # OTLP_HTTP_METRICS_ENDPOINT: <OTLP http metrics Endpoint>
    # OTLP_HTTP_LOGS_ENDPOINT: <OTLP http logs Endpoint>
    # OTLP_HTTP_PROFILES_ENDPOINT: <OTLP http profiles Endpoint>
    # OTLP_HTTP_TLS_ENABLED: <Enable TLS>
    # OTLP_HTTP_CA_PEM: <Certificate Authority>
    # OTLP_HTTP_MTLS_ENABLED: <Enable mTLS>
    # OTLP_HTTP_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
  destinationName: otlphttp
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: otlphttp-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  - PROFILES
  type: otlphttp

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   OTLP_HTTP_BASIC_AUTH_PASSWORD: <Base64 Basic Auth Password>
#   OTLP_HTTP_CLIENT_CERT_PEM: <Base64 Client Certificate>
#   OTLP_HTTP_CLIENT_KEY_PEM: <Base64 Client Key>
#   OTLP_HTTP_OAUTH2_CLIENT_SECRET: <Base64 OAuth2 Client Secret>
# kind: Secret
# metadata:
#   name: otlphttp-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f otlphttp.yaml
```


# Prometheus
Source: https://docs.odigos.io/enterprise/backends/prometheus

Configuring the Prometheus backend (Self-Hosted)

### Getting Started

![prometheus](https://d15jtxgb40qetw.cloudfront.net/prometheus.svg)

> [!NOTE]
>
> These instructions are for the **self hosted** Prometheus instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Prometheus](../backends/grafanacloudprometheus) page.

**Enabling Remote Write in Prometheus**\

Currently, Odigos supports Prometheus via the Remote Write API. In order to enable remote write in Prometheus, set [the following feature flag](https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver).

#### Using Authentication

To enable authentication for Prometheus remote write, check the `Use Authentication` checkbox. Odigos supports two authentication methods:

* **Bearer Token Authentication**: Use the `Bearer Token` field to provide a bearer token for authentication. This method is used if basic auth username is not provided.
* **Basic Authentication**: Use the `Basic Auth Username` and `Basic Auth Password` fields if your Prometheus remote write endpoint requires [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). Basic auth takes precedence if username is provided.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **PROMETHEUS\_REMOTEWRITE\_URL** `string` : Remote Write URL. The remote write endpoint. If the URL ends with a known remote write path (`/api/v1/write` for Prometheus/Thanos or `/api/v1/push` for Mimir) it is used as-is; otherwise `/api/v1/write` is appended.
  * This field is required
* **PROMETHEUS\_RESOURCE\_ATTRIBUTES\_LABELS** `string[]` : Resource Attributes Labels (Deprecated). deprecated. will be removed soon.
  * This field is optional
* **PROMETHEUS\_USE\_AUTHENTICATION** `boolean` : Use Authentication.
  * This field is optional and defaults to `False`
* **PROMETHEUS\_BEARER\_TOKEN** `string` : Bearer Token. use this bearer token if your prometheus remote write endpoint requires authentication
  * This field is optional
  * Example: `your-bearer-token`
* **PROMETHEUS\_BASIC\_AUTH\_USERNAME** `string` : Basic Auth Username. username for basic authentication (if your prometheus remote write endpoint requires basic auth)
  * This field is optional
  * Example: `username`
* **PROMETHEUS\_BASIC\_AUTH\_PASSWORD** `string` : Basic Auth Password. password for basic authentication (if your prometheus remote write endpoint requires basic auth)
  * This field is optional
  * Example: `password`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Prometheus` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `prometheus.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: prometheus-example
  namespace: odigos-system
spec:
  data:
    PROMETHEUS_REMOTEWRITE_URL: <Remote Write URL>
    # Note: The commented fields below are optional.
    # PROMETHEUS_RESOURCE_ATTRIBUTES_LABELS: <Resource Attributes Labels (Deprecated)>
    # PROMETHEUS_USE_AUTHENTICATION: <Use Authentication>
    # PROMETHEUS_BASIC_AUTH_USERNAME: <Basic Auth Username>
  destinationName: prometheus
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: prometheus-secret
  signals:
  - METRICS
  type: prometheus

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   PROMETHEUS_BASIC_AUTH_PASSWORD: <Base64 Basic Auth Password>
#   PROMETHEUS_BEARER_TOKEN: <Base64 Bearer Token>
# kind: Secret
# metadata:
#   name: prometheus-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f prometheus.yaml
```


# Pyroscope
Source: https://docs.odigos.io/enterprise/backends/pyroscope

Configuring the Pyroscope backend (Self-Hosted)

### Getting Started

![pyroscope](https://d15jtxgb40qetw.cloudfront.net/pyroscope.svg)

[Grafana Pyroscope](https://grafana.com/oss/pyroscope/) is an open-source continuous profiling backend. Odigos exports profile data over OTLP to Pyroscope so the same eBPF profiler that powers in-product views can also feed long-term, queryable storage.

> [!NOTE]
>
> This destination only emits the **Profiles** signal. The destination must include `PROFILES`, and continuous profiling must be enabled for the agent you use:
>
> * **K8s Agent**: [Continuous Profiling overview](/enterprise/instrumentations/profiling/overview) and [getting started](/enterprise/instrumentations/profiling/getting-started)
> * **VM Agent**: [Continuous Profiling overview](/vmagent/instrumentations/profiling/overview) and [getting started](/vmagent/instrumentations/profiling/getting-started)

**Pyroscope version**\

Use a Pyroscope build that accepts OTLP profiles on the endpoint you configure. If you are unsure, upgrade to a recent release before configuring Odigos.

**Quickstart: self-hosted Pyroscope**\

If you don't already run Pyroscope, the upstream Helm chart gets you a working instance in a couple of commands:

```bash theme={null}
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install pyroscope grafana/pyroscope \
  --namespace pyroscope --create-namespace
```

The distributor will be reachable in-cluster at `pyroscope.pyroscope:4040` — use that as the `PYROSCOPE_URL` value below.

> [!WARNING]
>
> Provide the distributor's `host:port` only. Do **not** add `/otlp` or other path suffixes — Odigos sends OTLP profiles to `/v1development/profiles` on the host you configure.

**TLS**\

Toggle `PYROSCOPE_TLS_ENABLED` when Pyroscope is fronted by TLS. For plain traffic, leave it off.

**Verifying the export**\

After applying the destination and sending traffic to a profiled source, open the Pyroscope UI (`kubectl -n pyroscope port-forward svc/pyroscope 4040`) and confirm that profiles appear under your service.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ❌ Metrics
> ❌ Logs
> ✅ Profiles

* **PYROSCOPE\_URL** `string` : Pyroscope Endpoint. Hostname and port of the Pyroscope distributor HTTP API (e.g. `pyroscope.pyroscope:4040`). Odigos sends OTLP profiles to `/v1development/profiles` on this host; do not add an `/otlp` path.
  * This field is required
  * Example: `host:port`
* **PYROSCOPE\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Pyroscope` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `pyroscope.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: pyroscope-example
  namespace: odigos-system
spec:
  data:
    PYROSCOPE_URL: <Pyroscope Endpoint>
    # Note: The commented fields below are optional.
    # PYROSCOPE_TLS_ENABLED: <Enable TLS>
  destinationName: pyroscope
  signals:
  - PROFILES
  type: pyroscope
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f pyroscope.yaml
```


# qryn
Source: https://docs.odigos.io/enterprise/backends/qryn

Configuring the qryn backend (Self-Hosted)

### Getting Started

![qryn-oss](https://d15jtxgb40qetw.cloudfront.net/qryn.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **QRYN\_OSS\_URL** `string` : API Url. The API Endpoint for for your qryn instance
  * This field is required
  * Example: `http://qryn.local:3100`
* **QRYN\_OSS\_PASSWORD** `string` : Basic auth password. The HTTP Basic Auth password for your qryn instance
  * This field is optional
* **QRYN\_OSS\_USERNAME** `string` : Basic auth username. The HTTP Basic Auth username for your qryn instance
  * This field is optional
* **QRYN\_OSS\_RESOURCE\_TO\_TELEMETRY\_CONVERSION** `boolean` : Convert container attributes to labels.
  * This field is optional and defaults to `True`
* **QRYN\_OSS\_ADD\_EXPORTER\_NAME** `boolean` : Add exporter name to labels.
  * This field is optional and defaults to `True`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `qryn` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `qryn-oss.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: qryn-oss-example
  namespace: odigos-system
spec:
  data:
    QRYN_OSS_URL: <API Url>
    # Note: The commented fields below are optional.
    # QRYN_OSS_USERNAME: <Basic auth username>
    # QRYN_OSS_RESOURCE_TO_TELEMETRY_CONVERSION: <Convert container attributes to labels (default: True)>
    # QRYN_OSS_ADD_EXPORTER_NAME: <Add exporter name to labels (default: True)>
  destinationName: qryn-oss
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: qryn-oss-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: qryn-oss

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   QRYN_OSS_PASSWORD: <Base64 Basic auth password>
# kind: Secret
# metadata:
#   name: qryn-oss-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f qryn-oss.yaml
```


# Quickwit
Source: https://docs.odigos.io/enterprise/backends/quickwit

Configuring the Quickwit backend (Self-Hosted)

### Getting Started

![quickwit](https://d15jtxgb40qetw.cloudfront.net/quickwit.svg)

You'll find more details here: [https://quickwit.io/docs/log-management/otel-service](https://quickwit.io/docs/log-management/otel-service)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **QUICKWIT\_URL** `string` : Quickwit OTLP gRPC Endpoint. The unencrypted gRPC endpoint of the Quickwit OTLP receiver (indexer component). The default Quickwit gRPC port if not changed is `7281`
  * This field is required
  * Example: `quickwit.example.com:7281`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Quickwit` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `quickwit.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: quickwit-example
  namespace: odigos-system
spec:
  data:
    QUICKWIT_URL: <Quickwit OTLP gRPC Endpoint>
  destinationName: quickwit
  signals:
  - TRACES
  - LOGS
  type: quickwit
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f quickwit.yaml
```


# Sentry
Source: https://docs.odigos.io/enterprise/backends/sentry

Configuring the Sentry backend

## Obtaining a Sentry DSN

To use Sentry, you need to obtain a DSN (Data Source Name) from your Sentry account.
You can find this in the Sentry UI under "Client Keys (DSN)".

## Configuring Sentry Backend

Add a new backend with the DSN from your Sentry account.


# Seq
Source: https://docs.odigos.io/enterprise/backends/seq

Configuring the Seq backend (Self-Hosted)

### Getting Started

![seq](https://d15jtxgb40qetw.cloudfront.net/seq.svg)

* [Deploy Seq into Kubernetes using Docker](https://docs.datalust.co/docs/getting-started-with-docker)
* [Deploy Seq into Kubernetes using Helm](https://docs.datalust.co/docs/using-helm)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **SEQ\_ENDPOINT** `string` : OTLP HTTP Endpoint. The format is `http://host:port`, `host` is required and equals the cluster internal DNS record of the Seq instance, `port` is required and equals any of the exposed ports of the Seq instance (defaults to 5341), do not add the rest of the path `/ingest/otlp/v1/*`.
  * This field is required
* **SEQ\_API\_KEY** `string` : API Key. The API key is generated in the Seq instance. The API key must have the `Ingest` permission.
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Seq` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `seq.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: seq-example
  namespace: odigos-system
spec:
  data:
    SEQ_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: seq
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: seq-secret
  signals:
  - TRACES
  - LOGS
  type: seq

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   SEQ_API_KEY: <Base64 API Key>
# kind: Secret
# metadata:
#   name: seq-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f seq.yaml
```


# SignalFx
Source: https://docs.odigos.io/enterprise/backends/signalfx

Configuring the SignalFx backend (Managed)

### Getting Started

![signalfx](https://d15jtxgb40qetw.cloudfront.net/signalfx.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **SIGNALFX\_ACCESS\_TOKEN** `string` : Access Token. The authentication token provided by Splunk Observability Cloud.
  * This field is required
* **SIGNALFX\_REALM** `string` : Realm. The SignalFx realm where your data will be sent (e.g., us0, us1, eu0).
  * This field is required
* **SIGNALFX\_CA\_PEM** `string` : CA Certificate (PEM). Optional CA certificate in PEM format for TLS verification.
  * This field is optional
* **SIGNALFX\_INSECURE\_SKIP\_VERIFY** `boolean` : Skip TLS Verification. If enabled, TLS certificate verification will be skipped. Not recommended for production.
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `SignalFx` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `signalfx.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: signalfx-example
  namespace: odigos-system
spec:
  data:
    SIGNALFX_REALM: <Realm>
    # Note: The commented fields below are optional.
    # SIGNALFX_INSECURE_SKIP_VERIFY: <Skip TLS Verification>
  destinationName: signalfx
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: signalfx-secret
  signals:
  - TRACES
  - METRICS
  type: signalfx

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   SIGNALFX_ACCESS_TOKEN: <Base64 Access Token>
#   SIGNALFX_CA_PEM: <Base64 CA Certificate (PEM)>
# kind: Secret
# metadata:
#   name: signalfx-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f signalfx.yaml
```


# SigNoz
Source: https://docs.odigos.io/enterprise/backends/signoz

Configuring the SigNoz backend (Self-Hosted)

### Getting Started

![signoz](https://d15jtxgb40qetw.cloudfront.net/signoz.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **SIGNOZ\_URL** `string` : Collector Endpoint.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `SigNoz` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `signoz.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: signoz-example
  namespace: odigos-system
spec:
  data:
    SIGNOZ_URL: <Collector Endpoint>
  destinationName: signoz
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: signoz
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f signoz.yaml
```


# Splunk (Legacy) (Deprecated)
Source: https://docs.odigos.io/enterprise/backends/splunk

Configuring the Splunk (Legacy) (Deprecated) backend (Managed)

### Getting Started

![splunk](https://d15jtxgb40qetw.cloudfront.net/splunk.svg)

**Obtaining Access Token**\

Go to **Settings** then **Access Tokens** and click **Create New Token**

![Splunk Access Tokens menu](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/splunk1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=3a14b36ce4e63b2b7de2ffb6c679a36d)

Give the token a name and select **Ingest Token** for the authorization scope and click create.

![Create Access Token form in Splunk](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/splunk2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=aca44406c34343662a6c0c0654a2e47c)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required
* **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`.
  * This field is required

> [!NOTE]
>
> This legacy destination now sends traces using OTLP/HTTP because the SAPM exporter was removed upstream.
> Existing `splunk` destinations retain their realm and access token. For new destinations, use [Splunk (OTLP)](/backends/splunkotlp).

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Splunk (Legacy) (Deprecated)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `splunk.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: splunk-example
  namespace: odigos-system
spec:
  data:
    SPLUNK_REALM: <Realm>
  destinationName: splunk
  secretRef:
    name: splunk-secret
  signals:
  - TRACES
  type: splunk

---

apiVersion: v1
data:
  SPLUNK_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: splunk-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f splunk.yaml
```


# Splunk (OTLP)
Source: https://docs.odigos.io/enterprise/backends/splunkotlp

Configuring the Splunk (OTLP) backend (Managed)

### Getting Started

![splunkotlp](https://d15jtxgb40qetw.cloudfront.net/splunk.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required
* **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`.
  * This field is required
* **SPLUNK\_OTLP\_COMPRESSION** `string` : Compression Type. Compression type to use for the destination. The default is `none`.
  * This field is optional and defaults to `none`
* **SPLUNK\_OTLP\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **SPLUNK\_OTLP\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **SPLUNK\_OTLP\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Splunk (OTLP)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `splunkotlp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: splunkotlp-example
  namespace: odigos-system
spec:
  data:
    SPLUNK_REALM: <Realm>
    # Note: The commented fields below are optional.
    # SPLUNK_OTLP_COMPRESSION: <Compression Type (default: none) (options: [none, gzip])>
    # SPLUNK_OTLP_TLS_ENABLED: <Enable TLS>
    # SPLUNK_OTLP_CA_PEM: <Certificate Authority>
    # SPLUNK_OTLP_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
  destinationName: splunkotlp
  secretRef:
    name: splunkotlp-secret
  signals:
  - TRACES
  type: splunkotlp

---

apiVersion: v1
data:
  SPLUNK_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: splunkotlp-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f splunkotlp.yaml
```


# Splunk (SAPM) (Deprecated)
Source: https://docs.odigos.io/enterprise/backends/splunksapm

Configuring the Splunk (SAPM) (Deprecated) backend (Managed)

### Getting Started

![splunksapm](https://d15jtxgb40qetw.cloudfront.net/splunk.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs

* **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required
* **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](../cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Splunk (SAPM) (Deprecated)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `splunksapm.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: splunksapm-example
  namespace: odigos-system
spec:
  data:
    SPLUNK_REALM: <Realm>
  destinationName: splunksapm
  secretRef:
    name: splunksapm-secret
  signals:
  - TRACES
  type: splunksapm

---

apiVersion: v1
data:
  SPLUNK_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: splunksapm-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f splunksapm.yaml
```


# Sumo Logic
Source: https://docs.odigos.io/enterprise/backends/sumologic

Configuring the Sumo Logic backend (Managed)

### Getting Started

![sumologic](https://d15jtxgb40qetw.cloudfront.net/sumologic.svg)

Follow the [following instructions](https://help.sumologic.com/docs/send-data/hosted-collectors/http-source/otlp/) to obtain a OTLP/HTTP Source URL.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **SUMOLOGIC\_COLLECTION\_URL** `string` : Source URL.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Sumo Logic` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `sumologic.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: sumologic-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: sumologic
  secretRef:
    name: sumologic-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: sumologic

---

apiVersion: v1
data:
  SUMOLOGIC_COLLECTION_URL: <Base64 Source URL>
kind: Secret
metadata:
  name: sumologic-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f sumologic.yaml
```


# TelemetryHub
Source: https://docs.odigos.io/enterprise/backends/telemetryhub

Configuring the TelemetryHub backend (Managed)

### Getting Started

![telemetryhub](https://d15jtxgb40qetw.cloudfront.net/telemetryhub.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **TELEMETRY\_HUB\_API\_KEY** `string` : API Key.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `https://otlp.telemetryhub.com:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `TelemetryHub` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `telemetryhub.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: telemetryhub-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: telemetryhub
  secretRef:
    name: telemetryhub-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: telemetryhub

---

apiVersion: v1
data:
  TELEMETRY_HUB_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: telemetryhub-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f telemetryhub.yaml
```


# Tempo
Source: https://docs.odigos.io/enterprise/backends/tempo

Configuring the Tempo backend (Self-Hosted)

### Getting Started

![tempo](https://d15jtxgb40qetw.cloudfront.net/tempo.svg)

> [!NOTE]
>
> These instructions are for the **self hosted** Tempo instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Tempo](../backends/grafanacloudtempo) page.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **TEMPO\_URL** `string` : Endpoint.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Tempo` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `tempo.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: tempo-example
  namespace: odigos-system
spec:
  data:
    TEMPO_URL: <Endpoint>
  destinationName: tempo
  signals:
  - TRACES
  type: tempo
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f tempo.yaml
```


# Tingyun 基调听云
Source: https://docs.odigos.io/enterprise/backends/tingyun

Configuring the Tingyun 基调听云 backend (Managed)

### Getting Started

![tingyun](https://d15jtxgb40qetw.cloudfront.net/tingyun.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **TINGYUN\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
* **TINGYUN\_LICENSE\_KEY** `string` : License Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Tingyun 基调听云` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `tingyun.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: tingyun-example
  namespace: odigos-system
spec:
  data:
    TINGYUN_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: tingyun
  secretRef:
    name: tingyun-secret
  signals:
  - TRACES
  - METRICS
  type: tingyun

---

apiVersion: v1
data:
  TINGYUN_LICENSE_KEY: <Base64 License Key>
kind: Secret
metadata:
  name: tingyun-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f tingyun.yaml
```


# Traceloop
Source: https://docs.odigos.io/enterprise/backends/traceloop

Configuring the Traceloop backend (Managed)

### Getting Started

![traceloop](https://d15jtxgb40qetw.cloudfront.net/traceloop.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **TRACELOOP\_ENDPOINT** `string` : Traceloop OTLP HTTP Endpoint.
  * This field is required and defaults to `https://api.traceloop.com`
* **TRACELOOP\_API\_KEY** `string` : Traceloop API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Traceloop` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `traceloop.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: traceloop-example
  namespace: odigos-system
spec:
  data:
    TRACELOOP_ENDPOINT: '<Traceloop OTLP HTTP Endpoint (default: https://api.traceloop.com)>'
  destinationName: traceloop
  secretRef:
    name: traceloop-secret
  signals:
  - TRACES
  - METRICS
  type: traceloop

---

apiVersion: v1
data:
  TRACELOOP_API_KEY: <Base64 Traceloop API Key>
kind: Secret
metadata:
  name: traceloop-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f traceloop.yaml
```


# Uptrace
Source: https://docs.odigos.io/enterprise/backends/uptrace

Configuring the Uptrace backend (Managed)

### Getting Started

![uptrace](https://d15jtxgb40qetw.cloudfront.net/uptrace.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **UPTRACE\_DSN** `string` : Data Source Name (DSN).
  * This field is required
* **UPTRACE\_ENDPOINT** `string` : Endpoint. Overwrite Uptrace endpoint when self-hosting
  * This field is optional and defaults to `https://otlp.uptrace.dev:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Uptrace` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `uptrace.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: uptrace-example
  namespace: odigos-system
spec:
  data:
    UPTRACE_DSN: <Data Source Name (DSN)>
    # Note: The commented fields below are optional.
    # UPTRACE_ENDPOINT: <Endpoint (default: https://otlp.uptrace.dev:4317)>
  destinationName: uptrace
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: uptrace
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f uptrace.yaml
```


# VictoriaMetrics Cloud
Source: https://docs.odigos.io/enterprise/backends/victoriametricscloud

Configuring the VictoriaMetrics Cloud backend (Managed)

### Getting Started

![victoriametricscloud](https://d15jtxgb40qetw.cloudfront.net/victoriametrics.svg)

**Creating Account**\

Go to the **[🔗 Cloud Console](https://console.victoriametrics.cloud) > Sign Up**

**Setting Up**\

Go to **⚙️ > Deployments** and click **Start using VictoriaMetrics Cloud**

**Obtaining Access Token**\

Go to **⚙️ > Deployments** and click **Overview**\

Then click **Access Tokens** and copy one of the keys

**Obtaining Endpoint**\

Go to **⚙️ > Integrations** and click **OpenTelemetry > Operator**\

Scroll to **Kubernetes commandline command** and copy the **endpoint**

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **VICTORIA\_METRICS\_CLOUD\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
* **VICTORIA\_METRICS\_CLOUD\_TOKEN** `string` : Access Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `VictoriaMetrics Cloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `victoriametricscloud.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: victoriametricscloud-example
  namespace: odigos-system
spec:
  data:
    VICTORIA_METRICS_CLOUD_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: victoriametricscloud
  secretRef:
    name: victoriametricscloud-secret
  signals:
  - METRICS
  type: victoriametricscloud

---

apiVersion: v1
data:
  VICTORIA_METRICS_CLOUD_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: victoriametricscloud-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f victoriametricscloud.yaml
```


# odigos
Source: https://docs.odigos.io/enterprise/cli/odigos



## odigos

Automate OpenTelemetry Observability in Kubernetes

### Synopsis

Odigos is primarily focused on automating OpenTelemetry observability pipelines for traces, metrics, and logs, without the need for extensive code changes. The core of Odigos functionality lies in the Kubernetes operators it deploys within your cluster, enabling seamless observability.

Key Features of Odigos:

* Automatic creation of OpenTelemetry observability pipelines.
* Simplified tracing, metrics, and log collection.
* Enhanced visibility into your Kubernetes services.
* Streamlined Kubernetes operations with observability at the forefront.

Get started with Odigos today to effortlessly improve the observability of your Kubernetes services!

### Options

```
  -h, --help                  help for odigos
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos cleanup](/cli/odigos_cleanup)	 - Remove Odigos Sources created by the user.
* [odigos describe](/cli/odigos_describe)	 - Show details of a specific odigos entity
* [odigos diagnose](/cli/odigos_diagnose)	 - Diagnose Client Cluster
* [odigos install](/cli/odigos_install)	 - Install and upgrade Odigos
* [odigos pro](/cli/odigos_pro)	 - Manage Odigos onprem tier for enterprise users
* [odigos profile](/cli/odigos_profile)	 - Manage presets of applied profiles to your odigos installation
* [odigos sources](/cli/odigos_sources)	 - Manage Odigos Sources in a cluster
* [odigos ui](/cli/odigos_ui)	 - Start the Odigos UI
* [odigos uninstall](/cli/odigos_uninstall)	 - Uninstall Odigos
* [odigos uninstall-deprecated](/cli/odigos_uninstall-deprecated)	 - Revert all the changes made by the `odigos install` command.
  This command will uninstall Odigos from your cluster. It will delete all Odigos objects
  and rollback any metadata changes made to your objects.
* [odigos upgrade](/cli/odigos_upgrade)	 - Upgrade Odigos
* [odigos version](/cli/odigos_version)	 - Print odigos version.


# odigos describe
Source: https://docs.odigos.io/enterprise/cli/odigos_describe



## odigos describe

Show details of a specific odigos entity

### Synopsis

This command can be used for troubleshooting and observing odigos state for its various entities.\
It is similar to `kubectl describe` but for Odigos entities.

```
odigos describe [flags]
```

### Examples

```

# Describe a source of kind deployment and name myservice in the default namespace
odigos describe source deployment myservice -n default

# Describe a source of kind deploymentconfig and name myservice in the default namespace (OpenShift)
odigos describe source deploymentconfig myservice -n default

Output:

Name:  myservice
Kind:  deployment
Namespace:  default

Labels:
  Instrumented:  true
  Workload:  instrumented
  Namespace: unset
  Decision: Workload is instrumented because there is an enabled Source for this workload
  Troubleshooting: https://docs.odigos.io/architecture/troubleshooting#1-source-configuration

Instrumentation Config:
  Created at 2024-07-30 19:00:40 +0300 IDT

Runtime inspection details:
  Created at 2024-07-30 19:00:40 +0300 IDT
  Detected Containers:
    - Container Name: myservice
      Language:       javascript
      Relevant Environment Variables:
        - NODE_OPTIONS : --require /var/odigos/nodejs/autoinstrumentation.js

Instrumentation Device:
  Status: Successfully applied instrumentation device to pod template
  - Container Name: myservice
    Instrumentation Devices: javascript-native-community

Pods (Total 1, Running 1):
  Pod Name: myservice-ffd68d8c-qqmxl
  Pod Phase: Running
  Pod Node Name: kind-control-plane
  Containers:
  - Container Name: myservice
    Instrumentation Devices: javascript-native-community
    Instrumentation Instances:
    - Healthy: true
      Reason: Healthy

```

### Options

```
  -h, --help     help for describe
  -r, --remote   use odigos ui service in the cluster to describe the entity
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes
* [odigos describe source](/cli/odigos_describe_source)	 - Show details of a specific odigos source


# odigos diagnose
Source: https://docs.odigos.io/enterprise/cli/odigos_diagnose



## odigos diagnose

Diagnose Client Cluster

### Synopsis

Retrieves Logs of all Odigos components in the odigos-system namespace and CRDs of Actions, instrumentation resources and more.
The results will be saved in a compressed file for further troubleshooting.
The file will be saved in this format: odigos\_debug\_ddmmyyyyhhmmss.tar.gz

```
odigos diagnose [flags]
```

### Options

```
      --all-workload-namespaces              Collect instrumented source workloads from all namespaces
  -h, --help                                 help for diagnose
      --odigos-configmaps                    Include Odigos ConfigMaps in the diagnose output (default true)
      --odigos-crds                          Include Odigos CRDs in the diagnose output (default true)
      --odigos-logs                          Include Odigos component pod logs in the diagnose output (default true)
      --odigos-metrics                       Include Odigos Prometheus metrics in the diagnose output (default true)
      --odigos-profiles                      Include Odigos pprof profiles in the diagnose output (default true)
      --source-workload-namespaces strings   Collect instrumented source workloads from specific namespaces (comma-separated)
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos install
Source: https://docs.odigos.io/enterprise/cli/odigos_install



## odigos install

Install and upgrade Odigos

### Synopsis

This subcommand installs or upgrades Odigos in your Kubernetes cluster.
It installs Kubernetes components that auto-instrument your applications with OpenTelemetry
and send traces, metrics, and logs to any telemetry backend.

```
odigos install [flags]
```

### Examples

```

# Install or upgrade Odigos open-source in your cluster with the default values
odigos install

# Install or upgrade Odigos on-prem tier for enterprise users
odigos install --set onPremToken=${ODIGOS_TOKEN}

# Install or upgrade Odigos and set specific values
odigos install --set collectorGateway.minReplicas=5 --set collectorGateway.maxReplicas=10

# Install or upgrade Odigos using a local values.yaml file
odigos install --values ./values.yaml

# Reset all values to chart defaults (opt out of reuse)
odigos install --reset-then-reuse-values=false

```

### Options

```
      --chart string              Helm chart to install (repo/name, local path, or URL) (default "odigos/odigos")
      --chart-version string      Override Helm chart version (defaults to CLI-baked version)
  -h, --help                      help for install
      --ns string                 Target Kubernetes namespace (default "odigos-system")
      --release-name string       Helm release name (default "odigos")
      --reset-then-reuse-values   Reset to chart defaults, then reuse values from the previous release (default: true). (default true)
      --set strings               Set values on the command line (key=value)
  -f, --values string             Path to a custom values.yaml file
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos pro
Source: https://docs.odigos.io/enterprise/cli/odigos_pro



## odigos pro

Manage Odigos onprem tier for enterprise users

### Synopsis

The pro command provides various operations and functionalities specifically designed for enterprise users. Use this command to access advanced features and manage your pro account.

```
odigos pro [flags]
```

### Examples

```

# Renew the on-premises token for Odigos,
odigos pro --onprem-token <token>



```

### Options

```
  -h, --help   help for pro
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](../cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes
* [odigos pro central](../cli/odigos_pro_central)	 - Manage Odigos Central (Enterprise tier)
* [odigos pro update-offsets](../cli/odigos_pro_update-offsets)	 - Update Odiglet to use the latest available Go instrumentation offsets


# odigos profile
Source: https://docs.odigos.io/enterprise/cli/odigos_profile



## odigos profile

Manage presets of applied profiles to your odigos installation

### Synopsis

This command can be used to interact with the applied profiles in your odigos installation.

```
odigos profile [flags]
```

### Examples

```

# Enable payload collection for all supported workloads and instrumentation libraries in the cluster
odigos profile add full-payload-collection

# Remove the full-payload-collection profile from the cluster
odigos profile remove full-payload-collection

```

### Options

```
  -a, --available   list all available profiles to use
  -h, --help        help for profile
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos sources
Source: https://docs.odigos.io/enterprise/cli/odigos_sources



## odigos sources

Manage Odigos Sources in a cluster

### Synopsis

This command can be used to create, delete, or update Sources to configure workload or namespace auto-instrumentation

### Examples

```
# Create a Source "foo-source" for deployment "foo" in namespace "default"
odigos sources create foo-source --workload-kind=Deployment --workload-name=foo --workload-namespace=default -n default

# Update all existing Sources in namespace "default" to disable instrumentation
odigos sources update --disable-instrumentation -n default

# Delete all Sources in group "mygroup"
odigos sources delete --group mygroup --all-namespaces
	
```

### Options

```
  -h, --help   help for sources
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes
* [odigos sources create](/cli/odigos_sources_create)	 - Create an Odigos Source
* [odigos sources delete](/cli/odigos_sources_delete)	 - Delete Odigos Sources
* [odigos sources disable](/cli/odigos_sources_disable)	 - Disable a source for Odigos instrumentation.
* [odigos sources enable](/cli/odigos_sources_enable)	 - Enable a source for Odigos instrumentation.
* [odigos sources status](/cli/odigos_sources_status)	 - Show the status of all Odigos Sources
* [odigos sources update](/cli/odigos_sources_update)	 - Update Odigos Sources


# odigos ui
Source: https://docs.odigos.io/enterprise/cli/odigos_ui



## odigos ui

Start the Odigos UI

### Synopsis

Start the Odigos UI. This will start a web server that you can access in your browser and enables you to manage and configure Odigos.

```
odigos ui [flags]
```

### Examples

```

# Start the Odigos UI on http://localhost:3000
odigos ui

# Start the Odigos UI on specific port if 3000 is already in use
odigos ui --port 3456

# Start the Odigos UI and have it manage and configure a specific cluster
odigos ui --kubeconfig <path-to-kubeconfig>

```

### Options

```
      --address string    Address to serve the UI on (default "localhost")
  -h, --help              help for ui
      --port int          Port to listen on (default 3000)
      --remote-port int   Port to forward to the remote UI (default 3000)
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos uninstall
Source: https://docs.odigos.io/enterprise/cli/odigos_uninstall



## odigos uninstall

Uninstall Odigos

### Synopsis

Revert all the changes made by the odigos install command. This command will uninstall Odigos from your cluster. It will delete all Odigos objects and rollback any metadata changes made to your objects.

```
odigos uninstall [flags]
```

### Examples

```

# Uninstall Odigos
odigos uninstall

```

### Options

```
  -h, --help                  help for uninstall
      --ns string             Target Kubernetes namespace (default "odigos-system")
      --release-name string   Helm release name (default "odigos")
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos upgrade
Source: https://docs.odigos.io/enterprise/cli/odigos_upgrade



## odigos upgrade

Upgrade Odigos

### Synopsis

Upgrades (or installs) Odigos in your Kubernetes cluster.
This command behaves identically to 'odigos install' and uses the same Helm-based flow.

```
odigos upgrade [flags]
```

### Examples

```

# Upgrade Odigos
odigos upgrade

# Upgrade Odigos with custom values
odigos upgrade --set collectorGateway.maxReplicas=10

# Reset all values to chart defaults (opt out of reuse)
odigos upgrade --reset-then-reuse-values=false

```

### Options

```
      --chart string              Helm chart to install (repo/name, local path, or URL) (default "odigos/odigos")
      --chart-version string      Override Helm chart version (defaults to CLI-baked version)
  -h, --help                      help for upgrade
      --ns string                 Target Kubernetes namespace (default "odigos-system")
      --release-name string       Helm release name (default "odigos")
      --reset-then-reuse-values   Reset to chart defaults, then reuse values from the previous release (default: true). (default true)
      --set strings               Set values on the command line (key=value)
  -f, --values string             Path to a custom values.yaml file
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos version
Source: https://docs.odigos.io/enterprise/cli/odigos_version



## odigos version

Print odigos version.

### Synopsis

This command is used to print the Odigos version.
Both the CLI version and the Odigos components in your cluster will be printed.

```
odigos version [flags]
```

### Examples

```

# Print the version of odigos CLI and Odigos deployment in your cluster
odigos version

```

### Options

```
      --cli       prints only the CLI version
      --cluster   prints only the version of odigos deployed in the cluster
  -h, --help      help for version
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# Custom Resources
Source: https://docs.odigos.io/enterprise/custom-resources



The different components of the Odigos observability control plane work together
to achieve observability for the cluster. The interaction between the components
is performed via the Kubernetes API server.

The instrumentor is a
[Kubernetes Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/).
It coordinates by writing and watching the odigos custom resources in the
Kubernetes API server.

Odigos adds the following custom resources to the Kubernetes API server:

## Source

This resource defines namespaces and workloads to instrument. It has the following `spec` fields:

| Field | Description | Type | Default | Required |
| - | - | - | - | - |
| `disableInstrumentation` | Excludes the specified workload from instrumentation. | `bool` | `false` | Optional |
| `workload.name` | Name of the workload for this Source. | `string` | n/a | Required |
| `workload.namespace` | Namespace of the workload for this Source. | `string` | n/a | Required |
| `workload.kind` | Kind of the workload for this Source. | `WorkloadKind` | n/a | Required |

Valid values for `WorkloadKind` are:

* `Deployment`
* `DaemonSet`
* `StatefulSet`
* `Namespace`

These values are case-sensitive.

## Destination

This custom resource is used to define the backend destinations for the
observability data. Destinations can be either vendors (Datadog, Honeycomb, etc)
or on premise.

The destination object holds any data that is needed to connect to the backend
system. Notice that sensitive fields such as API keys are stored in a Kubernetes
secret and referenced by the destination object.

## CollectorsGroup

The CollectorsGroup object is used to define a group of collectors that have a
shared role. For example, a collectors group might include collectors that are
deployed as DaemonSet that are responsible for collecting telemetry data, or
collectors that are deployed as Deployment and are responsible for shipping the
data to the selected destinations.


# Debugging Odigos
Source: https://docs.odigos.io/enterprise/debugging



Different components of Odigos can be debugged in different ways. This page describes how to debug each component.

## Odiglet

First make sure you have a running installation of Odigos on a local Kind cluster.
From the root of the Odigos repository, run:

```bash theme={null}
TAG=<your-odigos-version> make debug-odiglet
```

This command will replace the running Odiglet binary and run it in a container on the Kind cluster.

> [!NOTE]
>
> The new Odiglet will not start until a debug session is attached to it

Now all that left is to attach a remote debugger (using VSCode / GoLand) to `localhost:2345`


# OIDC (OpenID Connect)
Source: https://docs.odigos.io/enterprise/features/oidc

OIDC is a standard for authentication and authorization that allows you to authenticate users with an external identity provider.

> [!NOTE]
>
> This feature is available in Odigos
> [`v1.0.201`](https://github.com/odigos-io/odigos/releases/tag/v1.0.201) and
> later.

## Getting Started

To get started with OIDC, you need to configure the Odigos UI to use an external identity provider.

**1. Create a new OIDC application in your identity provider**

> [!NOTE]
>
> You can use the [Okta](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm) or [Auth0](https://auth0.com/docs/authenticate/identity-providers/enterprise-identity-providers/oidc) guides to create a new OIDC application.

> [!WARNING]
>
> Please note that the OIDC application should be configured to allow the Odigos UI to access the `Login redirect URLs`.
>
> * If you are using the default Odigos UI, the redirect URL should be `http://localhost:3000/auth/oidc-callback`.
> * If you are using a custom/remote UI, the redirect URL should be the URL of the Odigos UI, with the path `/auth/oidc-callback`.\\
>
>   For example, if you are using a remote UI at `https://my-odigos-ui.com`, the redirect URL should be `https://my-odigos-ui.com/auth/oidc-callback`.

**2. Set OIDC variables**

You should set the OIDC values during installation or upgrade. The values can be obtained from the identity provider, after creating the OIDC application.

**CLI**

You can use the `odigos install --set` command.

```bash theme={null}
odigos install --set ui.oidcTenantUrl=<value> --set ui.oidcClientId=<value> --set ui.oidcClientSecret=<value>
```

**Helm**

You can set these values in your `values.yaml` file for Helm installations:

```yaml theme={null}
ui:
  oidcTenantUrl: <value>
  oidcClientId: <value>
  oidcClientSecret: <value>
```

Alternatively, you can use the `--set` flag when installing/upgrading the Odigos Helm chart:

```bash theme={null}
helm repo update
helm upgrade odigos odigos/odigos --namespace odigos-system --set ui.oidcTenantUrl=<value> --set ui.oidcClientId=<value> --set ui.oidcClientSecret=<value>
```

**3. Set UI remote URL (optional)**

If you are hosting the Odigos UI on a custom/remote URL (e.g. `https://my-odigos-ui.com`), you should set the value during installation or upgrade.

**CLI**

You can use the `odigos install --set` command.

```bash theme={null}
odigos install --set ui.uiRemoteUrl=<value>
```

**Helm**

You can set this value in your `values.yaml` file for Helm installations:

```yaml theme={null}
ui:
  uiRemoteUrl: <value>
```

Alternatively, you can use the `--set` flag when installing/upgrading the Odigos Helm chart:

```bash theme={null}
helm repo update
helm upgrade odigos odigos/odigos --namespace odigos-system --set ui.uiRemoteUrl=<value>
```

## Authentication behavior

When OIDC is configured, both the UI pages and the GraphQL, REST, and event APIs
require authentication. Browsers use the `id_token` cookie set by the login
callback. API clients can send an OIDC **ID token** issued for the configured
client as `Authorization: Bearer <id_token>`; an OAuth access token or the
Odigos on-prem license token is not a substitute. Unauthenticated API requests
return HTTP `401`, while unauthenticated UI page requests redirect to the IdP.
Health probes, `/auth/csrf-token`, and `/auth/oidc-callback` remain public.

For `odigos describe --remote`, remote source operations, and remote token
updates on OIDC-enabled installations, set `ODIGOS_UI_ID_TOKEN` to a valid ID
token in the CLI's environment. Remote token updates forward it as a cookie
through the Kubernetes service proxy so Kubernetes authentication remains intact.

An `oidc` configuration block enables authentication. Supply the tenant URL,
client ID, and a nonempty `odigos-oidc` secret. Incomplete settings, missing
credentials, and unavailable configuration or IdP discovery return HTTP `503`;
they do not silently disable authentication. To disable OIDC intentionally,
remove all three `ui.oidc*` values so the `oidc` block is absent.

Authentication work is canceled when the client disconnects and has a ten-second
limit. Each request uses one configuration snapshot for its OAuth settings and
token verifier.


# Istio Service Mesh Tracing
Source: https://docs.odigos.io/enterprise/instrumentations/advanced/Istio



Odigos can collect the distributed traces emitted by the [Istio](https://istio.io/) service mesh and route them through the same pipeline as your application traces. Once enabled, the Envoy sidecars report a span for every meshed hop;connection-level latency, retries, response codes, and mTLS — over OTLP to the Odigos collector, where they are correlated with the spans Odigos already generates for your applications. The result is a single trace that spans both the network and application layers.

> [!NOTE]
>
> This page covers how to point an **existing** Istio mesh at Odigos. It is not a guide to installing or operating Istio. For mesh setup, see the [Istio installation guide](https://istio.io/latest/docs/setup/install/helm/).

## Prerequisites

**1. Kubernetes 1.26 or newer**

**Kubernetes 1.26+ is required** — the `odigos-data-collection-local-traffic` service depends on `internalTrafficPolicy: Local`, which became GA in 1.26.

**2. A running Istio mesh**

Istio's control plane (`istiod`) must be installed in your cluster. Follow the [Istio installation guide](https://istio.io/latest/docs/setup/install/helm/).

**3. Workloads enrolled in the mesh**

Istio only emits spans for workloads that have an Envoy sidecar. Label the namespaces you want traced and restart their workloads so the sidecar is injected:

```bash theme={null}
kubectl label namespace <namespace> istio-injection=enabled
kubectl rollout restart deployment -n <namespace>
```

For the full set of injection options, see Istio's [sidecar injection](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/) documentation.

## Configuration

**1. Register Odigos as an OpenTelemetry provider**

Add an OpenTelemetry extension provider to Istio's `MeshConfig` that points at the Odigos node collector's OTLP/HTTP endpoint. For background on this field, see Istio's reference for [enabling OpenTelemetry tracing](https://istio.io/latest/docs/tasks/observability/distributed-tracing/opentelemetry/).

Apply it using whichever method you use to manage your mesh:

**Helm**

Add the `meshConfig` block to the values for the `istiod` chart (for example, `istiod-values.yaml`):

```yaml istiod-values.yaml theme={null}
meshConfig:
  enableTracing: true
  extensionProviders:
    - name: odigos-otel
      opentelemetry:
        service: odigos-data-collection-local-traffic.odigos-system.svc.cluster.local
        port: 4318
        http:
          path: "/v1/traces"
          timeout: 5s
        resource_detectors:
          environment: {}
```

```bash theme={null}
helm upgrade istiod istio/istiod -n istio-system --reuse-values -f istiod-values.yaml
```

**istioctl**

Define the `meshConfig` block in an `IstioOperator` configuration file (for example, `istio-otel.yaml`):

```yaml istio-otel.yaml theme={null}
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    extensionProviders:
      - name: odigos-otel
        opentelemetry:
          service: odigos-data-collection-local-traffic.odigos-system.svc.cluster.local
          port: 4318
          http:
            path: "/v1/traces"
            timeout: 5s
          resource_detectors:
            environment: {}
```

```bash theme={null}
istioctl install -f istio-otel.yaml
```

Defining the provider on its own does not start any tracing — it only tells Istio where Odigos is.

> [!NOTE]
>
> Requires **Kubernetes 1.26+**. Below that, `odigos-data-collection-local-traffic` does not exist and Istio has no endpoint to reach.

**2. Enable tracing with the Telemetry API**

Create a `Telemetry` resource in the Istio root namespace (usually `istio-system`) that references the provider. This is what actually turns on span reporting. See the [Istio Telemetry API](https://istio.io/latest/docs/tasks/observability/telemetry/) for scoping options.

```yaml theme={null}
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: odigos-otel
      randomSamplingPercentage: 100
```

> [!TIP]
>
> Leave `randomSamplingPercentage` at `100` so every meshed hop is reported. If you need to reduce volume, do it with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) rather than at the Envoy layer — that way the decision sees the full trace (errors, latency, and downstream spans) instead of being made blindly per request at the sidecar.

**3. Confirm a destination on the default data stream**

The Odigos collector only exports data when at least one destination is configured. Because Envoy emits Istio spans on behalf of the mesh rather than any specific Odigos-instrumented workload, the collector routes them through the **default data stream** — so make sure the destination you want the Istio traces to reach is assigned to the default data stream.

> [!NOTE]
>
> Istio produces spans only for L7 HTTP traffic on Service ports named `http` / `http-*` (or with `appProtocol: http`). Plain TCP protocols such as Redis or AMQP will not generate mesh spans.

## Validation

After traffic flows through a meshed workload, look for Envoy spans in your tracing backend. They appear under service names in the form `<workload>.<namespace>` (for example, `frontend.default`), distinct from the application service names Odigos reports. You can identify them by:

* `otel.scope.name` = `envoy`
* `component` = `proxy`
* `istio.*` resource attributes such as `istio.canonical_service` and `istio.mesh_id`

### Example

Real meshed traces interleave Istio and Odigos spans — every meshed hop adds a pair of Istio sidecar spans (one on each side of the wire) that wrap the callee's application work, which itself can fan out into more application spans and another mesh hop. A two-hop call through `frontend → checkout → payments` lays out like this:

Horizontal position is wall-clock time; label indentation follows the true parent/child nesting. A child bar always starts after and ends within its parent.

```
                                0ms      50       100      150      200ms
                                 ├────────┼────────┼────────┼────────┤
  frontend · POST /checkout       ████████████████████████████████████
    frontend · Istio→checkout          ┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
      checkout · Istio server           ┏━━━━━━━━━━━━━━━━━━━┓
        checkout · POST /checkout        ██████████████████████
          checkout · SELECT cart         ██████
          checkout · Istio→payments              ┏━━━━━━━━━┓
            payments · Istio server               ┏━━━━━━━┓
              payments · POST /charge             █████████
                payments · INSERT txn              ██████

  Legend:  █ Odigos (app)   ┏━┓ Istio (mesh)
```

Each Istio client/server pair brackets the next service's application work. The gap between the two Istio spans is **on-the-wire time** (DNS, TCP, mTLS handshakes, retries); the gap between the Istio server span and the application span is **proxy overhead**. With each additional meshed hop, another sidecar pair is added — both costs become directly attributable instead of being absorbed into a single application-latency number.

> [!NOTE]
>
> Istio spans are additive — they do not replace the application spans Odigos generates. Each meshed HTTP hop yields an Istio client span, an Istio server span, and the application span(s), all sharing the same trace.


# NGINX Ingress Tracing
Source: https://docs.odigos.io/enterprise/instrumentations/advanced/Nginx



Odigos can collect the distributed traces emitted by the [Ingress-NGINX Controller](https://kubernetes.github.io/ingress-nginx/) and route them through the same pipeline as your application traces. Once enabled, the controller reports a span for every request that enters the cluster through it — route, upstream, response code, and latency — over OTLP to the Odigos collector, where they are correlated with the spans Odigos already generates for your applications. The result is a single trace that begins at the edge and follows the request all the way through your services.

> [!WARNING]
>
> The community Ingress-NGINX project has been **retired** as of March 2026 — it no longer receives releases, bug fixes, or security patches, so running it carries growing security risk. See the [Kubernetes retirement announcement](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/). For new deployments, migrate to the [Gateway API](https://gateway-api.sigs.k8s.io/) instead.

> [!NOTE]
>
> This page covers how to point an **existing** Ingress-NGINX deployment at Odigos. It is not a guide to installing or operating the controller. For controller setup, see the [Ingress-NGINX installation guide](https://kubernetes.github.io/ingress-nginx/deploy/).

> [!NOTE]
>
> Tracing happens at the **Ingress-NGINX Controller**, not in your application's nginx pods. Stock `nginx` images lack the OpenTelemetry module, but the controller bundles it and turns a few ConfigMap keys into the right OTel directives. Put the workloads you want traced behind an Ingress and enable OTel on the controller.

## Prerequisites

**1. Kubernetes 1.26 or newer**

The `odigos-data-collection-local-traffic` service depends on `internalTrafficPolicy: Local`, which became GA in 1.26.

**2. A running Ingress-NGINX Controller**

The Ingress-NGINX Controller must be installed in your cluster, with the OpenTelemetry module available. Follow the [Ingress-NGINX installation guide](https://kubernetes.github.io/ingress-nginx/deploy/).

**3. Workloads exposed through an Ingress**

The controller only emits spans for traffic that flows through it. Make sure the workloads you want traced are fronted by an `Ingress` resource that routes to their `Service`. For background on routing, see the [Ingress-NGINX user guide](https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/).

## Configuration

**1. Point the controller at the Odigos OTLP endpoint**

Configure the Ingress-NGINX Controller to export OpenTelemetry traces to the Odigos node collector's OTLP/gRPC endpoint. For background on these keys, see the controller's [OpenTelemetry documentation](https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/).

Choose one of the methods below based on how you manage the controller. If you installed it with Helm, use Helm values so a later `helm upgrade` does not overwrite the settings; otherwise apply the ConfigMap directly.

**Helm**

Enable the OTel module and set the exporter through the `ingress-nginx` chart values:

```bash theme={null}
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.opentelemetry.enabled=true \
  --set controller.config.enable-opentelemetry="true" \
  --set controller.config.otlp-collector-host="odigos-data-collection-local-traffic.odigos-system.svc.cluster.local" \
  --set controller.config.otlp-collector-port="4317" \
  --set controller.config.otel-service-name="nginx-ingress" \
  --set controller.config.opentelemetry-trust-incoming-span="true" \
  --set controller.config.opentelemetry-operation-name="HTTP \$request_method \$uri"
```

**ConfigMap**

Add the exporter keys to the controller's ConfigMap (named `ingress-nginx-controller` in the `ingress-nginx` namespace by default — match whatever your controller reads via its `--configmap` flag):

```yaml ingress-nginx-controller-cm.yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-opentelemetry: "true"
  otlp-collector-host: "odigos-data-collection-local-traffic.odigos-system.svc.cluster.local"
  otlp-collector-port: "4317"
  otel-service-name: "nginx-ingress"
  opentelemetry-trust-incoming-span: "true"
  opentelemetry-operation-name: "HTTP $request_method $uri"
```

After changing the ConfigMap, roll the controller so it reloads:

```bash theme={null}
kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller
```

> [!WARNING]
>
> **OTLP/gRPC only.** The Ingress-NGINX OTel module speaks OTLP gRPC on port **4317**. There is no HTTP/protobuf (4318) option — pointing it at 4318 will silently fail to export. The `odigos-data-collection-local-traffic` service accepts OTLP/gRPC on 4317.

> [!TIP]
>
> By default the controller samples every request through the ingress, so no sampler settings are needed. If you need to reduce volume, do it with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) rather than at the controller — that way the decision sees the full trace (errors, latency, and downstream spans) instead of being made blindly per request at the edge.

**2. Enable tracing per route**

Enabling OTel on the controller does not trace every route by default. Activate tracing on a route with per-`Ingress` annotations. For the full set of options, see the controller's [OpenTelemetry documentation](https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/).

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/enable-opentelemetry: "true"
    nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true"
spec:
  rules:
    - host: frontend.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend
                port:
                  number: 80
```

**3. Confirm a destination on the default data stream**

The Odigos collector only exports data when at least one destination is configured. Because the controller emits ingress spans on behalf of the edge rather than any specific Odigos-instrumented workload, the collector routes them through the **default data stream** — so make sure the destination you want the NGINX traces to reach is assigned to the default data stream.

## Validation

After traffic flows through the ingress, look for the controller's spans in your tracing backend. They appear under the service name you set in `otel-service-name` (for example, `nginx-ingress`), distinct from the application service names Odigos reports.

Confirm the path end to end. Neither the NGINX OTel module nor the Odigos collector logs anything per span, so **no OTel errors in the logs is the healthy state** — any OTel output is an error. To confirm spans are actually arriving, check your destination.

* **Controller logs** — confirm there are no export errors:

```bash theme={null}
kubectl -n ingress-nginx logs deployment/ingress-nginx-controller | grep -i otel
```

If the collector is unreachable, the gRPC exporter logs an `UNAVAILABLE` error on every flush:

```text theme={null}
[Error] File: .../otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "failed to connect to all addresses; last error: ... ipv4:<host>:4317: Connection refused"
```

That almost always means a wrong `otlp-collector-host`/`otlp-collector-port`, port 4318 instead of 4317, or a NetworkPolicy blocking egress to `odigos-system`.

* **Collector logs** — the collector is also quiet on success. Watch for export errors toward your destination:

```bash theme={null}
kubectl -n odigos-system logs -l odigos.io/collector-role=DATA_COLLECTION --tail=50
```

A misconfigured or unreachable destination shows up as repeated exporter errors. Depending on where the failure happens, you will see either `Exporting failed. Dropping data.` or `Exporting failed. Rejecting data.`:

```text theme={null}
error   exporterhelper/queue_sender.go:50   Exporting failed. Dropping data.   {"otelcol.component.id": "otlp/<destination>", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "...", "dropped_items": 5}
```

* **Your destination** — open your backend and look for a service named after `otel-service-name` (for example, `nginx-ingress`).

### Example

Real traces interleave the NGINX ingress span with the Odigos application spans — the ingress span wraps the upstream's application work, which itself can fan out into more application spans. A request through `ingress → frontend → checkout` lays out like this:

Horizontal position is time; indentation shows parent/child nesting.

```
                                0ms      50       100      150      200ms
                                 ├────────┼────────┼────────┼────────┤
  nginx-ingress · GET /            ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    frontend · GET /                ███████████████████████████████████
      frontend · GET /checkout            ████████████████████████
        checkout · GET /checkout            ██████████████████
          checkout · SELECT cart            ██████

  Legend:  █ Odigos (app)   ┏━┓ NGINX (ingress)
```

The ingress span wraps all the application work behind it. The gap before the first application span is **time spent at the edge** (routing and connecting upstream); everything inside is application latency.

> [!NOTE]
>
> NGINX ingress spans are additive — they do not replace the application spans Odigos generates. Each traced request yields an ingress span plus the application span(s), all sharing the same trace, as long as `opentelemetry-trust-incoming-span` is enabled so context propagates to your workloads.

## Tuning

| Key | Effect |
| - | - |
| `otel-service-name` | The service name spans show up under. Set per environment. |
| `opentelemetry-operation-name` | Template for the span name. `HTTP $request_method $uri` produces names like `HTTP GET /checkout`; supports NGINX variables. |
| `otel-sampler` | Leave at the default (`AlwaysOn`), which captures 100%, and configure sampling with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) instead. |
| `opentelemetry-trust-incoming-span` | Whether to continue trace context from inbound requests. Keep `true` so ingress spans link to upstream callers and downstream workloads. |
| `nginx.ingress.kubernetes.io/enable-opentelemetry` (annotation) | Per-`Ingress` enablement. Remove to stop tracing a specific route. |

## Troubleshooting

* **No spans at all** — the most common cause is a wrong `otlp-collector-host`. The `odigos-data-collection-local-traffic` service must exist in `odigos-system` and accept OTLP/gRPC. Verify with `kubectl get svc -n odigos-system`.
* **Set 4318 and nothing arrives** — the module is gRPC-only; use 4317.
* **OTel keys added to an application nginx ConfigMap did nothing** — expected; that is the stock-nginx ConfigMap, not the controller's. OTel config belongs in the controller ConfigMap.
* **Config changes ignored** — roll the controller: `kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller`.


# Agent Mount Method
Source: https://docs.odigos.io/enterprise/instrumentations/configuration/mount-method

For Odigos agents to run inside instrumented pod containers, certain files must be mounted into the container. In Kubernetes, these files are mounted under `/var/odigos` directory, with subdirectories for each agent. There are few mechanisms to achieve this, which are explained in this document.

## Mount Method

> [!WARNING]
>
> This section is for advanced users and odigos administrators.
>
> It is recommended to use the default settings, unless you have specific requirements or run into an issue.

## Supported Mount Methods

Odigos supports 4 mount methods, which can be used depending on the user preference, cluster policies, integration with existing tools, etc.

### 1. VirtualDevice

> [!NOTE]
>
> This is the default mount method

#### Pod Manifest Additions

For agents and languages that requires filesystem mounts, odigos webhook will add the following:

* For each instrumented container in the pod spec, odigos will add a resource requirement under the `spec.containers[].resources` field:

```yaml theme={null}
    resources:
      limits:
        instrumentation.odigos.io/generic: "1"
      requests:
        instrumentation.odigos.io/generic: "1"
```

#### Caveats

* With [Karpenter](../../setup/odigos-with-karpenter), provisioning can fail because Karpenter does not know about the `instrumentation.odigos.io/generic` extended resource when simulating new nodes. Support depends on your Karpenter version and provider: when [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is available and enabled, apply an overlay that adds this capacity (see the [Karpenter guide](../../setup/odigos-with-karpenter#virtual-device-nodeoverlay)). Otherwise use [`k8s-init-container`](#3-InitContainer), or [`k8s-host-path`](#2-HostPath) / [`k8s-csi-driver`](#4-CSI-Driver) with the Karpenter integration steps.
* Odiglet daemonset needs to be running on a node for instrumented pods to be scheduled on that node.

### 2. HostPath

This method is an opt-in configuration option which can be used if the virtual device method is not suitable for your cluster.

If it is supported in the cluster, it is preferred to use over the VirtualDevice method which requires odiglet component to run.

**Enabling HostPath**

* Profile: `odigos profile add mount-method-k8s-host-path`

* Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-host-path`

* Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-host-path`, or use helm cli `--set instrumentor.mountMethod=k8s-host-path` flag with helm upgrade.

* Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-host-path`.

#### Pod Manifest Additions

For agents and languages that requires filesystem mounts, odigos webhook will add the following:

* Volume to the pod spec, under the `spec.volumes` field:

```yaml theme={null}
  - hostPath:
      path: /var/odigos
      type: ""
    name: odigos-agent
```

* VolumeMount to the instrumented container specs, under the `spec.containers[].volumeMounts` field:

```yaml theme={null}
    - mountPath: /var/odigos/{agent_sub_dir}
      name: odigos-agent
      readOnly: true
      subPath: {agent_sub_dir}
```

#### Caveats

* The "HostPath" option should be enabled in the cluster. Some policy tools may block this, like [Open Policy Agent](https://www.openpolicyagent.org/) or [Kyverno](https://kyverno.io/). If blocked, use the VirtualDevice method, or ask the cluster administrator to whitelist the Odigos agent host path.
* With [Karpenter](../../setup/odigos-with-karpenter), additional configuration is required. By default Odigos adds node affinity for the odiglet-installed label, which blocks Karpenter from provisioning nodes. Enable `karpenter.enabled` and configure the NodePool `startupTaints` as described in the [Karpenter guide](../../setup/odigos-with-karpenter).

### 3. InitContainer

This is an opt-in configuration method designed for scenarios where using hostPath is not feasible due to security policies or virtual device limitations.

For agents and languages that require filesystem mounts, Odigos will inject an InitContainer that copies the necessary files into a shared volume mounted by the application container.

**Enabling InitContainer**

* Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-init-container`
* Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-init-container`, or use helm cli `--set instrumentor.mountMethod=k8s-init-container` flag with helm upgrade.
* Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-init-container`.

#### Pod Manifest Additions

Odigos will add an InitContainer that copies the necessary files for instrumentation, depending on the application language. These files will be placed into a shared volume that is also mounted by the application container.

* Volume to the pod spec, under the `spec.volumes` field:

```yaml theme={null}
  - hostPath:
      path: /var/odigos
      type: ""
    name: odigos-agent
```

#### Caveats

The InitContainer method must be enabled in the cluster and introduces a slight delay during pod initialization.
This approach is ideal for users who prioritize avoiding hostPath or virtual device usage over startup speed.

Works with [Karpenter](../../setup/odigos-with-karpenter) out of the box. No additional Karpenter configuration is required.

#### Init container image pull secrets

Kubernetes 1.35+ enables [`KubeletEnsureSecretPulledImages`](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) by default. The kubelet [re-verifies pull credentials](https://kubernetes.io/docs/concepts/containers/images/#ensure-image-pull-credential-verification) even when the agents image is already cached on the node. The injected `odigos-agents` init container therefore needs the same pull secret in the **application** namespace, not only in the Odigos namespace.

This applies when you use `k8s-init-container` with a private registry:

* the enterprise registry (`registry.odigos.io` via `odigos-enterprise-registry`, including `externalOnpremPullSecret` / `externalOnpremTokenSecret`)
* a private mirrored registry (`imagePrefix` + `imagePullSecrets`)

On Kubernetes 1.35+, the pod webhook adds those secret **names** to instrumented pods. You must create the same-named dockerconfig secret in every instrumented namespace yourself (External Secrets, Reflector, or a GitOps manifest). Below 1.35, odiglet pre-pulls the agents image onto each node and kubelet will use that cache without per-pod credentials.

### 4. CSI Driver

This is an opt-in configuration method that uses the Container Storage Interface (CSI) standard for mounting instrumentation files. It provides improved user experience over the VirtualDevice method by ensuring pods wait gracefully when the mount system isn't ready.

**Enabling CSI Driver**

* Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-csi-driver`
* Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-csi-driver`, or use helm cli `--set instrumentor.mountMethod=k8s-csi-driver` flag with helm upgrade.
* Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-csi-driver`.

#### Pod Manifest Additions

For agents and languages that requires filesystem mounts, odigos webhook will add the following:

* Volume to the pod spec, under the `spec.volumes` field:

```yaml theme={null}
  - name: odigos-agent
    csi:
      driver: odigos.csi.driver
      volumeAttributes:
        type: instrumentation
```

* VolumeMount to the instrumented container specs, under the `spec.containers[].volumeMounts` field:

```yaml theme={null}
    - mountPath: /var/odigos/{agent_sub_dir}
      name: odigos-agent
      readOnly: true
      subPath: {agent_sub_dir}
```

#### Benefits

* **Better User Experience**: Pods wait in "Creating" state until CSI driver is ready.
* **Standard Compliance**: Uses Kubernetes CSI standard for storage integration.
* **Same Functionality**: Provides identical mount behavior as the VirtualDevice method.

#### Caveats

* Requires CSI support in the cluster (available in Kubernetes 1.13+).
* Like VirtualDevice, requires odiglet daemonset to be running on nodes for instrumented pods to be scheduled.
* With [Karpenter](../../setup/odigos-with-karpenter), additional configuration is required. By default Odigos adds node affinity for the odiglet-installed label, which blocks Karpenter from provisioning nodes. Enable `karpenter.enabled` and configure the NodePool `startupTaints` as described in the [Karpenter guide](../../setup/odigos-with-karpenter).


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/dotnet/enrichment



Odigos will automatically instrument your .NET services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project by running:

```bash theme={null}
dotnet add package OpenTelemetry
```

## Creating Activity

To create a new `Activity`, use the `ActivitySource` class.

```csharp theme={null}
using System.Diagnostics;

public static class Telemetry
{
    // Name it after the service name for your app.
    // It can come from a config file, constants file, etc.
    public static readonly ActivitySource MyActivitySource = new(TelemetryConstants.ServiceName);

    public static void MyMethod()
    {
        using (var activity = MyActivitySource.StartActivity("MyMethod"))
        {
            // Your code here
        }
    }
}
```

## Additional Information

For more use cases, see the [OpenTelemetry .NET API documentation](https://opentelemetry.io/docs/languages/net/instrumentation/#creating-activities).


# Automatic Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/dotnet/native



> [!NOTE]
>
> Native Instrumentation is currently the default setting of Odigos **Open-Source** version.
>
> eBPF-Based Instrumentation is currently not supported for this language.

## Supported Versions

Supported versions of the .NET Framework range from **4.6.2** to **10.0.3**.

The following CPU architectures are supported:

* amd64 (x86\_64)
* arm64

## Instrumentation Libraries

The following .NET modules will be auto instrumented by Odigos:

### Application Frameworks

| Framework | Tracing Support | Metrics Support | Notes |
| - | :-: | :-: | - |
| [ASP.NET](https://docs.microsoft.com/aspnet/overview) | Yes | Yes | |
| [ASP.NET Core](https://docs.microsoft.com/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-6.0) | Yes | Yes | |

### Databases

| Library | Tracing Support | Metrics Support | Notes |
| - | - | - | - |
| [Microsoft.Data.SqlClient](https://www.nuget.org/packages/Microsoft.Data.SqlClient) | Yes | | |
| [System.Data.SqlClient](https://www.nuget.org/packages/System.Data.SqlClient) | Yes | | |
| MySql.Data | Partial | | This is the official MySQL library. |
| [MySqlConnector](https://mysqlconnector.net/) | Yes | | Seems to be the [recommended library for MariaDB](https://mariadb.com/docs/connectors/mariadb-connector-net/mariadb-connector-net-guide). |
| [Npgsql](https://www.npgsql.org/) | Yes | | |
| [Microsoft.Data.SqlLite](https://docs.microsoft.com/dotnet/standard/data/sqlite/?tabs=netcore-cli) | | | |
| [MongoDB.Driver](https://www.nuget.org/packages/mongodb.driver) | Yes | | |
| [StackExchange.Redis](https://stackexchange.github.io/StackExchange.Redis/) | Partial | | |
| [Elasticsearch-net](https://github.com/elastic/elasticsearch-net) | Yes | | 8.0.0+ only. |
| [Oracle.ManagedDataAccess.Core](https://www.nuget.org/packages/Oracle.ManagedDataAccess.Core) | Yes | | 23.4.0+ only. |
| [Oracle.ManagedDataAccess](https://www.nuget.org/packages/Oracle.ManagedDataAccess) | Yes | | 23.4.0+ only. |
| [AWSSDK.DynamoDBv2](https://www.nuget.org/packages/AWSSDK.DynamoDBv2) | | | |
| [Microsoft.Azure.Cosmos](https://www.nuget.org/packages/Microsoft.Azure.Cosmos) | | | |

### Inter-process communication (IPC)

| Library | Tracing Support | Metrics Support | Notes |
| - | - | - | - |
| [HttpClient](https://docs.microsoft.com/dotnet/api/system.net.http.httpclient) | Yes | Yes | |
| [HttpWebRequest](https://docs.microsoft.com/dotnet/api/system.net.httpwebrequest) | Yes | Yes | |
| [WCF](https://docs.microsoft.com/dotnet/framework/wcf/whats-wcf) | Yes (Client and Service side on .NET Framework) | | |
| [CoreWCF](https://github.com/CoreWCF/CoreWCF) | | | |
| [RestSharp](https://restsharp.dev/) | | | This library may be implicitly supported by instrumenting the underlying HttpClient or HttpWebRequest. |
| [gRPC-dotnet](https://github.com/grpc/grpc-dotnet) | Partial | | Client and service support should be added. |
| [GraphQL](https://www.nuget.org/packages/GraphQL/) | Partial | | |
| [GraphQL Client](https://github.com/graphql-dotnet/graphql-client) | | | |
| [RabbitMQ](https://www.nuget.org/packages/RabbitMQ.Client) | Yes | | `RabbitMQ.Client` 7.0.0+ has native support for traces. |
| [Kafka](https://www.nuget.org/packages/Confluent.Kafka) | | | |
| [NServiceBus](https://docs.particular.net/nservicebus/) | Yes | Yes | |
| [MassTransit](https://masstransit-project.com/) | Yes | | |

## Automatic Testing

The following scenarios are automatically tested with every release:

| .NET Version | LibC type | CPU |
| - | - | - |
| 6.0 | glibc | amd64 |
| 6.0 | musl | amd64 |
| 8.0 | glibc | amd64 |
| 8.0 | musl | amd64 |


# Automatic Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/golang/ebpf



## Supported Versions

Odigos uses the official [opentelemetry-go-instrumentation](https://github.com/open-telemetry/opentelemetry-go-instrumentation) OpenTelemetry Auto Instrumentation using eBPF, thus it supports the same golang versions as this project.

* Go runtime versions **1.19** through **1.26** are supported.

> [!NOTE]
>
> Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.
>
> The required Linux capabilities depend on your kernel version:
>
> * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`.
> * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`.
>
> We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`.

## Traces

Odigos will automatically instrument your golang services to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Go official auto Instrumentation Libraries.

## Instrumentation Libraries

The following go modules will be auto instrumented by Odigos:

### HTTP Frameworks

* [`github.com/gin-gonic/gin`](https://pkg.go.dev/github.com/gin-gonic/gin) HTTP web framework ⭐️
* [`net/http`](https://pkg.go.dev/net/http) go standard library for http server and client

> [!NOTE]
>
> Please note that in order for the `http.route` parameter to be properly extracted by Go auto instrumentation, you should use Go 1.22 or above and
> declare a template handler as described in [this article](https://go.dev/blog/routing-enhancements).

### HTTP Clients

* [`net/http`](https://pkg.go.dev/net/http) go standard library for http server and client

### Database Clients, ORMs, and data access libraries

* [`github.com/gocql/gocql`](https://pkg.go.dev/github.com/gocql/gocql) ⭐️ versions `>= v1.0.0`. database client library for Apache Cassandra
* [`github.com/elastic/go-elasticsearch/v8`](https://pkg.go.dev/github.com/elastic/go-elasticsearch/v8) ⭐️ versions >= v8.0.0
* [`database/sql`](https://pkg.go.dev/database/sql) go standard library for SQL database access

### Messaging Systems Clients

* [`github.com/rabbitmq/amqp091-go`](https://pkg.go.dev/github.com/rabbitmq/amqp091-go) ⭐️ versions `>= v1.4.0`. messaging client for RabbitMQ
* [`github.com/segmentio/kafka-go`](https://pkg.go.dev/github.com/segmentio/kafka-go) versions `>= v0.4.1`. messaging client for Apache Kafka
* [`github.com/apache/pulsar-client-go`](https://pkg.go.dev/github.com/apache/pulsar-client-go) ⭐️ versions `>= v0.12.0`. messaging client for Apache Pulsar
* [`github.com/IBM/sarama`](https://github.com/IBM/sarama) ⭐️ versions `>= v1.40.0`. messaging client for Apache Kafka

### RPC (Remote Procedure Call)

* [`google.golang.org/grpc`](https://pkg.go.dev/google.golang.org/grpc) versions `>= v1.14.0`. rpc client and server for gRPC framework

## Custom Instrumentations

We support adding custom eBPF-based instrumentations for arbitrary Go functions and invokable sites that exist in your application code or its dependencies.
By defining custom instrumentation rules, you can extend Odigos's auto-instrumentation capabilities and cover your specific use cases and application logic.
To learn more about creating custom eBPF-based instrumentations for Go applications, refer to the [Custom eBPF Instrumentation for Go Applications](../../pipeline/rules/custominstrumentation) guide.

## About Go Offsets

Auto-instrumentation for Go with eBPF works by using eBPF uprobes to dynamically read variables in memory at runtime. For example,
when instrumenting the `net/http` package Odigos is able to determine the `http.request.method` by probing the [`Method`](https://pkg.go.dev/net/http#Request)
field in the current `Request`.

Doing so relies on knowing the exact memory location of this field. This is known as the field's "offset" (because it refers to the field's location as a
number of bytes offset from the start of the current struct). With this information, our eBPF code can access the value directly.

However, most production Go programs do not preserve this offset information at compile time. Only Go programs compiled with [debug information](https://go.dev/doc/gdb)
will include this information by default. To work with all Go programs, Odigos ships with a precompiled list of known offsets for all currently supported
versions of instrumented libraries.

This precompiled list is updated to be up-to-date with every release of Odigos. But, this means that you may encounter errors instrumenting applications
that use very newly released versions of dependencies. If this occurs, it will be resolved by updating to the next release of Odigos, or alternatively
upgrading the locally-stored offsets with the [`odigos pro update-offsets`](../../../enterprise/cli/odigos_pro_update-offsets) command.

### Updating Offsets Between Versions

It is possible to pull the latest Go library support for Odigos without updating the entire Odigos installation. There are 3 ways to do so:

1. Manual updates (with [`odigos pro update-offsets`](../../../enterprise/cli/odigos_pro_update-offsets)).
2. Direct automatic updates (recommended, see below).
3. Mirrored image automatic updates (for disconnected environments, see below).

In both automatic modes, a Kubernetes CronJob is created which updates the locally stored Go library offsets used by the Odiglet. The differences
between these modes and how to enable them are described below.

#### Direct automatic updates

In "direct" mode, the CronJob will pull the latest Go library offsets directly from the Odigos cloud server and apply it to the cluster. This
option consumes the fewest resources but requires that your cluster has direct internet access to this file hosted on the web. If you have
restricted Internet egress policies, see "mirrored image updates" below.

Enable automatic updates on any standard cron schedule by setting the `goAutoOffsetsCron` option during installation or upgrade.
For example, enable nightly updates at midnight with `odigos install --set goAutoOffsetsCron="0 0 * * *"`.

To disable automatic updates, set the cron value to an empty string with `odigos install --set goAutoOffsetsCron=""`.

Alternatively, you can set these values in your `values.yaml` file for Helm installations:

```yaml theme={null}
goAutoOffsetsCron: "0 0 * * *"  # Enable nightly updates at midnight
goAutoOffsetsMode: "direct"     # Use direct mode
```

#### Mirrored image automatic updates

For clusters with restricted traffic policies, you have the option to [mirror the Go library updates as an image](../../../setup/docker-registry#install-odigos-images-from-a-custom-docker-registry).
In this mode, the CronJob will run an image that contains the latest Go library offsets bundled into the image itself. This mode
does not require direct Internet access from the cluster, but it does require you to mirror this image into your own registry and maintain
the latest version on your nodes.

To enable mirrored image updates, run `odigos install --set goAutoOffsetsMode="image"`. Then use the `goAutoOffsetsCron` option described above to set the schedule.

To disable mirrored image updates, run `odigos install --set goAutoOffsetsMode="direct"` (to switch back to direct mode), or set the `goAutoOffsetsCron` setting
to an empty string (`""`) as described above to disable automatic updates entirely.


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/golang/enrichment



Odigos will automatically instrument your Go services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Odigos fully supports the OpenTelemetry Go eBPF Auto SDK. See the [official docs on the Auto SDK](https://opentelemetry.io/docs/zero-code/go/autosdk/)
for information on using it.

Add the following dependencies to your project by running:

```bash theme={null}
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace
```

## Spans

Spans are the OpenTelemetry representation of operations in the execution of your code. You should record a span if you think the operation is meaningful and that observing it will help you understand the behavior of your system and operate it.

A span includes a name, start and end time, and key-value attributes for detailed information about the execution.

## Tracer

In OpenTelemetry, spans are created using the `Tracer` object which you can acquire from the OpenTelemetry API.

Each tracer is associated with a name and optional version.
These name and version are recorded as "instrumentation scope" into all spans created by this tracer.
Humans viewing the trace and data processors downstream can use this information to understand the origin of each span.

It is best practice to use the go module and package name for the tracer name.

```go theme={null}
package main

import (
   "context"
   "go.opentelemetry.io/otel"
)

var tracer = otel.Tracer("github.com/<yourusername>/<reponame>/<packagename>")
```

> [!WARNING]
>
> **Note**: Manually initializing a global TracerProvider will conflict with the Go Auto SDK's ability to link
> eBPF spans to custom spans and lead to broken trace IDs. If you are enriching your Odigos application with
> custom spans, do not initialize a global TracerProvider. See the [OpenTelemetry docs](https://opentelemetry.io/docs/zero-code/go/autosdk/)
> for more information.

## Creating Spans

```go theme={null}
func my_function(ctx context.Context) {
   // Span start time is set here
   ctx, span := tracer.Start(ctx, "my_function")
   // Span end time is set here
   defer span.End()
   // ...Do some work, use ctx if needed when calling next functions
}
```

Important Notes:

1. **Always End a span**:
   Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function.
2. **Propagate and use a valid context object**:
   When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace.
3. **Pass the context object downstream**:
   When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow.
4. **Assign meaningful names to spans**:
   Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context.
5. **Avoid dynamic, high cardinality data in span names**:
   Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

### Recording Span Attributes

Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting

Examples:

* User ID, organization ID, Account ID or other identifiers.
* Inputs - the relevant parameters or configuration that influenced the operation.
* Outputs - the result of the operation.
* Entities - the entities or resources that the operation interacted with.

Attribute names are lowercased strings in the form `my_application.my_attribute`, example: `my_service.user_id`.
Read more [here](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/)

To record attributes, use the `SetAttributes` method on the span object.

```go theme={null}
import (
   "context"
   "go.opentelemetry.io/otel/attribute"
)

func my_function(ctx context.Context, userId string) {
   _, span := tracer.Start(ctx, "my_function")
   defer span.End()
   span.SetAttributes(attribute.String("my_service.user_id", userId))
   // ...Do some work
}
```

Important Notes:

1. **Be cautious when recording data**:
   Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
2. **Attribute cost considerations**:
   Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
3. **Use static names for attributes**:
   Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute\_name) to provide clear context for downstream consumers.
4. **Adhere to OpenTelemetry semantic conventions**:
   Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.

### Recording Errors

To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts.

If an operation you are tracing fails, you can mark the span's status as an error and record the error details within the span. Here's how you can do it:

```go theme={null}
import (
   "context"
   "go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

func my_function(ctx context.Context, userId string) {
   _, span := tracer.Start(ctx, "my_function")
   defer span.End()
   err := doSomeWork() // ...Do some work
   if err != nil {
      span.SetStatus(codes.Error, "Operation failed")
      span.RecordError(err)
   }
}
```

## Additional Information

For more use cases, see the [OpenTelemetry Go API documentation](https://opentelemetry.io/docs/languages/go/instrumentation/#creating-spans).


# Go Hooks
Source: https://docs.odigos.io/enterprise/instrumentations/golang/go-hooks



Odigos provides Go functions to augment and enhance auto-instrumentation. These functions are intended
to be used in conjunction with auto-instrumentation to provide enriched functionality and interaction for
developers running in an Odigos environment.

## Trace and Span Context

The following hooks allow you to access the current trace and span IDs in your application.
This is useful when you need to correlate your application's logs with the distributed traces generated by Odigos.

These functions are available in the `github.com/odigos-io/odigos/hooks/go` module, ie:

```
go get github.com/odigos-io/odigos/hooks/go
```

### Get W3C Trace Context

```go theme={null}
traceContext := gohooks.GetW3CTraceContext(context.Context) string
```

This function returns the full [W3C Trace Context](https://www.w3.org/TR/trace-context/) for the current function.

### Get Current Trace ID

```go theme={null}
traceId := gohooks.GetTraceID(context.Context) string
```

Returns the current trace ID. If there is no active trace, it returns a zero value.

### Get Current Span ID

```go theme={null}
spanId := gohooks.GetSpanID(context.Context) string
```

Returns the current span ID. If there is no active span, it returns a zero value.

### Check for zero values

The following helper functions can be used to check if the hooks returned a zero value. Zero
values indicate that no trace is currently active at the call site of the hook.

```go theme={null}
isZeroTraceContext := gohooks.IsZeroTraceContext(traceContext)
isZeroTraceId := gohooks.IsZeroTraceId(traceId)
isZeroSpanId := gohooks.IsZeroSpanId(spanId)
```

## Gin Middleware

Odigos can instrument Gin middleware using a provided wrapper handler from `github.com/odigos-io/odigos/hooks/go/gin`:

```go theme={null}
ginhooks.OdigosGinMiddleware(middlewares ...gin.HandlerFunc)
```

For example, see the following application that uses 2 middleware handlers, `LoggerMiddleware` and `AuthMiddleware`:

```go theme={null}
package main

import (
	"log"
	"net/http"
	"time"

	"github.com/gin-gonic/gin"
	gohooks "github.com/odigos-io/odigos/hooks/go/gin"
)

func main() {
	r := gin.New()

	// Define all individual middleware
	middlewares := []gin.HandlerFunc{
		LoggerMiddleware(),
		AuthMiddleware(),
	}

	// Use the wrapper that chains the middleware
	r.Use(gohooks.OdigosGinMiddleware(middlewares...)...)

	// Sample route
	r.GET("/hello", func(c *gin.Context) {
		c.JSON(http.StatusOK, gin.H{"message": "Hello, world!"})
	})

	r.Run(":8080")
}

// LoggerMiddleware logs the request method, path, and duration.
func LoggerMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		start := time.Now()
		c.Next()
		log.Printf("Request: %s %s | Duration: %v", c.Request.Method, c.Request.URL.Path, time.Since(start))
	}
}

// AuthMiddleware checks for a static bearer token in the Authorization header.
func AuthMiddleware() gin.HandlerFunc {
	return func(c *gin.Context) {
		token := c.GetHeader("Authorization")
		if token != "Bearer secret-token" {
			c.JSON(http.StatusUnauthorized, gin.H{"error": "Unauthorized"})
			c.Abort()
			return
		}
		c.Next()
	}
}
```

The created spans will list the middleware function in the span attribute `odigos.gin.middleware.func`.
In the above example, these are `main.main.LoggerMiddleware.func2` and `main.main.AuthMiddleware.func3`.

## Example Application

First, follow the steps in the [Quickstart](../../quickstart/introduction) to set up a Kind cluster,
install Jaeger, and [install Odigos](../../quickstart/installation) (note: Go hooks are only available with
an enterprise token, which you can set in this walkthrough by installing with `--onprem-token=<YOUR_TOKEN>`).

Next create a folder for your sample app and initialize the Go module:

```bash theme={null}
mkdir odigos-hooks-example && cd odigos-hooks-example
touch main.go
```

Now, here's a simple example using `net/http` that demonstrates how to use the Go hooks:

```go theme={null}
package main

import (
  "fmt"
  "log"
  "net/http"

  gohooks "github.com/odigos-io/odigos/hooks/go"
)

func main() {
  // Create a simple HTTP server
  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    // Get the current trace and span IDs
    traceID := gohooks.GetTraceID(r.Context())
    spanID := gohooks.GetSpanID(r.Context())

    // Log the IDs
    log.Printf("Current Trace ID: %s", traceID)
    log.Printf("Current Span ID: %s", spanID)

    fmt.Fprintf(w, "Trace ID: %s\nSpan ID: %s\n", traceID, spanID)
	})

  // Start the server
  log.Println("Server starting on port 8080")
  if err := http.ListenAndServe(":8080", nil); err != nil {
    log.Fatalf("Failed to start server: %v", err)
  }
}
```

Initialize the Go module:

```bash theme={null}
go mod init github.com/yourusername/odigos-hooks-example
go mod tidy
```

Create the following `Dockerfile`:

```dockerfile theme={null}
# Build stage
FROM golang:1.24-alpine AS builder

WORKDIR /app

# Copy go mod and sum files
COPY go.mod go.sum ./

# Download dependencies
RUN go mod download

# Copy source code
COPY . .

# Build the application
RUN CGO_ENABLED=0 GOOS=linux go build -o hooks-example

# Final stage
FROM alpine:3.19

WORKDIR /app

# Copy the binary from builder
COPY --from=builder /app/hooks-example .

# Expose the application port
EXPOSE 8080

# Run the application
CMD ["./hooks-example"]
```

Build and push the image:

```bash theme={null}
export IMAGE_TAG=docker.io/your-username/hooks-example
```

```bash theme={null}
docker build -t $IMAGE_TAG .
docker push $IMAGE_TAG
```

Create the following `deployment.yaml`:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hooks-example
  labels:
    app: hooks-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hooks-example
  template:
    metadata:
      labels:
        app: hooks-example
    spec:
      containers:
      - name: hooks-example
        image: ${IMAGE_TAG}
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: hooks-example
spec:
  selector:
    app: hooks-example
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
```

Apply the deployment using the `IMAGE_TAG` environment variable you defined above:

```bash theme={null}
envsubst < deployment.yaml | kubectl apply -f -
```

Now, follow the Quickstart steps to [Build a pipeline](../../quickstart/building-a-pipeline) in order to
instrument the sample application.

To generate traces, connect to the service with `kubectl port-forward`:

```bash theme={null}
kubectl port-forward svc/hooks-example 8080:80
```

And in another terminal, generate spans with:

```bash theme={null}
curl localhost:8080
```

And you will see the current trace and span ID returned:

```bash theme={null}
$ curl localhost:8080
Trace ID: 1c3f5ea2cff194ba12d06b8353ae08ed
Span ID: 57460850f7634b30
```

Verify this is the same trace context shown in the Jaeger UI:

```bash theme={null}
kubectl port-forward -n tracing svc/jaeger 16686:16686
```

And open `localhost:16686` in your browser.

## Troubleshooting

If you're not seeing trace or span IDs:

1. Make sure Odigos is properly configured and running
2. Verify that your application is being instrumented by Odigos
3. Check that you've initialized the hooks correctly
4. Ensure your application is receiving requests that are part of a trace


# eBPF-Based Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/java/ebpf



## Supported Versions

Odigos eBPF-Based Java Instrumentation requires a Java Virtual Machine runtime version of **8** through **25**.

> [!NOTE]
>
> Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.
>
> The required Linux capabilities depend on your kernel version:
>
> * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`.
> * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`.
>
> We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`.

The following JVM runtimes are continuously tested and verified:

| JVM Runtime | Version |
| - | - |
| OpenJDK | 8 |
| OpenJDK | 11 |
| OpenJDK | 17 |
| OpenJDK | 21 |
| OpenJDK | 25 |
| Azul Zulu | 8 |
| Azul Zulu | 11 |
| Azul Zulu | 17 |
| Azul Zulu | 21 |
| Azul Zulu | 25 |

## Instrumentation Libraries

The following packages will be auto instrumented by Odigos:

### HTTP Frameworks

* [`Akka HTTP`](https://doc.akka.io/docs/akka-http/current/) versions `10.0.15 - 10.5.3`. HTTP server and client toolkit for Akka
* [`Grizzly HTTP Server`](https://javaee.github.io/grizzly/) versions `2.3.35 - 4.0.2`. NIO HTTP server framework
* [`http4s Blaze`](https://http4s.org/) version `0.23.18`. HTTP server and client for Scala (Blaze backend)
* [`http4s Ember`](https://http4s.org/) version `0.23.36`. HTTP server and client for Scala (Ember backend)
* [`Jetty (via Dropwizard)`](https://jetty.org/) versions `4.0.14 - 5.0.2`. HTTP server via Dropwizard
* [`Netty HTTP`](https://netty.io/) versions `4.1.121.Final - 4.2.17.Final`. asynchronous event-driven network framework
* [`Spring Boot`](https://spring.io/projects/spring-boot) versions `2.5.15 - 4.1.1`. server side applications framework
* [`Spring Async Server`](https://docs.spring.io/spring-framework/reference/web/webmvc-concurrency.html) versions `3.2.12 - 4.1.1`. asynchronous Spring MVC server support

### HTTP Clients

* [`Apache HttpClient 4`](https://hc.apache.org/httpcomponents-client-4.5.x/) versions `4.0.3 - 4.5.14`. HTTP client library
* [`Apache HttpClient 5`](https://hc.apache.org/httpcomponents-client-5.5.x/) versions `5.0.4 - 5.6.4`. HTTP client library
* [`AsyncHttpClient`](https://github.com/AsyncHttpClient/async-http-client) versions `2.1.2 - 3.0.13`. asynchronous HTTP client
* [`OkHttp`](https://square.github.io/okhttp/) versions `3.4.2 - 4.12.0`. HTTP client for Java and Kotlin
* [`Reactor Netty HTTP Client`](https://projectreactor.io/docs/netty/release/reference/index.html) versions `1.0.48 - 1.3.7`. reactive HTTP client
* [`Retrofit 2`](https://square.github.io/retrofit/) versions `2.0.2 - 3.0.0`. type-safe HTTP client for Android and Java
* [`Spring REST Client`](https://docs.spring.io/spring-framework/reference/integration/rest-clients.html) versions `2.5.15 - 3.5.16`. rest client library for Spring

### RPC (Remote Procedure Call)

* [`gRPC`](https://grpc.io/docs/languages/java/) versions `1.6.1 - 1.83.1`. RPC client and server for gRPC
* [`GraphQL Java`](https://www.graphql-java.com/) versions `24.3 - 26.1`. GraphQL server implementation for Java

### Database Clients, ORMs, and data access libraries

* [`Apache Derby`](https://db.apache.org/derby/) versions `10.14.2.0 - 10.16.1.1`. embedded relational database
* [`Azure Cosmos DB Java SDK`](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos) versions `4.0.1 - 4.82.0`. database client library for Azure Cosmos DB
* [`Cassandra (DataStax Driver)`](https://docs.datastax.com/en/developer/java-driver/) versions `4.0.1 - 4.17.0`. database client library for Apache Cassandra
* [`Cassandra (Driver 3.x)`](https://docs.datastax.com/en/developer/java-driver/3.11/) versions `3.0.8 - 3.11.5`. legacy DataStax Cassandra driver
* [`Elasticsearch (REST Client)`](https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/index.html) versions `8.9.2 - 8.19.20`. Elasticsearch low-level REST client
* [`H2 Database`](https://www.h2database.com/) versions `2.0.206 - 2.4.240`. embedded relational database
* [`Hibernate ORM`](https://hibernate.org/orm/) versions `3.0.13 - 4.1.1`. object-relational mapping framework (via Spring Data JPA)
* [`HikariCP`](https://github.com/brettwooldridge/HikariCP) versions `3.0.0 - 7.1.0`. JDBC connection pool
* [`HSQLDB`](https://hsqldb.org/) versions `2.0.0 - 2.7.4`. embedded relational database
* [`IBM Informix JDBC`](https://www.ibm.com/docs/en/informix-servers) versions `4.50.14 - 15.0.1.4`. JDBC driver for IBM Informix
* [`MariaDB JDBC`](https://mariadb.com/kb/en/about-mariadb-connector-j/) versions `3.0.11 - 3.5.10`. JDBC driver for MariaDB
* [`MongoDB (Sync Driver)`](https://www.mongodb.com/docs/drivers/java/sync/current/) versions `5.4.0 - 5.11.0`. synchronous MongoDB driver
* [`MySQL JDBC`](https://dev.mysql.com/doc/connector-j/en/) versions `8.0.33 - 26.7.0`. JDBC driver for MySQL
* [`Oracle JDBC`](https://www.oracle.com/database/technologies/appdev/jdbc.html) versions `19.3.0.0 - 23.26.3.0.0`. JDBC driver for Oracle Database
* [`PostgreSQL JDBC`](https://jdbc.postgresql.org/) versions `42.2.29 - 42.7.13`. JDBC driver for PostgreSQL
* [`R2DBC H2`](https://github.com/r2dbc/r2dbc-h2) versions `1.0.1.RELEASE - 1.1.0.RELEASE`. reactive H2 driver
* [`R2DBC MySQL`](https://github.com/asyncer-io/r2dbc-mysql) version `1.4.3`. reactive MySQL driver
* [`R2DBC PostgreSQL`](https://github.com/pgjdbc/r2dbc-postgresql) versions `1.0.9.RELEASE - 1.1.2.RELEASE`. reactive PostgreSQL driver
* [`Redis (Jedis)`](https://github.com/redis/jedis) versions `2.7.3 - 8.0.1`. Redis client for Java
* [`Redis (Lettuce)`](https://github.com/redis/lettuce) versions `5.1.8.RELEASE`, `6.4.2.RELEASE - 7.2.3.RELEASE`. advanced Redis client for Java
* [`SQLite JDBC`](https://github.com/xerial/sqlite-jdbc) versions `3.30.1 - 3.53.4.0`. JDBC driver for SQLite
* [`Spymemcached`](https://github.com/couchbase/spymemcached) versions `2.9.1 - 2.12.3`. Memcached client for Java
* [`Sybase JDBC (jTDS)`](https://sourceforge.net/projects/jtds/) versions `1.2.8 - 1.3.1`. JDBC driver for Sybase / SQL Server
* [`WildFly JDBC`](https://www.wildfly.org/) versions `26.0.1.Final - 41.0.1.Final`. JDBC instrumentation for WildFly (dynamic attach and premain)

### Messaging Systems Clients

* [`ActiveMQ Artemis`](https://activemq.apache.org/components/artemis/) versions `2.30.0 - 2.43.0`. messaging client for ActiveMQ Artemis
* [`ActiveMQ V5`](https://activemq.apache.org/) versions `5.17.7 - 5.19.1`. messaging client for ActiveMQ 5.x
* [`ActiveMQ V6`](https://activemq.apache.org/) versions `6.0.1 - 6.1.8`. messaging client for ActiveMQ 6.x
* [`Apache Kafka`](https://kafka.apache.org/) versions `2.5.1 - 4.3.1`. messaging client for Apache Kafka. Includes both producer and consumer clients
* [`Apache Pulsar`](https://pulsar.apache.org/) versions `3.0.17 - 4.2.4`. messaging client for Apache Pulsar
* [`Google Cloud Pub/Sub`](https://cloud.google.com/pubsub/docs/reference/libraries) versions `1.116.4 - 1.154.0`. messaging client for Google Cloud Pub/Sub
* [`RabbitMQ`](https://www.rabbitmq.com/client-libraries/java-api-guide) versions `2.7.1 - 5.35.0`. messaging client for RabbitMQ
* [`Spring Integration`](https://spring.io/projects/spring-integration) versions `3.0.13 - 4.1.1`. enterprise integration patterns for Spring

### Scheduling and Batch

* [`Quartz Scheduler`](https://www.quartz-scheduler.org/) versions `2.0.2 - 2.5.2`. job scheduling library
* [`Spring Batch`](https://spring.io/projects/spring-batch) versions `3.0.13 - 3.5.16`. batch processing framework
* [`Spring Scheduling`](https://docs.spring.io/spring-framework/reference/integration/scheduling.html) versions `2.7.18 - 4.1.1`. Spring task scheduling support

### Concurrency Libraries

* [`Reactor Core`](https://projectreactor.io/) versions `3.3.22.RELEASE - 3.8.7`. reactive streams library
* [`Reactor Netty`](https://projectreactor.io/docs/netty/release/reference/index.html) versions `1.0.48 - 1.3.7`. reactive network application framework
* [`Resilience4j`](https://resilience4j.readme.io/) versions `2.0.2 - 2.4.0`. fault tolerance library (CircuitBreaker and Retry)
* [`RxJava`](https://github.com/ReactiveX/RxJava) versions `3.0.13 - 3.1.12`. library for composing asynchronous and event-based programs using observable sequences
* [`ExecutorService`](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html) any supported JVM. executor service for managing asynchronous tasks

### Other

* [`Apache Karaf OSGi`](https://karaf.apache.org/) versions `4.4.10 - 4.4.11`. OSGi runtime (dynamic attach)
* [`Context Propagation`](https://docs.spring.io/spring-framework/reference/integration/observability.html) versions `3.2.12 - 4.1.1`. distributed context propagation for Spring applications


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/java/enrichment



Odigos will automatically instrument your Java services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project:

**Maven**

```xml theme={null}
<project>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry</groupId>
                <artifactId>opentelemetry-bom</artifactId>
                <version>1.35.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
        </dependency>
    </dependencies>
</project>
```

**Gradle**

```groovy theme={null}
dependencyManagement {
    imports {
        mavenBom("io.opentelemetry:opentelemetry-bom:1.35.0")
    }
}

dependencies {
implementation("org.springframework.boot:spring-boot-starter-web");
implementation("io.opentelemetry:opentelemetry-api");
}

```

If you are not using Spring and its `io.spring.dependency-management` dependency management plugin, install the OpenTelemetry BOM and API using Gradle dependencies only.

```groovy theme={null}
dependencies {
    implementation(platform("io.opentelemetry:opentelemetry-bom:1.35.0"));
    implementation("io.opentelemetry:opentelemetry-api");
}
```

## Creating Spans

To create a span, use the `Tracer` interface. The `Tracer` interface is the main entry point for the OpenTelemetry API. It allows you to create spans and manage the current span.
Acquire a `Tracer` instance using the `OpenTelemetry` class:

```java theme={null}
import io.opentelemetry.api.GlobalOpenTelemetry;

Tracer tracer = GlobalOpenTelemetry.getTracer("instrumentation-scope-name", "instrumentation-scope-version");
```

Now you can create a span using the `Tracer` instance:

```java theme={null}
Span span = tracer.spanBuilder("span-name").startSpan();

try (Scope scope = span.makeCurrent()) {
    // Your code goes here.
} finally {
    span.end();
}
```

## Additional Information

For more use cases, see the [OpenTelemetry Java API documentation](https://opentelemetry.io/docs/languages/java/instrumentation/#create-spans).


# eBPF-Based Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/nodejs/ebpf



## Supported Versions

Odigos eBPF-Based NodeJS Instrumentation requires a NodeJS runtime version of **14** through **25**.

> [!NOTE]
>
> Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.
>
> The required Linux capabilities depend on your kernel version:
>
> * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`.
> * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`.
>
> We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`.

## Instrumentation Libraries

The following npm packages will be auto instrumented by Odigos:

### HTTP Frameworks

* [`fastify`](https://www.npmjs.com/package/fastify) versions `>=3.0.0 <5`. http web application framework
* [`http`](https://nodejs.org/api/http.html) `node:http` http server and client modules
* [`https`](https://nodejs.org/api/https.html) `node:https` https server and client modules
* [`@hapi/hapi`](https://www.npmjs.com/package/@hapi/hapi) versions `>=17.0.0 <22`. http web application framework
* [`restify`](https://www.npmjs.com/package/restify) versions `>=4.0.0 <12`. http web application framework
* [`router`](https://www.npmjs.com/package/router) versions `>=1.0.0 <2`. http middleware router

### HTTP Clients

* [`http`](https://nodejs.org/api/http.html) `node:http` http server and client modules
* [`https`](https://nodejs.org/api/https.html) `node:https` https server and client modules

### RPC (Remote Procedure Call)

* [`@grpc/grpc-js`](https://www.npmjs.com/package/@grpc/grpc-js) versions `>=1.0.0 <2`. rpc client and server for gRPC framework


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/nodejs/enrichment



Odigos will automatically instrument your Node.js services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following npm packages to your service by running:

```bash theme={null}
npm install @opentelemetry/api@1.9.0
```

> [!WARNING]
>
> Odigos agent implements OpenTelemetry API version 1.9.0. Any version greater than 1.9.0 may not be compatible with Odigos agent and fail to produce data.\\
>
> Please do not use caret range ~~`@opentelemetry/api@^1.9.0`~~ for this dependency in your package.json to avoid pulling in incompatible version.

## Creating Spans

To enhance your traces with spans for your custom business logic operations, you need to create a span in your code and wrap the operation you want to observe.

The `tracer` object is the entry point for the OpenTelemetry API.

You can create a `tracer` object using the `getTracer` method from the `@opentelemetry/api` package.

**JavaScript**

```javascript theme={null}
const opentelemetry = require('@opentelemetry/api');

const tracer = opentelemetry.trace.getTracer(
   'instrumentation-scope-name',
   'instrumentation-scope-version',
);

function myFunction() {
   const span = tracer.startSpan('myFunction');
   // Your code here
   span.end();
}
```

**TypeScript**

```typescript theme={null}
import opentelemetry from '@opentelemetry/api';

const tracer = opentelemetry.trace.getTracer(
   'instrumentation-scope-name',
   'instrumentation-scope-version',
);

function myFunction() {
   const span = tracer.startSpan('myFunction');
   // Your code here
   span.end();
}
```

Make sure to replace `instrumentation-scope-name` and `instrumentation-scope-version` with the name and version of your instrumented file.

Important Notes:

1. **Always End a span**:
   Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function.
2. **Propagate and use a valid context object**:
   When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace.
3. **Pass the context object downstream**:
   When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow.
4. **Assign meaningful names to spans**:
   Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context.
5. **Avoid dynamic, high cardinality data in span names**:
   Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

## Additional Information

For more use cases, see the [OpenTelemetry JavaScript API documentation](https://opentelemetry.io/docs/languages/js/instrumentation/#create-spans).


# Overview
Source: https://docs.odigos.io/enterprise/instrumentations/overview

Odigos will automatically apply instrumentation to your selected workloads to record observability signals from your services such as collecting traces, metrics, and logs without any code changes.

## Supported Runtimes

Odigos provides automatic instrumentations for the following runtimes:

> **[Go](/enterprise/instrumentations/golang/ebpf)**
>
> Versions 1.19 - 1.26 are supported

> **[Javascript](/enterprise/instrumentations/nodejs/ebpf)**
>
> Versions 14 - 25 are supported

> **[Python](/enterprise/instrumentations/python/native)**
>
> Versions 3.8 - 3.14 are supported

> **[Java](/enterprise/instrumentations/java/ebpf)**
>
> Versions 8 - 25 are supported

> **[PHP](/enterprise/instrumentations/php/native)**
>
> Versions 8.1.0 - 8.4.x are supported

> **[Ruby](/enterprise/instrumentations/ruby/native)**
>
> Versions 3.1.0 - 3.5.x are supported

> **[.NET](/enterprise/instrumentations/dotnet/native)**
>
> Versions 4.6.2 - 10.0.3 are supported

## Enrich with OpenTelemetry APIs

Odigos automatically generates data for open-source libraries and frameworks.
Additional trace spans, metrics datapoints and log records can be added using the OpenTelemetry APIs.
Odigos will automatically capture this data and deliver it to the chosen [destination](../backends-overview) alongside the automatically generated data.

> [!NOTE]
>
> No need to configure the OpenTelemetry SDK, Odigos will automatically
> configure it for you.

Select a language to learn how to enrich your data with OpenTelemetry APIs:

> **[Go](/enterprise/instrumentations/golang/enrichment)**
>
> For applications written in Go

> **[Javascript](/enterprise/instrumentations/nodejs/enrichment)**
>
> For applications written in JavaScript or Node.js

> **[Python](/enterprise/instrumentations/python/enrichment)**
>
> For applications written in Python

> **[Java](/enterprise/instrumentations/java/enrichment)**
>
> For applications written in Java or JVM-based languages

> **[PHP](/enterprise/instrumentations/php/enrichment)**
>
> For applications written in PHP

> **[Ruby](/enterprise/instrumentations/ruby/enrichment)**
>
> For applications written in Ruby

> **[.NET](/enterprise/instrumentations/dotnet/enrichment)**
>
> For applications written in C# or .NET

## Instrumentation Flow:

1. User selects a workload to auto instrument by creating a `Source` object (per workload or per namespace).
2. Odigos `instrumentor` component watches for changes with `Source` objects, and creates a relative `InstrumentationConfig` object.
3. Odigos `odiglet` component watches for changes with the `InstrumentationConfig` objects, and runs a runtime inspection on running pods to detect the programming language for every container.
4. Odigos `instrumentor` component watches for changes with the `InstrumentationConfig` object, and adds a resource request into each pod spec of all relevant containers in the workload manifest. These resource requests are called `Instrumentation Devices`.
5. Kubernetes detects the changes in the manifest, and rollout-restarts the pods with the new `Instrumentation Devices`.
6. New pods are scheduled and started. Odiglet resolves the resource request by mounting the auto instrumentation code and relevant environment variables into the container.
7. The auto instrumentation code starts the OpenTelemetry SDK and sends telemetry data to the odigos pipeline.

## Auto-rollback and Recover

If pods enter CrashLoopBackOff or ImagePullBackOff after instrumentation is injected, Odigos **disables the agent** and restarts the workload **without** instrumentation. The UI may show that Odigos detected a crash and rolled back instrumentation to protect the workload.

That rollback is **not** a Kubernetes Deployment undo. Odigos does not restore a previous ReplicaSet or Helm revision. It only removes Odigos agents from the workload.

**Recover** (in the UI) means **try instrumentation again**. It clears the rollback latch and rolls the workload **with** agents. Clicking Recover does not remove remaining instrumentation or roll back the Deployment a second time.

Disable automatic uninstrumentation with:

```bash theme={null}
odigos install --set autoRollback.disabled=true
```

or Helm `autoRollback.disabled=true`.


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/php/enrichment



Odigos will automatically instrument your PHP services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project by running:

```bash theme={null}
composer require open-telemetry/api
```

## Creating a Span

To create a new span, use the `TracerProvider`.

```php theme={null}
<?php

use OpenTelemetry\API\Globals;
use Psr\Http\Message\ResponseInterface as Response;
use Psr\Http\Message\ServerRequestInterface as Request;
use Slim\Factory\AppFactory;

require __DIR__ . '/vendor/autoload.php';

$tracer = Globals::tracerProvider()->getTracer('demo');

$app = AppFactory::create();
$app->get('/rolldice', function (Request $request, Response $response) use ($tracer) {
    $span = $tracer
        ->spanBuilder('manual-span')
        ->startSpan();

    $result = random_int(1,6);
    $response->getBody()->write(strval($result));

    $span
        ->addEvent('rolled dice', ['result' => $result])
        ->end();
    return $response;
});

$app->run();
```

## Additional Information

For more use cases, see the [OpenTelemetry PHP API documentation](https://opentelemetry.io/docs/languages/php).


# Automatic Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/php/native



> [!NOTE]
>
> This language instrumentation is using OpenTelemetry native-php SDK and instrumentation libraries.

## Supported Versions

The minimal supported version of PHP is **8.1.0**.
The maximal supported version of PHP is **8.4.x**, when PHP releases a new version we'll add the support for it.

> [!CAUTION]
>
> PHP version **8.0** has been deprecated and is no longer supported.

> [!WARNING]
>
> PHP version **8.1** has limited library support and will be the next version to be deprecated.

> [!TIP]
>
> PHP version **8.2+** is the recommended version for new projects.

## Instrumentation Libraries

The following PHP modules will be auto instrumented by Odigos:

### PHP Standards Recommendations (PSR):

* [`psr/3 (log)`](https://packagist.org/packages/psr/log) versions `>=1`
* [`psr/6 (cache)`](https://packagist.org/packages/psr/cache) versions `>=1`
* [`psr/16 (simple-cache)`](https://packagist.org/packages/psr/simple-cache) versions `>=1`
* [`psr/14 (event-dispatcher)`](https://packagist.org/packages/psr/event-dispatcher) versions `>=1`
* [`psr/15 (http-server-middleware)`](https://packagist.org/packages/psr/http-server-middleware) versions `>=1`
* [`psr/18 (http-client)`](https://packagist.org/packages/psr/http-client) versions `>=1`

### HTTP Clients:

* [`php-http/httplug`](https://packagist.org/packages/php-http/httplug) versions `>=2`
* [`guzzlehttp/guzzle`](https://packagist.org/packages/guzzlehttp/guzzle) versions `>=7`
* [`ext-curl`](https://www.php.net/manual/en/book.curl.php) versions `*`

> [!WARNING]
>
> `ext-curl` not supported for PHP 8.1

### Frameworks:

> [!NOTE]
>
> We're working on adding support for `laravel`, `cakephp`, `wordpress`, `yii2` and `codeigniter` frameworks.

* [`slim/slim`](https://packagist.org/packages/slim/slim) versions `>=4`
* [`symfony/http-kernel`](https://packagist.org/packages/symfony/http-kernel) versions `*`
* [`symfony/http-client-contracts`](https://packagist.org/packages/symfony/http-client-contracts) versions `*`

### Databases:

* [`ext-mongodb`](https://www.php.net/manual/en/book.mongodb.php) versions `>=1.13`
* [`ext-mysqli`](https://www.php.net/manual/en/book.mysqli.php) versions `*`

> [!WARNING]
>
> `ext-mysqli` not supported for PHP 8.1

* [`ext-pdo`](https://www.php.net/manual/en/book.pdo.php) versions `*`

> [!WARNING]
>
> `ext-pdo` not supported for PHP 8.1

* [`doctrine/dbal`](https://packagist.org/packages/doctrine/dbal) versions `>=3`

> [!WARNING]
>
> `doctrine/dbal` not supported for PHP 8.1

### Messaging Systems:

* [`ext-amqp`](https://pecl.php.net/package/amqp) versions `*`

> [!WARNING]
>
> `ext-amqp` not supported for PHP 8.1

* [`ext-rdkafka`](https://pecl.php.net/package/rdkafka) versions `*`

> [!WARNING]
>
> `ext-rdkafka` not supported for PHP 8.1

### Atificial Intelligence:

* [`openai-php/client`](https://packagist.org/packages/openai-php/client) versions `*`


# Exporting Profiles to Destinations
Source: https://docs.odigos.io/enterprise/instrumentations/profiling/destinations

Send continuous profiling data from Odigos to third-party observability backends for long-term retention and analysis through Odigos Destinations

## Exporting Profiles to a Destination

The Odigos UI gives you fast, in-memory access to profiles for enabled Sources — ideal for live investigation when you don't need historical data but just want to inspect certain workloads on demand.
For **long-term retention**, **time-range analysis**, and **cross-service comparison**, you can route profile data to an external backend the same way you route traces, metrics or logs, by configuring a Destination in Odigos.

When a profile-capable Destination is configured, the gateway starts exporting OTLP Profiles to that backend alongside the Odigos UI.

## Supported Format

Profile data is emitted as **OTLP Profiles** — the OpenTelemetry profiles signal. Any backend that can ingest OTLP Profiles over HTTP can, in principle, receive data from Odigos.

> [!NOTE]
>
> OTLP Profiles is still **experimental** — the wire format and ingestion path (`/v1development/profiles`) may change. Keep your backend on a recent version that speaks a compatible protocol revision.

## Prerequisites

Before exporting profiles to a Destination:

1. Profiling is enabled cluster-wide (`profiling.enabled=true` Helm value). See [Getting Started](/enterprise/instrumentations/profiling/getting-started).
2. The workload you want to profile is registered as a [Source](/enterprise/pipeline/sources/introduction).
3. The Destination CRD is configured with the `PROFILES` signal.
4. Your chosen backend is reachable from the Odigos gateway and accepts OTLP Profiles.

The Destination can be created through the Odigos UI, CLI, or Kubernetes CRD. Profile export starts once profiling is enabled and the Destination includes the `PROFILES` signal.

## Try out configuring a profiling Destinations

> **[Pyroscope](/enterprise/backends/pyroscope)**
>
> Self-hosted Grafana Pyroscope. Receives OTLP profiles over HTTP at `/v1development/profiles`.

## Example: Pyroscope

![Screen recording: add Pyroscope destination with Profiles, then view exported profiles in Pyroscope](https://mintcdn.com/odigos/HulnWKhzpxgFDURc/images/enterprise/profiling/profiling-pyroscope-destinations-demo.gif?s=32f1a5a131fd6997611b91899936e0a4)

**1. Enable profiling on the cluster**

Follow [Getting started](/enterprise/instrumentations/profiling/getting-started) and set `profiling.enabled=true` so profiling is active on the cluster.

**2. Add a Pyroscope destination with Profiles**

In **Destinations**, add **Pyroscope** and enable the **Profiles** signal so the gateway forwards OTLP Profiles to Pyroscope as well as to the UI.

![Odigos Destinations list showing a Pyroscope destination with Profiles enabled](https://mintcdn.com/odigos/HulnWKhzpxgFDURc/images/enterprise/profiling/pyroscope-destinations-1.png?fit=max\&auto=format\&n=HulnWKhzpxgFDURc\&q=85\&s=1a151bbfa2639681f4cbfaf5432fdcfe)

**3. Configure the Pyroscope endpoint**

Set **Pyroscope URL** to the distributor HTTP API as `host:port` (for example `pyroscope.pyroscope:4040`). Do **not** add an `/otlp` prefix—Odigos sends OTLP Profiles to the path Pyroscope expects on that host.

![Destination detail showing Pyroscope URL and Profiles signal](https://mintcdn.com/odigos/HulnWKhzpxgFDURc/images/enterprise/profiling/pyroscope-destination-added-2.png?fit=max\&auto=format\&n=HulnWKhzpxgFDURc\&q=85\&s=4d526c8d311b00f976ce27a266828af3)

**4. Confirm profiles in Pyroscope**

After the gateway picks up the destination, open Pyroscope and confirm profile data for your services. Export does not depend on having a profiling slot open in the Odigos UI.

![Pyroscope explorer showing ingested profile data from Odigos](https://mintcdn.com/odigos/HulnWKhzpxgFDURc/images/enterprise/profiling/pyroscope-profiles-3.png?fit=max\&auto=format\&n=HulnWKhzpxgFDURc\&q=85\&s=71f344e170a97bc6d64ee1e979ec7f5f)

## Topology

The export path mirrors the rest of the Odigos pipeline:

```
eBPF profiler running as collector receiver (per node)
        │  OTLP Profiles
        ▼
Data Collector (per node)
        │  OTLP Profiles
        ▼
Gateway profiles pipeline
        ├──► Odigos UI buffer (on demand, in-memory flame graph)
        └──► Configured Destination (Pyroscope, or any OTLP backend supporting backend)
```

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Getting Started with Profiling
Source: https://docs.odigos.io/enterprise/instrumentations/profiling/getting-started

Enable Out of the box, dynamic, and in-memory continuous CPU profiling for your applications.

## Prerequisites

* Odigos **v1.25.0** or newer installed on the cluster
* The workload you want to profile must already be a [Source](/enterprise/pipeline/sources/introduction)

## Step 1: Enable Profiling

Profiling is an **opt-in** feature controlled by a single Helm value.
Set `profiling.enabled=true` when installing or upgrading Odigos (enterprise installs also require your on‑prem token, same as a normal Odigos install):

```bash theme={null}
helm upgrade --install odigos odigos/odigos \
  --namespace odigos-system \
  --create-namespace \
  --set onPremToken=$ODIGOS_ONPREM_TOKEN \
  --set profiling.enabled=true
```

Export `ODIGOS_ONPREM_TOKEN` in your shell before running the command (or replace the reference with your token value). For other Helm flags, see [Enterprise installation](/enterprise/setup/installation).

When this flag is set Odigos provisions the following automatically:

* The eBPF profiler receiver is activated on every Data Collector
* A `profiles` pipeline gets enabled in Odigos components and as soon as profile samples start to arrive, Odigos UI buffers start generating the flame graph
* You can export profiling data to destinations that ingest **OTLP Profiles** (see [Destinations](/enterprise/instrumentations/profiling/destinations))

> [!NOTE]
>
> Profiling does not replace or interfere with your existing traces, metrics, or logs pipelines. It runs as a separate pipeline inside the same collector processes.

## Step 2: View Profiles in the Odigos UI

Once profiling is enabled on the cluster, you can inspect profiles for any Source directly from the Odigos UI.

![Demo of continuous CPU profiling workflow in Odigos — enabling a slot, viewing the flame graph, and inspecting symbols](https://mintcdn.com/odigos/oddsQxX1gl4PNwT6/images/enterprise/profiling/profiling-demo.gif?s=6398bb3dbab0d95397cd95686485a165)

**1. Open the Profiling tab**

Navigate to **Sources** in the Odigos UI and click on the workload you want to investigate. This opens the source drawer. Inside the source drawer, click the **Profiling** tab. Click **Enable Profiling** to open a slot for this workload. Odigos will start buffering profile data as it arrives from the node.

![Source drawer showing the Profiling tab with flame graph and symbol table](https://mintcdn.com/odigos/oddsQxX1gl4PNwT6/images/enterprise/profiling/profiling4.png?fit=max\&auto=format\&n=oddsQxX1gl4PNwT6\&q=85\&s=8f5243e52b4dcf78a9d5a8cf11865a46)

**2. Inspect the flame graph**

Send traffic to your service or wait for background activity. The eBPF profiler samples continuously at the kernel level, higher CPU activity means more profiles getting collected for that workload.
The flame graph updates as profile data accumulates in the buffer. Wider bars represent more CPU time. Click any frame to zoom in.

**3. Inspect the symbols table**

The symbols table lists every function captured in the profile with its **Self** time (CPU spent exclusively in that function) and **Total** time (including all callees). Use it alongside the flame graph to quickly rank the top CPU consumers without navigating the call tree visually.

> [!NOTE]
>
> Closing the Profiling tab does **not** immediately destroy the slot. The slot and its buffered data persist until the TTL expires or the UI pod restarts. This is intentional, it lets you reopen the tab and see the same data during an ongoing investigation.

## Step 3: Send Profiles to a Backend (Optional)

The in-memory UI buffer is great for live investigation, but for long-term retention and cross-service comparison you'll want to route profiles to a profile-capable Destination. Today Odigos supports [Pyroscope](/enterprise/backends/pyroscope); see [Exporting Profiles to Destinations](/enterprise/instrumentations/profiling/destinations) for the full topology and prerequisites.

## Advanced: Buffer Tuning

The in-memory profile store is sized from **environment variables** read by the UI pod at startup.

| Environment variable | Default | Description |
| - | - | - |
| `PROFILES_MAX_SLOTS` | `24` | Maximum number of concurrently active profiling slots (one per workload). When the limit is reached the oldest slot is evicted (LRU). |
| `PROFILES_SLOT_TTL_SECONDS` | `120` | Seconds of inactivity after which a slot is swept by the background cleanup loop. |
| `PROFILES_SLOT_MAX_BYTES` | `8388608` (8 MiB) | Maximum bytes of raw OTLP profile chunks retained per slot. Older chunks are dropped when this cap is reached. |
| `PROFILES_CLEANUP_INTERVAL_SECONDS` | `15` | How often the background TTL sweep runs inside the UI process. |

## OTLP Exporter Tuning

If you need to adjust retry behaviour or compression on the profiling OTLP pipelines (node → gateway, gateway → UI), set the nested keys under `profiling.exporter` in your Helm values (defaults live in the chart’s [`values.yaml`](https://github.com/odigos-io/odigos/blob/main/helm/odigos/values.yaml) under the `profiling:` block):

```yaml theme={null}
profiling:
  enabled: true
  exporter:
    enableDataCompression: false
    timeout: 5s
    retryOnFailure:
      enabled: true
      initialInterval: 5s
      maxInterval: 30s
      maxElapsedTime: 300s
```

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Continuous Profiling
Source: https://docs.odigos.io/enterprise/instrumentations/profiling/overview

Continuous CPU profiling in Odigos: how it works, and how it can helps finding performance bottlenecks without code changes.

## What is CPU Profiling?

CPU profiling continuously samples what your processes are spending time on. Every few milliseconds the profiler captures a snapshot of every running thread's call stack. Aggregated over time, these snapshots form a **flame graph** — a visualization that makes it immediately obvious where CPU cycles are going, which functions are hot, and which library call is the unexpected bottleneck.

Unlike traces, which tell you *that* a request was slow, a CPU profile tells you *why*: the exact code path consuming the most time. This makes profiling the natural next step after you've identified a latency problem in your trace data.

![Continuous CPU Profiling in Odigos — symbol table and flame graph view](https://mintcdn.com/odigos/oddsQxX1gl4PNwT6/images/enterprise/profiling/profiling1.png?fit=max\&auto=format\&n=oddsQxX1gl4PNwT6\&q=85\&s=fe4f3617d55ff9b8fce48b902d4b0f29)

## Why Profiling in Odigos?

Continuous profiling in Odigos is built directly into the same pipeline that handles your traces, metrics, and logs:

* **Zero-code changes.** Profiling uses an eBPF kernel agent. No SDK, no agent library, no application restart required.
* **Language-agnostic.** Because collection happens at the OS level, every process on the node is profiled regardless of runtime — Java, Python, Ruby, PHP, Node.js, Perl, Erlang, .NET, Go, and Native executables (including C/C++, Rust, and Zig)
* **No separate infrastructure.** Profiles flow through the Odigos collector pipeline you already run. For quick investigation workflows the Odigos UI holds profiles in memory; for long-term storage you route them to a destination backend.
* **Per-workload, on-demand.** After enabling profiling, it works efficiently by only dynamically enabling in-memory buffer for workloads you are actively investigating, keeping Odigos resource usage predictable and under control.

## How It Works

Odigos integrates the upstream [OpenTelemetry eBPF Profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) as a receiver inside the Data Collector.
Profiles are emitted as OTLP Profiles signals and travel through the same collector pipeline as your other telemetry.

Odigos doesn't require you to provision any data storage layer to view profiling data live, it does that by creating an efficient **in-memory and per workload** cache which maintains the lifecycle of profiling data based on cache size and TTL expiry.

## Learn More

> **[Continuous Profiling in Odigos](https://odigos.io/blog/continuous-profiling-in-odigos)**
>
> Deep-dive blog post on how profiling was built into the Odigos pipeline.

> **[Export to Pyroscope](/enterprise/backends/pyroscope)**
>
> Send profile data to Grafana Pyroscope for long-term retention and cross-service analysis.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/python/enrichment



Odigos will automatically instrument your Python services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Install the API from PyPI using pip:

```bash theme={null}
pip install opentelemetry-api
```

## Creating Spans

To create a span, use the `tracer` object from the `opentelemetry.trace` module. The `tracer` object is a factory for creating spans.

```python theme={null}
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def my_function():
   with tracer.start_as_current_span("my_function") as span:
      print("Hello world!")
```

Important Notes:

1. **Assign meaningful names to spans**:
   Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context.
2. **Avoid dynamic, high cardinality data in span names**:
   Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

### Recording Span Attributes

Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting

Examples:

* User ID, organization ID, Account ID or other identifiers.
* Inputs - the relevant parameters or configuration that influenced the operation.
* Outputs - the result of the operation.
* Entities - the entities or resources that the operation interacted with.

Attribute names are lowercased strings in the form `my_application.my_attribute`, example: `my_service.user_id`.
Read more [here](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/)

To record attributes, use the `set_attribute` method on the span object.

```python theme={null}
def my_function(arg: str):
   with tracer.start_as_current_span("my_function") as span:
      span.set_attribute("argument_name", arg)
```

Important Notes:

1. **Be cautious when recording data**:
   Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
2. **Attribute cost considerations**:
   Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
3. **Use static names for attributes**:
   Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute\_name) to provide clear context for downstream consumers.
4. **Adhere to OpenTelemetry semantic conventions**:
   Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.

### Recording Errors

To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts.

If an operation you are tracing fails, you can mark the span's status as an error and record the error details within the span. Here's how you can do it:

* An exception has been raised to demonstrate an error that occurred in your code.

```python theme={null}
def my_function():
   with tracer.start_as_current_span("my_function") as span:
      try:
         print("Hello world!")
         raise Exception("Some Exception")
      except Exception as e:
         span.record_exception(e)
         span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
```

## Advanced Capabilities

The Odigos Python agent extends the standard OpenTelemetry SDK with capabilities that apply to **all spans** — including custom spans you create with the API above. These features are configured remotely through OpAMP and take effect without restarting your application.

### Head Sampling

The Python agent supports [head sampling](../../pipeline/sampling/head-and-tail-sampling#head-sampling-agent) at the root span. Odigos pushes [Noisy Operation](../../pipeline/sampling/rules/noisy-operation) rules to the agent, which evaluates them through a custom `OdigosSampler` wired into the `TracerProvider`.

Custom spans participate in the same sampling pipeline as auto-instrumented spans:

* **Root spans** — If your custom span is the entry point of a trace (no active parent), head-sampling rules can match it and decide whether the entire trace is kept or dropped.
* **Child spans** — Follow parent-based sampling: if the parent trace is sampled, custom child spans are recorded; if not, they are dropped (or recorded for metrics only — see [Span metrics mode](#span-metrics-mode) below).
* **Context propagation** — Use `tracer.start_as_current_span(...)` (as shown above) so custom spans attach to the active trace and inherit the sampling decision.

Head-sampling rules match HTTP server spans (`SpanKind.SERVER`) and HTTP client spans (`SpanKind.CLIENT`) using OpenTelemetry semantic conventions. The sampler accepts both stable and legacy attribute names — for example `http.request.method` and `http.method`, `server.address` and `http.host`. Server rules match on route (with `{param}` wildcard support), route prefix, and method. Client rules match on server address, templated path, path prefix, and method.

When multiple rules match, the **lowest keep percentage** wins. Even when a trace is dropped, the agent attaches an `odigos` [tracestate](https://www.w3.org/TR/trace-context/#tracestate-header) entry so downstream services and the Odigos tail sampler can coordinate on the same decision.

For rule configuration and evaluation order, see [Sampling rules overview](../../pipeline/sampling/rules/overview).

### Dry Run Mode

When cluster-wide dry run (`samplingDryRun` in [OdigosConfiguration](../../api-reference/odigos.io.v1alpha1)) is enabled, the Python agent still exports every span but annotates the sampling decision in tracestate (`dry:t` if the trace would have been kept, `dry:f` if it would have been dropped). Use dry run to validate Noisy rules before committing to real drops.

### Span Metrics Mode

The global `spanMetricsMode` setting controls what happens to spans that head sampling marks as not sampled:

| Mode | Not-sampled spans |
| - | - |
| `sampled-spans-only` (default) | Dropped entirely — no trace export, no span metrics |
| `all-spans` | Recorded with `RECORD_ONLY` — excluded from trace export but forwarded for span metrics computation |

This applies to custom spans the same way it applies to auto-instrumented spans. If you rely on span metrics for RED dashboards, set `spanMetricsMode: all-spans` in your Odigos configuration so unsampled traffic still contributes to rate and error metrics.

### Agent Diagnostics

Debug logging for the Odigos Python agent and its OpenTelemetry components can be toggled at runtime through an [InstrumentationRule](../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-InstrumentationRule) with `agentDiagnostics`:

```yaml theme={null}
spec:
  agentDiagnostics:
    odigosLogLevel: debug
    openTelemetryComponentsLogLevel: debug
```

Supported levels are `error`, `warn`, `info`, and `debug`. Remote config from OpAMP **fully overrides** the startup environment variables — if a level is not set in the remote config, that logger is silenced.

For the window before the first OpAMP message arrives, you can set startup log levels with environment variables:

| Variable | Logger | Values |
| - | - | - |
| `ODIGOS_LOG_LEVEL` | Odigos agent (`odigos`) | `ALL`, `VERBOSE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `NONE` |
| `OTEL_LOG_LEVEL` | OpenTelemetry SDK components | Same as above |

At `debug` level, the sampler logs each rule evaluation — useful when troubleshooting why a custom or auto-instrumented span was kept or dropped.

## Additional Information

For more use cases, see the [OpenTelemetry Python API documentation](https://opentelemetry.io/docs/languages/python/instrumentation/#creating-spans).


# Automatic Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/python/native



> [!NOTE]
>
> Native Instrumentation is currently the default setting of Odigos **Open-Source** version.
>
> eBPF-Based Instrumentation is currently not supported for this language.

## Supported Versions

Odigos uses the official [opentelemetry-python-instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation) OpenTelemetry Auto Instrumentation, thus it supports the same Python versions as this project.

* In the enterprise version, Odigos leverages eBPF to enhance performance in the Python instrumentation process.

* Odigos ships **two separate Python distros** — one for Python 3.8 and one for Python 3.9 and above. Odigos selects the correct distro automatically based on the runtime version of your workload.

* **Python 3.8** uses a dedicated legacy distro with OpenTelemetry instrumentation `0.54b1`.

* **Python 3.9 through 3.14** uses the main distro with the latest OpenTelemetry instrumentation.

## Traces

Odigos will automatically instrument your Python services to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Python official auto Instrumentation Libraries.

## Instrumentation Libraries

The following Python modules will be auto instrumented by Odigos. Supported libraries differ between the Python 3.8 and Python 3.9+ distros:

**Python 3.8**

The Python 3.8 distro includes the libraries below. It does not include Gen AI instrumentations other than OpenAI.

### Database Clients, ORMs, and Data Access Libraries

* [`aiopg`](https://pypi.org/project/aiopg/) versions `aiopg >= 0.13.0, < 2.0.0`
* [`dbapi`](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/dbapi/dbapi.html)
* [`mysql`](https://pypi.org/project/mysql-connector-python/) version `mysql-connector-python >= 8.0, < 10.0`
* [`mysqlclient`](https://pypi.org/project/mysqlclient/) version `mysqlclient < 3`
* [`psycopg`](https://pypi.org/project/psycopg/) versions `psycopg >= 3.1.0`
* [`psycopg2`](https://pypi.org/project/psycopg2/) versions `psycopg2 >= 2.7.3.1`, `psycopg2-binary >= 2.7.3.1`
* [`pymemcache`](https://pypi.org/project/pymemcache/) versions `pymemcache >= 1.3.5, < 5`
* [`pymssql`](https://pypi.org/project/pymssql/) versions `pymssql >= 2.1.5, < 3`
* [`pymongo`](https://pypi.org/project/pymongo/) versions `pymongo >= 3.1, < 5.0`
* [`pymysql`](https://pypi.org/project/PyMySQL/) versions `PyMySQL < 2`
* [`redis`](https://pypi.org/project/redis/) versions `redis >= 2.6`
* [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) versions `sqlalchemy >= 1.0.0, < 2.1.0`
* [`sqlite3`](https://docs.python.org/3/library/sqlite3.html)
* [`tortoiseorm`](https://pypi.org/project/tortoise-orm/) versions `tortoise-orm >= 0.17.0`
* [`cassandra`](https://pypi.org/project/cassandra-driver/) versions `cassandra-driver ~= 3.25`, `scylla-driver ~= 3.25`
* [`elasticsearch`](https://pypi.org/project/elasticsearch/) versions `elasticsearch >= 6.0`
* [`asyncpg`](https://pypi.org/project/asyncpg/) versions `asyncpg >= 0.12.0`

### HTTP Frameworks

* [`asgi`](https://pypi.org/project/asgiref/) versions `asgiref ~= 3.0`
* [`django`](https://pypi.org/project/Django/) versions `django >= 1.10`
  > [!NOTE]
  >
  > Please note that for Django instrumentation to work, you need to set the
  > [DJANGO\_SETTINGS\_MODULE](https://docs.djangoproject.com/en/5.1/topics/settings/#envvar-DJANGO_SETTINGS_MODULE) environment variable. Make sure to
  > set this in your Kubernetes manifest or Dockerfile to ensure proper configuration.
* [`fastapi`](https://pypi.org/project/fastapi/) versions `fastapi ~= 0.58`
* [`flask`](https://pypi.org/project/Flask/) versions `flask >= 1.0`
* [`pyramid`](https://pypi.org/project/pyramid/) versions `pyramid >= 1.7`
* [`starlette`](https://pypi.org/project/starlette/) versions `starlette >= 0.13, < 0.15`
* [`falcon`](https://pypi.org/project/falcon/) versions `falcon >= 1.4.1, < 5.0.0`
* [`tornado`](https://pypi.org/project/tornado/) versions `tornado >= 5.1.1`

### HTTP Clients

* [`aiohttp-client`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`httpx`](https://pypi.org/project/httpx/) versions `httpx >= 0.18.0`
* [`requests`](https://pypi.org/project/requests/) versions `requests ~= 2.0`
* [`urllib`](https://docs.python.org/3/library/urllib.html)
* [`urllib3`](https://pypi.org/project/urllib3/) versions `urllib3 >= 1.0.0, < 3.0.0`

### Messaging Systems Clients

* [`aio-pika`](https://pypi.org/project/aio-pika/) versions `aio_pika >= 7.2.0, < 10.0.0`
* [`celery`](https://pypi.org/project/celery/) versions `celery >= 4.0, < 6.0`
* [`confluent-kafka`](https://pypi.org/project/confluent-kafka/) versions `confluent-kafka >= 1.8.2, <= 2.7.0`
* [`kafka-python`](https://pypi.org/project/kafka-python/) versions `kafka-python >= 2.0, < 3.0`, `kafka-python-ng >= 2.0, < 3.0`
* [`pika`](https://pypi.org/project/pika/) versions `pika >= 0.12.0`
* [`remoulade`](https://pypi.org/project/remoulade/) versions `remoulade >= 0.50`

### RPC (Remote Procedure Call)

* [`grpc`](https://pypi.org/project/grpcio/) versions `grpcio >= 1.42.0`

### Web Servers

* [`aiohttp-server`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`wsgi`](https://docs.python.org/3/library/wsgiref.html)

### Cloud Services and SDKs

* [`boto`](https://pypi.org/project/boto/) versions `boto ~= 2.0`
* [`boto3sqs`](https://pypi.org/project/boto3/) versions `boto3 ~= 1.0`
* [`botocore`](https://pypi.org/project/botocore/) versions `botocore ~= 1.0`

### Framework and Library Utilities

* [`jinja2`](https://pypi.org/project/Jinja2/) versions `jinja2 >= 2.7, < 4.0`

### Other

* [`asyncio`](https://pypi.org/project/asyncio/)
* [`click`](https://pypi.org/project/click/) versions `click >= 8.1.3, < 9.0.0`
* [`threading`](https://docs.python.org/3/library/threading.html)

### Gen AI

* [`openai`](https://pypi.org/project/openai/) versions `openai >= 1.26.0`

### Loggers

Automatic injection of trace context (trace id and span id) into log records for the following loggers:

* [`logging`](https://docs.python.org/3/library/logging.html)

**Python 3.9+**

The Python 3.9+ distro includes the libraries below, including additional Gen AI instrumentations not available on Python 3.8.

### Database Clients, ORMs, and Data Access Libraries

* [`aiopg`](https://pypi.org/project/aiopg/) versions `aiopg >= 0.13.0, < 2.0.0`
* [`dbapi`](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/dbapi/dbapi.html)
* [`mysql`](https://pypi.org/project/mysql-connector-python/) version `mysql-connector-python >= 8.0, < 10.0`
* [`mysqlclient`](https://pypi.org/project/mysqlclient/) version `mysqlclient < 3`
* [`psycopg`](https://pypi.org/project/psycopg/) versions `psycopg >= 3.1.0`
* [`psycopg2`](https://pypi.org/project/psycopg2/) versions `psycopg2 >= 2.7.3.1`, `psycopg2-binary >= 2.7.3.1`
* [`pymemcache`](https://pypi.org/project/pymemcache/) versions `pymemcache >= 1.3.5, < 5`
* [`pymssql`](https://pypi.org/project/pymssql/) versions `pymssql >= 2.1.5, < 3`
* [`pymongo`](https://pypi.org/project/pymongo/) versions `pymongo >= 3.1, < 5.0`
* [`pymysql`](https://pypi.org/project/PyMySQL/) versions `PyMySQL < 2`
* [`redis`](https://pypi.org/project/redis/) versions `redis >= 2.6`
* [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) versions `sqlalchemy >= 1.0.0, < 2.1.0`
* [`sqlite3`](https://docs.python.org/3/library/sqlite3.html)
* [`tortoiseorm`](https://pypi.org/project/tortoise-orm/) versions `tortoise-orm >= 0.17.0`
* [`cassandra`](https://pypi.org/project/cassandra-driver/) versions `cassandra-driver ~= 3.25`, `scylla-driver ~= 3.25`
* [`elasticsearch`](https://pypi.org/project/elasticsearch/) versions `elasticsearch >= 6.0`
* [`asyncpg`](https://pypi.org/project/asyncpg/) versions `asyncpg >= 0.12.0`

### HTTP Frameworks

* [`asgi`](https://pypi.org/project/asgiref/) versions `asgiref ~= 3.0`
* [`django`](https://pypi.org/project/Django/) versions `django >= 2.0`
  > [!NOTE]
  >
  > Please note that for Django instrumentation to work, you need to set the
  > [DJANGO\_SETTINGS\_MODULE](https://docs.djangoproject.com/en/5.1/topics/settings/#envvar-DJANGO_SETTINGS_MODULE) environment variable. Make sure to
  > set this in your Kubernetes manifest or Dockerfile to ensure proper configuration.
* [`fastapi`](https://pypi.org/project/fastapi/) versions `fastapi ~= 0.92`
* [`flask`](https://pypi.org/project/Flask/) versions `flask >= 1.0`
* [`pyramid`](https://pypi.org/project/pyramid/) versions `pyramid >= 1.7`
* [`starlette`](https://pypi.org/project/starlette/) versions `starlette >= 0.13`
* [`falcon`](https://pypi.org/project/falcon/) versions `falcon >= 1.4.1, < 5.0.0`
* [`tornado`](https://pypi.org/project/tornado/) versions `tornado >= 5.1.1`

### HTTP Clients

* [`aiohttp-client`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`httpx`](https://pypi.org/project/httpx/) versions `httpx >= 0.18.0`
* [`requests`](https://pypi.org/project/requests/) versions `requests ~= 2.0`
* [`urllib`](https://docs.python.org/3/library/urllib.html)
* [`urllib3`](https://pypi.org/project/urllib3/) versions `urllib3 >= 1.0.0, < 3.0.0`

### Messaging Systems Clients

* [`aio-pika`](https://pypi.org/project/aio-pika/) versions `aio_pika >= 7.2.0, < 10.0.0`
* [`celery`](https://pypi.org/project/celery/) versions `celery >= 4.0, < 6.0`
* [`confluent-kafka`](https://pypi.org/project/confluent-kafka/) versions `confluent-kafka >= 1.8.2, < 3.0.0`
* [`kafka-python`](https://pypi.org/project/kafka-python/) versions `kafka-python >= 2.0, < 3.0`, `kafka-python-ng >= 2.0, < 3.0`
* [`pika`](https://pypi.org/project/pika/) versions `pika >= 0.12.0`
* [`remoulade`](https://pypi.org/project/remoulade/) versions `remoulade >= 0.50`

### RPC (Remote Procedure Call)

* [`grpc`](https://pypi.org/project/grpcio/) versions `grpcio >= 1.42.0`

### Web Servers

* [`aiohttp-server`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`wsgi`](https://docs.python.org/3/library/wsgiref.html)

### Cloud Services and SDKs

* [`boto3sqs`](https://pypi.org/project/boto3/) versions `boto3 ~= 1.0`
* [`botocore`](https://pypi.org/project/botocore/) versions `botocore ~= 1.0`, `aiobotocore ~= 2.0`

### Framework and Library Utilities

* [`jinja2`](https://pypi.org/project/Jinja2/) versions `jinja2 >= 2.7, < 4.0`

### Other

* [`asyncio`](https://pypi.org/project/asyncio/)
* [`click`](https://pypi.org/project/click/) versions `click >= 8.1.3, < 9.0.0`
* [`threading`](https://docs.python.org/3/library/threading.html)

### Gen AI

* [`anthropic`](https://pypi.org/project/anthropic/) versions `anthropic >= 0.3.11`
* [`google-genai`](https://pypi.org/project/google-genai/) versions `google-genai >= 1.32.0`
* [`langchain`](https://pypi.org/project/langchain-core/) versions `langchain-core > 0.1.0`
* [`openai-agents`](https://pypi.org/project/openai-agents/) versions `openai-agents >= 0.3.3`
* [`openai`](https://pypi.org/project/openai/) versions `openai >= 1.26.0`
* [`vertexai`](https://pypi.org/project/google-cloud-aiplatform/) versions `google-cloud-aiplatform >= 1.64`
* [`weaviate`](https://pypi.org/project/weaviate-client/) versions `weaviate-client >= 3.26.0, < 5`

### Loggers

Automatic injection of trace context (trace id and span id) into log records for the following loggers:

* [`logging`](https://docs.python.org/3/library/logging.html)


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/enterprise/instrumentations/ruby/enrichment



Odigos will automatically instrument your Ruby services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project by running:

```bash theme={null}
gem install opentelemetry-sdk
```

## Creating a Span

To acquire a tracer:

```ruby theme={null}
# If in a Rails app, this lives in config/initializers/opentelemetry.rb
require "opentelemetry/sdk"

OpenTelemetry::SDK.configure do |c|
  c.service_name = '<YOUR_SERVICE_NAME>'
end

# 'MyAppTracer' can be used throughout your code now
MyAppTracer = OpenTelemetry.tracer_provider.tracer('<YOUR_TRACER_NAME>')
```

To create a new span:

```ruby theme={null}
require "opentelemetry/sdk"

def do_work
  MyAppTracer.in_span("do_work") do |span|
    # do some work that the 'do_work' span tracks!
  end
end
```

## Additional Information

For more use cases, see the [OpenTelemetry Ruby API documentation](https://opentelemetry.io/docs/languages/ruby).


# Automatic Instrumentation
Source: https://docs.odigos.io/enterprise/instrumentations/ruby/native



> [!NOTE]
>
> This language instrumentation is using OpenTelemetry native-ruby SDK and
> instrumentation libraries.

## Supported Versions

The minimal supported version of Ruby is **3.1.0**.
The maximal supported version of Ruby is **3.5.x**, when Ruby releases a new version we'll add the support for it.

## Instrumentation Libraries

The following Ruby gems will be auto instrumented by Odigos:

### HTTP Clients:

* [`http`](https://rubygems.org/gems/http) versions `*`
* [`httpclient`](https://rubygems.org/gems/httpclient) versions `*`
* [`restclient`](https://rubygems.org/gems/restclient) versions `*`
* [`net-http`](https://rubygems.org/gems/net-http) versions `*`
* [`faraday`](https://rubygems.org/gems/faraday) versions `*`
* [`excon`](https://rubygems.org/gems/excon) versions `*`
* [`ethon`](https://rubygems.org/gems/ethon) versions `*`

### gRPC Clients:

* [`grpc`](https://rubygems.org/gems/grpc) versions `*`

### Web Frameworks & APIs:

* [`rails`](https://rubygems.org/gems/rails) versions `*`
* [`rack`](https://rubygems.org/gems/rack) versions `*`
* [`sinatra`](https://rubygems.org/gems/sinatra) versions `*`
* [`grape`](https://rubygems.org/gems/grape) versions `*`
* [`gruf` (gRPC framework for Ruby)](https://rubygems.org/gems/gruf) versions `*`
* [`racecar` (Kafka consumer framework for Rails)](https://rubygems.org/gems/racecar) versions `*`

### Processing & Task Runners:

* [`rake`](https://rubygems.org/gems/rake) versions `>=13.0`
* [`concurrent-ruby` (Threading and scheduling utilities)](https://rubygems.org/gems/concurrent-ruby) versions `*`

### Query & Schema APIs:

* [`graphql`](https://rubygems.org/gems/graphql) versions `*`
* [`koala` (Facebook Graph API client)](https://rubygems.org/gems/koala) versions `*`

### Messaging & Background Jobs:

* [`sidekiq`](https://rubygems.org/gems/sidekiq) versions `*`
* [`resque`](https://rubygems.org/gems/resque) versions `*`
* [`delayed_job`](https://rubygems.org/gems/delayed_job) versions `*`
* [`que`](https://rubygems.org/gems/que) versions `*`
* [`bunny` (RabbitMQ)](https://rubygems.org/gems/bunny) versions `*`
* [`ruby-kafka`](https://rubygems.org/gems/ruby-kafka) versions `*`
* [`rdkafka`](https://rubygems.org/gems/rdkafka) versions `*`

### Databases & Caches:

* [`pg` (PostgreSQL)](https://rubygems.org/gems/pg) versions `*`
* [`mysql2`](https://rubygems.org/gems/mysql2) versions `*`
* [`trilogy` (MySQL client by GitHub)](https://rubygems.org/gems/trilogy) versions `*`
* [`mongo`](https://rubygems.org/gems/mongo) versions `*`
* [`redis`](https://rubygems.org/gems/redis) versions `*`
* [`dalli` (Memcached)](https://rubygems.org/gems/dalli) versions `*`
* [`lmdb` (Lightning Memory-Mapped Database)](https://rubygems.org/gems/lmdb) versions `*`

### Cloud SDKs:

* [`aws-sdk`](https://rubygems.org/gems/aws-sdk) versions `*`
* [`aws-sdk-lambda`](https://rubygems.org/gems/aws-sdk-lambda) versions `*`

### Serialization:

* [`active_model_serializers`](https://rubygems.org/gems/active_model_serializers) versions `>=0.10.0`


# Enterprise MCP Server
Source: https://docs.odigos.io/enterprise/mcp/overview

Explore the MCP tools and MCP servers provided by Odigos — get insights and configure your cluster at your fingertips.

Odigos MCP lets AI agents (Cursor, Claude Desktop, and others) inspect and
configure a live Odigos cluster. The endpoint is mounted at `/mcp` on the UI
service (port `3000`).

## Requirements

* Odigos Enterprise installed with a valid on-prem token
* Network access to the Odigos UI service (typically via port-forward or an
  in-cluster / load-balancer URL)
* An MCP-compatible client (for example, Cursor)

## Connect from Cursor

The simplest way to connect locally is to port-forward the UI service, then
point your MCP client at the `/mcp` endpoint.

**1. Port-forward the UI**

```bash theme={null}
kubectl port-forward -n odigos-system svc/ui 3000:3000
```

**2. Add the MCP server in Cursor**

Open **Cursor Settings → MCP** and add a server with a streamable HTTP URL:

```json theme={null}
{
  "mcpServers": {
    "odigos": {
      "url": "http://localhost:3000/mcp"
    }
  }
}
```

For a remote UI (for example, behind a load balancer), use that base URL
instead: `https://<your-ui-host>/mcp`.

**3. Verify the connection**

In your AI agent's MCP tool registry, open the MCP server connections — the
Odigos server should be listed along with its MCP tools.

**4. Ask your cluster a question**

Query Odigos directly through MCP, for example:
*"Which sources are unhealthy in the `default` namespace?"*

## Access modes

The MCP server starts in **read-only** mode by default. Read operations always
work; write operations are blocked until an operator switches it to
**read-write** (see [Configuration](#configuration)).

| Mode | Behavior |
| - | - |
| `read-only` (default) | Safe inspection only. Write requests are rejected. |
| `read-write` | Mutating operations are allowed. Use only when you trust the agent to change the cluster. |

This setting is **independent** of the UI's read-only mode (`ui.uiMode`).

## Configuration

Control MCP behavior during install or upgrade:

**Helm**

```yaml theme={null}
ui:
  mcp:
    enabled: true
  mcpAccessMode: read-only   # or read-write
```

```bash theme={null}
helm upgrade odigos odigos/odigos \
  --namespace odigos-system \
  --reuse-values \
  --set ui.mcpAccessMode=read-write
```

**CLI**

```bash theme={null}
odigos install --set ui.mcpAccessMode=read-write
```

Set `ui.mcp.enabled: false` to disable the `/mcp` endpoint entirely.

## What you can do

Odigos supports a growing list of both low-level and high-level MCP tools to
confidently and securely access your clusters. They are grouped by domain —
common starting points:

| Goal | Tools to try |
| - | - |
| Is Odigos healthy? | `describe_odigos`, `get_gateway_info`, `get_odiglet_info` |
| What is instrumented? | `list_sources`, `list_workloads_in_namespace` |
| Why is a workload not sending data? | `describe_source`, `get_runtime_detection`, `get_instrumentation_health` |
| How much data is flowing? | `get_overview_metrics`, `get_service_map` |
| Configure a backend | `list_destination_types`, `get_destination_schema`, `create_destination` |
| Tune sampling | `get_sampling`, `create_sampling_group`, `create_noisy_operation_rule` |
| Profile a hot path | `enable_source_profiling`, `get_source_hot_functions`, `add_custom_instrumentation` |
| Collect support bundle | `collect_diagnose_bundle` |

Write tools support `dry_run: true` (the default) so agents can preview
changes before applying them. When the server is in read-only mode, write tools
are not executed even with `dry_run: false`.

> [!NOTE]
>
> Tool responses include `context` and `next_steps` hints to help agents chain
> calls without manual guidance.

## Audit logs

Every MCP tool call is audit-logged by the Odigos UI pod. The audit middleware
records the tool name, outcome, and duration on every `tools/call` — including
calls blocked in read-only mode.

| Field | Description |
| - | - |
| `tool` | MCP tool name (for example `describe_odigos`, `create_destination`) |
| `outcome` | `ok` — succeeded · `tool_error` — handler returned an error result (includes writes rejected in read-only mode) · `error` — unexpected failure |
| `duration_ms` | Wall-clock time for the call, in milliseconds |

Example log lines from the UI pod (`kubectl logs -n odigos-system deploy/odigos-ui`):

```json theme={null}
{"level":"info","ts":"2026-06-18T11:42:03.512Z","caller":"audit/audit.go:37","msg":"mcp tool call","component":"mcp","tool":"describe_odigos","outcome":"ok","duration_ms":38}
{"level":"info","ts":"2026-06-18T11:42:18.904Z","caller":"audit/audit.go:37","msg":"mcp tool call","component":"mcp","tool":"create_destination","outcome":"ok","duration_ms":214}
{"level":"info","ts":"2026-06-18T11:43:01.271Z","caller":"audit/audit.go:37","msg":"mcp tool call","component":"mcp","tool":"enable_source_profiling","outcome":"tool_error","duration_ms":1}
```

In the third line, `enable_source_profiling` was rejected because the server was
in read-only mode — the call is still logged, but with `outcome: tool_error`
instead of `ok`.

## Security notes

* The MCP endpoint is intended for **in-cluster or trusted network** use. There
  is no built-in authentication on `/mcp` in the current release — protect
  access at the network layer (port-forward, private ingress, VPN).
* Prefer **read-only** mode for day-to-day agent use. Switch to read-write only
  when an operator is actively supervising changes.

## What's coming

* **Fine-grained access control via token** — scoped, per-token MCP permissions,
  so you can grant each agent only the tools it needs instead of a single
  cluster-wide access mode.

Until then, the recommended approach is to issue a dedicated on-prem token per
agent, keep the server in **read-only** mode for untrusted agents, and restrict
who can reach `/mcp` at the network layer.


# Overview
Source: https://docs.odigos.io/enterprise/overview



Odigos is an observability control plane. Odigos leverages the power of [OpenTelemetry](https://opentelemetry.io) and [eBPF](https://ebpf.io) to automatically instrument applications.

Odigos automatically deploys and scales collectors according to the traffic of applications. No need to waste time deploying and configuring collectors.

> [!TIP]
>
> The name Odigos originates from Greek, meaning "guide".

## Goals

* **Automatic instrumentation** - Odigos automatically instruments your applications and produces distributed traces and metrics **without any code changes**.
* **Collector management** - Odigos automatically deploys and scales collectors according to application traffic. No need to spend time deploying and configuring collectors.

Odigos achieves its goals by deploying a set of components that work together to collect, process, and export telemetry data.

## Who is Odigos for?

* Software Developers
* Platform Engineers
* IT Professionals
* Anyone comfortable with command-line interfaces and basic cloud infrastructure

> [!NOTE]
>
> Odigos is designed to be easy to use and deploy. It is a great tool for anyone who wants to get started with observability without the hassle of setting up and managing collectors.

## Adopt OpenTelemetry in minutes

Odigos currently supports all the popular managed and open source [destinations](./backends-overview). By producing data in the [OpenTelemetry](https://opentelemetry.io) format, Odigos can be used with any observability tool that supports OTLP and allow you to easily switch to another vendor in the future.

> **[Getting Started](./quickstart/introduction)**
>
> Use Odigos to generate distributed traces, metrics and logs with OpenTelemetry.

> **[Telemetry Types](./telemetry-types)**
>
> Telemetry types that Odigos supports and produce.

> **[Supported Languages](./instrumentations/overview)**
>
> Programming languages that Odigos supports and applies auto-instrumentation to.

> **[Supported Destinations](./adding-new-dest)**
>
> Destinations that Odigos supports and sends telemetry data to.


# Kubernetes RBAC Permissions
Source: https://docs.odigos.io/enterprise/permissions



This page lists the Kubernetes Roles and ClusterRoles used by Odigos and the Odigos Operator.

# Components

This section lists the RBAC policies used by the Odigos components.

## ClusterRoles

Below are the ClusterRoles used by Odigos components.

### cleanup-clusterrole

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| odigos.io | sources | \* | list delete |
| \* | pods | \* | list |
| \* | nodes | \* | list patch |

### odigos-gateway

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| odigos.io | instrumentationconfigs | \* | get list watch |
| \* | pods namespaces | \* | get list watch |
| apps | replicasets deployments statefulsets daemonsets | \* | get list watch |

### odigos-go-offsets-updater

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | list |

### odigos-instrumentor

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | nodes | \* | list watch get |
| \* | namespaces | \* | list watch get |
| \* | pods | \* | list watch get |
| batch | cronjobs | \* | list watch get |
| apps | daemonsets | \* | get list watch update patch |
| apps | deployments | \* | get list watch update patch |
| apps | statefulsets | \* | get list watch update patch |
| \* | pods/finalizers configmaps/finalizers | \* | update |
| apps | statefulsets/finalizers deployments/finalizers daemonsets/finalizers | \* | update |
| batch | cronjobs/finalizers jobs/finalizers | \* | update |
| argoproj.io | rollouts/finalizers | \* | update |
| apps.openshift.io | deploymentconfigs deploymentconfigs/finalizers | \* | get list watch update patch |
| argoproj.io | rollouts | \* | get list watch patch |
| operator.odigos.io | odigos/finalizers | \* | update |
| odigos.io | instrumentationconfigs/status | \* | get patch update |
| odigos.io | instrumentationconfigs | \* | create delete get list patch update watch |
| odigos.io | sources | \* | create delete get list patch update watch |
| odigos.io | sources/finalizers | \* | update |
| admissionregistration.k8s.io | mutatingwebhookconfigurations | \* | get list watch |
| admissionregistration.k8s.io | mutatingwebhookconfigurations | odigos-source-mutating-webhook-configuration odigos-pod-mutating-webhook-configuration | update |
| admissionregistration.k8s.io | validatingwebhookconfigurations | \* | get list watch |
| admissionregistration.k8s.io | validatingwebhookconfigurations | odigos-source-validating-webhook-configuration odigos-action-validating-webhook-configuration | update |
| apiregistration.k8s.io | apiservices | \* | get list watch update patch delete |

### odiglet

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | pods services | \* | get list watch |
| \* | pods/status | \* | get |
| \* | pods/finalizers | \* | update |
| \* | nodes | \* | get list watch patch update |
| odigos.io | instrumentationinstances | \* | create get list patch update watch delete |
| odigos.io | instrumentationinstances/status | \* | get patch update |
| odigos.io | instrumentationconfigs | \* | get list watch patch update |
| odigos.io | instrumentationconfigs/status | \* | get patch update |
| \* | nodes/stats | \* | get list |
| \* | pods namespaces | \* | get list watch |
| apps | replicasets deployments daemonsets statefulsets | \* | get list watch |
| discovery.k8s.io | endpointslices | \* | get list watch |

### odigos-ui

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | namespaces | \* | get list watch patch |
| apps | deployments statefulsets daemonsets | \* | get list watch update patch |
| batch | cronjobs | \* | get list watch update patch |
| apps.openshift.io | deploymentconfigs | \* | get list watch update patch |
| argoproj.io | rollouts | \* | get list watch update patch |
| apps | replicasets | \* | get list |
| \* | services | \* | get list |
| \* | pods | \* | get list watch delete |
| odigos.io | \* | \* | get list watch |
| odigos.io | instrumentationconfigs instrumentationinstances sources samplings | \* | update patch create delete |
| operator.odigos.io | odigos | \* | get list watch |
| actions.odigos.io | \* | \* | get list watch |

## Roles

Below are the Roles used by Odigos components. These Roles are only scoped to the Namespace in which Odigos is installed.

### cleanup-role

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | odigos-deployment odigos-configuration | get |
| \* | configmaps | \* | list |

### odigos-gateway

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | odigos-gateway | get list watch |

### odigos-go-offsets-updater

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | odigos-go-offsets | get update |
| \* | secrets | odigos-pro | get |

### odigos-instrumentor

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | get list watch create patch update delete |
| \* | services | \* | get list watch create patch update delete |
| \* | pods | \* | get list watch |
| apps | deployments | \* | create delete get list patch update watch |
| apps | deployments/status | \* | get |
| autoscaling | horizontalpodautoscalers | \* | create patch update delete |
| odigos.io | collectorsgroups | \* | get list create patch update watch delete |
| odigos.io | collectorsgroups/status | \* | get list watch patch update |
| odigos.io | destinations | \* | get list watch |
| odigos.io | destinations/status | \* | get patch update |
| odigos.io | instrumentationrules processors | \* | get list watch patch update delete create |
| odigos.io | processors/status | \* | get patch update |
| odigos.io | instrumentationrules/status | \* | get patch update |
| odigos.io | actions | \* | get list watch patch update delete create |
| odigos.io | samplings | \* | get list watch |
| odigos.io | actions/status | \* | get patch update |
| odigos.io | recommendations | \* | get list watch create patch update delete |
| odigos.io | recommendations/status | \* | get patch update |
| actions.odigos.io | \* | \* | get list watch update |
| actions.odigos.io | \*/status | \* | get patch update |
| odigos.io | samplings/status | \* | get patch update |
| \* | secrets | \* | get list watch |
| \* | secrets | instrumentor-webhooks-cert | update |
| \* | secrets | autoscaler-webhook-cert autoscaler-webhooks-cert | delete |
| apps | daemonsets | odiglet | get list watch |
| apps | deployments/finalizers | \* | update |
| odigos.io | collectorsgroups/finalizers | \* | get patch update |
| batch | cronjobs | odigos-go-offsets-updater | get list watch create update patch delete |
| batch | jobs | \* | create |

### odigos-leader-election-role

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | events | \* | create patch |
| coordination.k8s.io | leases | \* | get list watch create update patch delete |

### odiglet

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | list watch |
| \* | configmaps | odigos-data-collection effective-config | get list watch |

### odigos-ui

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | get list watch create update patch |
| \* | secrets | \* | get list watch create patch update delete |
| apps | replicasets | \* | get list |
| autoscaling | horizontalpodautoscalers | \* | get |
| odigos.io | \* | \* | get list watch |
| odigos.io | instrumentationrules destinations actions samplings | \* | create patch update delete |
| odigos.io | recommendations | \* | update patch |
| odigos.io | collectorsgroups | \* | get list watch |
| actions.odigos.io | \* | \* | get list watch create patch update delete |
| operator.odigos.io | odigos | \* | get list watch |
| apps | deployments/scale | odigos-instrumentor | get update patch |
| apps | daemonsets | odiglet | get update patch |
| \* | pods | \* | get list delete |
| \* | pods/log | \* | get |
| \* | pods/proxy | \* | get |

# Operator

This section lists the RBAC policies used by the Odigos Operator. Many of these permissions are necessary in order to create the RBAC policies for the components listed above.

## ClusterRoles

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps endpoints secrets services | \* | create delete deletecollection get list patch update watch |
| \* | configmaps/finalizers pods/finalizers | \* | update |
| \* | events | \* | create get list patch watch |
| \* | namespaces | \* | get list patch watch |
| \* | namespaces/status nodes/spec nodes/stats replicationcontrollers replicationcontrollers/status resourcequotas | \* | get list watch |
| \* | nodes | \* | get list patch update watch |
| \* | pods | \* | delete get list watch |
| \* | pods/log pods/proxy pods/status | \* | get |
| \* | serviceaccounts | \* | create delete get list patch watch |
| actions.odigos.io | \* | \* | create delete deletecollection get list patch update watch |
| actions.odigos.io | \*/status | \* | get patch update |
| admissionregistration.k8s.io | mutatingwebhookconfigurations validatingwebhookconfigurations | \* | create delete get list patch update watch |
| apiextensions.k8s.io | customresourcedefinitions | \* | create delete deletecollection get list patch update watch |
| apiregistration.k8s.io | apiservices | \* | create get list patch update watch |
| apps | daemonsets deployments replicasets statefulsets | \* | create delete deletecollection get list patch update watch |
| apps | daemonsets/finalizers deployments/finalizers replicasets/finalizers statefulsets/finalizers | \* | update |
| apps | daemonsets/status deployments/status statefulsets/status | \* | get |
| apps | deployments/scale | \* | get patch update |
| apps.openshift.io | deploymentconfigs deploymentconfigs/finalizers | \* | get list patch update watch |
| argoproj.io | rollouts | \* | get list patch update watch |
| argoproj.io | rollouts/finalizers | \* | update |
| autoscaling | horizontalpodautoscalers | \* | create delete get list patch update watch |
| batch | cronjobs | \* | create delete get list patch update watch |
| batch | cronjobs/finalizers jobs/finalizers | \* | update |
| batch | jobs | \* | create delete get list watch |
| coordination.k8s.io | leases | \* | create delete get list patch update watch |
| discovery.k8s.io | endpointslices | \* | get list watch |
| extensions | daemonsets deployments replicasets | \* | get list watch |
| odigos.io | \* | \* | \* |
| odigos.io | collectorsgroups/finalizers sources/finalizers | \* | update |
| odigos.io | collectorsgroups/status destinations/status instrumentationconfigs/status instrumentationinstances/status processors/status | \* | get list patch update watch |
| odigos.io | instrumentationrules/status recommendations/status | \* | get patch update |
| odigos.io | recommendations sampling | \* | create delete get list patch update watch |
| operator.odigos.io | odigos | \* | create delete get list patch update watch |
| operator.odigos.io | odigos/finalizers | \* | update |
| operator.odigos.io | odigos/status | \* | get patch update |
| policy | podsecuritypolicies | privileged | use |
| rbac.authorization.k8s.io | clusterrolebindings clusterroles rolebindings roles | \* | create delete deletecollection get list patch update watch |
| security.openshift.io | securitycontextconstraints | \* | use |
| authentication.k8s.io | tokenreviews | \* | create |
| authorization.k8s.io | subjectaccessreviews | \* | create |


# Add Cluster Info
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/addclusterinfo

This action adds static resource attributes to spans, metrics data points and log records. It is useful to add cluster-wide attributes to all telemetry signals.

## Considerations

> [!WARNING]
>
> Before enabling **add cluster info**, please note the following:
>
> * This action is meant to be used for **static** attributes that are known in advanced. It is not suitable for dynamic attributes that need to be extracted from an API or calculated at runtime.
> * Some destinations may require specific configuration to index resource attributes and make them available for querying. Please refer to the documentation of the destination you are using for more information.
> * If the attribute already exists in the telemetry signal, the value will be overwritten by the value provided in the action.
> * An empty string is a valid value for the `attributeStringValue` field

## Configuration Options

The AddClusterInfo action is configured using the `odigos.io/v1alpha1.Action` CRD with the `addClusterInfo` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*addClusterInfo \*\*\*
>
> **addClusterInfo** `object` : Configuration for the AddClusterInfo action.
>
> * This field is *required* for this action type
>
> > \*\*clusterAttributes \*\*\*
> >
> > **clusterAttributes** `object[]` : An array of attributes to be added to the telemetry signals.
> >
> > * This field is *required*
> >
> > > \*\*attributeName \*\*\*
> > >
> > > **attributeName** `string` : The name of the attribute to be added to the telemetry signals.
> > >
> > > * This field is *required*
> >
> > > \*\*attributeStringValue \*\*\*
> > >
> > > **attributeStringValue** `string` : The value of the attribute to be added to the telemetry signals.
> > >
> > > * This field is *required*
>
> > **overwriteExistingValues**
> >
> > **overwriteExistingValues** `boolean` : Whether to overwrite existing attribute values.
> >
> > * This field is *optional*, and defaults to `false`

You can include any static value attribute that is meaningful to you, like `team.name` etc.

> [!TIP]
>
> We recommend:
>
> * `k8s.cluster.name`: The name of the k8s cluster which can be useful to distinguish between different clusters.
> * `deployment.environment`: The environment of the cluster (e.g. `production`, `staging`, `development`, etc).

## Basic Example

The following example demonstrates how to insert the `k8s.cluster.name` attribute to all telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml insert-cluster-name.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: insert-cluster-name
  namespace: odigos-system
spec:
  actionName: insert-cluster-name
  signals:
    - TRACES
    - METRICS
    - LOGS
  addClusterInfo:
    clusterAttributes:
      - attributeName: k8s.cluster.name
        attributeStringValue: my-cluster
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f insert-cluster-name.yaml
```


# DB Query Templatization
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/dbquerytemplatization

This action templatizes database query text by replacing number and string literals with placeholders to reduce cardinality.

## Considerations

> [!WARNING]
>
> Before enabling **DB query templatization**, please note the following:
>
> * The action rewrites `db.query.text` (or the deprecated `db.statement`) in place when present.
> * Number and string literals are replaced with `?`. Identifiers, bind/positional parameters, booleans, and `NULL` are left unchanged.
> * Spans without a `db.system.name` (or the deprecated `db.system`) attribute, or with a known non-SQL system (e.g. MongoDB, Redis), are skipped.

## Use Cases

**Cardinality control**

* Raw SQL with embedded literals (e.g. `WHERE id = 1` vs `WHERE id = 2`) creates high-cardinality query attributes and span names. Templatizing literals collapses those into a shared template such as `WHERE id = ?`.

**Cost and backend indexing**

* Lower cardinality on database spans improves aggregations, span metrics, and indexing in your observability backend, and can reduce storage volume.

**Sensitive values in queries**

* Replacing string and number literals removes many accidental secrets or PII that were inlined into SQL (passwords, emails, IDs) while keeping the query structure intact.

## Configuration Options

The DbQueryTemplatization action is configured using the `odigos.io/v1alpha1.Action` CRD with the `dbQueryTemplatization` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*dbQueryTemplatization \*\*\*
>
> **dbQueryTemplatization** `object` : Configuration for the DbQueryTemplatization action.
>
> * This field is *required* for this action type
>
> > **scopes**
> >
> > **scopes** `object` : Limits which sources this templatization config applies to.
> >
> > * This field is *optional*
> >
> > * If unset or empty, the config is applied to all sources.
> >
> > * If multiple source scopes are set, they must all match simultaneously (AND logic).
> >
> > > **sources**
> > >
> > > **sources** `object[]` : A list of workloads to apply this action to.
> > >
> > > * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, etc.).
> >
> > > **namespaces**
> > >
> > > **namespaces** `string[]` : Apply this action to all sources in the listed namespaces.
> >
> > > **languages**
> > >
> > > **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages.
>
> > **templatizeLiterals**
> >
> > **templatizeLiterals** `boolean` : When enabled, number and string literals in SQL queries are replaced with placeholders.
> >
> > * This field is *optional*, and defaults to `false`
> > * What is templatized:
> >   * number literals (e.g. `1`, `3.14`)
> >   * string literals (e.g. `'alice'`)
> > * What is **not** templatized:
> >   * identifiers (table names, column names, etc.)
> >   * positional parameters (e.g. `$1`)
> >   * bind parameters (e.g. `:name`)
> >   * boolean literals and `NULL`
> >   * keywords, operators, punctuation, whitespace, comments, etc.

## Basic Example

Given a span with:

```text theme={null}
db.query.text: SELECT * FROM users WHERE id = 1 AND name = 'alice'
```

the action rewrites it to:

```text theme={null}
db.query.text: SELECT * FROM users WHERE id = ? AND name = ?
```

**Step 1**

Create a YAML file with the following content:

```yaml db-query-templatization.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: db-query-templatization
  namespace: odigos-system
spec:
  actionName: DB Query Templatization
  signals:
    - TRACES
  dbQueryTemplatization:
    templatizeLiterals: true
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f db-query-templatization.yaml
```

## Scoped Example

To limit templatization to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources.

```yaml db-query-templatization-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: db-query-templatization-my-app
  namespace: odigos-system
spec:
  actionName: DB Query Templatization for my-app
  signals:
    - TRACES
  dbQueryTemplatization:
    scopes:
      sources:
        - name: my-app
          namespace: default
          kind: Deployment
    templatizeLiterals: true
```

> [!TIP]
>
> Pair this action with [Infer DB Attributes](./inferdbattributes) so database spans get both templatized query text and inferred `db.operation.name` / `db.collection.name` attributes.


# Delete Attribute
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/deleteattribute

This action will delete the specified attributes from all telemetry signals that are specified in the `signals` field, regardless of the source, or any other condition.

## Considerations

> [!WARNING]
>
> Before enabling **delete attribute**, please note the following:
>
> * This action will delete selected attributes from spans, metric data points and log records, but will not delete the entire span, metric data point or log record.
> * For matching attributes, the attribute value will be deleted entirely. This action is not suitable for partial deletion or masking of the attribute value. For example, if you collect an `http.request.body` attribute, this action can be used to delete the attribute but not a key from the JSON object.
> * While it is common for information to be recorded in attributes, it can also be recorded into names or other fields, for which this action will not be effective.

## Use Cases

**Security**

* By default, OpenTelemetry should not record PII (Personally Identifiable Information) or sensitive data such as passwords, api tokens, creditcard numbers etc. However, if the data is accidentally added to the telemetry signals, or if a manual instrumentation recorded it into an attribute, this action can be used to delete such data.

**Cost Reduction**

* Some vendors charge based on the amount of data ingested. For self hosted destinations, the cost is correlated to the use of cloud resources which grows with the based on the amount of data you process and store. By deleting unnecessary attributes, you can reduce the amount of data ingested and reduce costs.

**Usability**

* Remove duplications - sometimes, the same attribute is added to the telemetry signals by multiple sources. This action can be used to delete the duplicates and keep the data clean. For example, you might find the host name recorded both as `host.name` and `k8s.node.name` which are the same value.
* Verbosity - not all data is useful to all users. You might wish to delete certain attributes when they give no value to your organization or observability needs. Sometimes, recording too much data can obscure the important information and make it harder to find.

## Configuration Options

The DeleteAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `deleteAttribute` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*deleteAttribute \*\*\*
>
> **deleteAttribute** `object` : Configuration for the DeleteAttribute action.
>
> * This field is *required* for this action type
>
> > \*\*attributeNamesToDelete \*\*\*
> >
> > **attributeNamesToDelete** `string[]` : An array of strings representing the names of the attributes to be deleted from the telemetry signals.
> >
> > * This field is *required*

## Basic Example

The following example demonstrates how to delete the `host.name` attribute from all telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml delete-host-name.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: delete-host-name
  namespace: odigos-system
spec:
  actionName: "delete host.name"
  signals:
    - TRACES
    - METRICS
    - LOGS
  deleteAttribute:
    attributeNamesToDelete:
      - host.name
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f delete-host-name.yaml
```


# Extract Attribute
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/extractattribute

This action extracts values from span attributes (payloads, SQL statements, URL paths) into new top-level span attributes.

## Considerations

> [!WARNING]
>
> Before enabling **extract attribute**, please note the following:
>
> * This action only operates on **trace** signals.
> * Extraction reads from a fixed set of high-cardinality string attributes and writes the captured value to a **new** span attribute. The source attributes are scanned in order until a match is found, and only the **first** match is used per extraction rule.
> * The action will scan the following span attributes for matches:
>   * `db.statement`
>   * `db.query.text`
>   * `messaging.message.payload`
>   * `http.request.payload`
>   * `http.response.payload`
> * If the `targetAttributeName` attribute already exists on the span, it will **not** be overridden.
> * Each extraction must specify either `lookupKey` + `dataFormat`, **or** a custom `regex` — not both. When using `regex`, the pattern must contain exactly one capture group; the value of the first capture group is written to the attribute named `targetAttributeName`.
> * Each `targetAttributeName` must be unique within a single action.

## Use Cases

**Accessing deeply nested attributes within strings**

* Easily extract and access values of deeply nested attributes within payloads of JSONs, SQL statements or URLs.

**Indexing high-cardinality identifiers**

* Extract domain-specific identifiers (e.g. `order_id`, `user_id`, `study_id`) embedded inside payloads, SQL statements, or URL paths into top-level span attributes so they can be indexed, filtered, and aggregated by your backend.

**Correlating spans across services**

* Lift correlation IDs that are buried inside structured payloads (JSON envelopes, request/response bodies) onto the span itself so traces can be searched and joined by business identifiers.

**Cost-aware enrichment**

* Avoid sending the full payload to your backend by extracting only the meaningful fields, then dropping or masking the original payload attribute with another action.

## Configuration Options

The ExtractAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `extractAttribute` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*extractAttribute \*\*\*
>
> **extractAttribute** `object` : Configuration for the ExtractAttribute action.
>
> * This field is *required* for this action type
>
> > \*\*extractions \*\*\*
> >
> > **extractions** `object[]` : An ordered array of extraction rules to apply to each span. Must contain at least one entry.
> >
> > * This field is *required*
> >
> > > \*\*targetAttributeName \*\*\*
> > >
> > > **targetAttributeName** `string` : The name of the new span attribute the extracted value will be written to.
> > >
> > > * This field is *required*
> > > * Must be unique across all extractions in the same action.
> >
> > > **lookupKey**
> > >
> > > **lookupKey** `string` : The literal key to look up inside the scanned attributes (resolved via `dataFormat` into a regex).
> > >
> > > * This field is *required* when `dataFormat` is set, and must be omitted when `regex` is set.
> >
> > > **dataFormat**
> > >
> > > **dataFormat** `string` : A pre-set format that defines how `lookupKey` is matched inside the scanned attributes.
> > >
> > > * This field is *required* when `lookupKey` is set, and must be omitted when `regex` is set.
> > > * Supported values:
> > >   * `json` — matches `key: value` pairs in JSON envelopes (e.g. `{"order_id": "42"}` or `order_id: 42`).
> > >   * `sql` — matches `key = value` pairs in SQL statements (e.g. `WHERE order_id = '42'` or `order_id=42`).
> > >   * `resource_path` — matches `key/value` segments inside URL paths (e.g. `/projects/test-project`), stopping at the next `/`, whitespace, `?`, `&`, `#`, or quote.
> >
> > > **regex**
> > >
> > > **regex** `string` : A custom regular expression with a single capture group whose value is written to the attribute named `targetAttributeName`.
> > >
> > > * This field is *required* when `lookupKey` and `dataFormat` are not set.
> > > * The first capture group of the first match is used.

> [!TIP]
>
> Each extraction rule must use **exactly one** of:
>
> * `lookupKey` + `dataFormat` — for the built-in JSON/SQL/URL matchers, or
> * `regex` — for a custom pattern with a single capture group.

## Basic Example

The following example demonstrates the four extraction variations against a single span that has the following attributes:

* `messaging.message.payload`: `{"task_id": "task-7f3a9b2c", "status": "Completed"}`
* `http.request.payload`: `tenants/acme-corp/regions/us-east/workspaces/main/services/analytics/jobs/task-7f3a9b2c/runs/run-001`
* `db.statement`: `SELECT * FROM invoices WHERE invoice_id = '42' -- request_id req-xyz9`

After the action runs, the span will be enriched with the following extracted attributes:

| New attribute name | Extracted value |
| - | - |
| `extracted.json.task_id` | `task-7f3a9b2c` |
| `extracted.sql.invoice_id` | `42` |
| `extracted.url.tenant` | `acme-corp` |
| `extracted.url.region` | `us-east` |
| `extracted.url.job` | `task-7f3a9b2c` |
| `extracted.regex.request_id` | `xyz9` |

**Step 1**

Create a YAML file with the following content:

```yaml extract-attribute.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: extract-attribute
  namespace: odigos-system
spec:
  actionName: extract attribute
  signals:
    - TRACES
  extractAttribute:
    extractions:
      # JSON variation - matches a JSON envelope inside messaging.message.payload
      - lookupKey: task_id
        dataFormat: json
        targetAttributeName: extracted.json.task_id

      # SQL variation - matches `key = 'value'` syntax in SQL statements
      - lookupKey: invoice_id
        dataFormat: sql
        targetAttributeName: extracted.sql.invoice_id

      # URL variation - matches `key/value` segments inside http.request.payload
      - lookupKey: tenants
        dataFormat: resource_path
        targetAttributeName: extracted.url.tenant
      - lookupKey: regions
        dataFormat: resource_path
        targetAttributeName: extracted.url.region
      - lookupKey: jobs
        dataFormat: resource_path
        targetAttributeName: extracted.url.job

      # Custom regex variation - one capture group, value goes to the new attribute
      - regex: "req-([a-z0-9]+)"
        targetAttributeName: extracted.regex.request_id
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f extract-attribute.yaml
```


# Infer DB Attributes
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/inferdbattributes

This action parses database query text and adds attributes such as db.operation.name and db.collection.name.

## Considerations

> [!WARNING]
>
> Before enabling **infer DB attributes**, please note the following:
>
> * Currently, only trace signals are supported.
> * The action parses `db.query.text` when present, otherwise `db.statement`.
> * Existing `db.operation.name` / `db.collection.name` (or legacy `db.operation` / `db.sql.table`) values are **not** overwritten.
> * Attributes are added only when inference is unambiguous: a single SQL operation and a single table/collection.
> * When new operation (and optionally collection) attributes are added, the span name may be updated to `OPERATION collection` (e.g. `SELECT users`) if it does not already contain those values.
> * Spans without a `db.system` / `db.system.name` attribute, or with a known non-SQL system (e.g. MongoDB, Redis), are skipped.

## Use Cases

**Enrich incomplete database spans**

* Some instrumentations record only the raw query text. This action derives OpenTelemetry semantic attributes (`db.operation.name`, `db.collection.name`) so backends can filter and group by operation and table.

**Better span names and metrics**

* Normalized span names such as `SELECT users` make traces easier to read and improve aggregations when span metrics are enabled.

**Consistent attributes across languages**

* Apply the same inference rules cluster-wide, regardless of which language or library produced the span.

## Configuration Options

The InferDbAttributes action is configured using the `odigos.io/v1alpha1.Action` CRD with the `inferDbAttributes` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*inferDbAttributes \*\*\*
>
> **inferDbAttributes** `object` : Configuration for the InferDbAttributes action.
>
> * This field is *required* for this action type
>
> * No additional options are required; an empty object enables inference for matching sources.
>
> > **scopes**
> >
> > **scopes** `object` : Limits which sources this config applies to.
> >
> > * This field is *optional*
> >
> > * If unset or empty, the config is applied to all sources.
> >
> > * If multiple source scopes are set, they must all match simultaneously (AND logic).
> >
> > > **sources**
> > >
> > > **sources** `object[]` : A list of workloads to apply this action to.
> > >
> > > * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, or `DaemonSet`).
> >
> > > **namespaces**
> > >
> > > **namespaces** `string[]` : Apply this action to all sources in the listed namespaces.
> >
> > > **languages**
> > >
> > > **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages.

## Basic Example

Given a span with:

```text theme={null}
db.query.text: SELECT * FROM users WHERE id = 1
```

and without `db.operation.name` / `db.collection.name`, the action adds:

| Attribute | Value |
| - | - |
| `db.operation.name` | `SELECT` |
| `db.collection.name` | `users` |

and updates the span name to `SELECT users` if it hasn't been set to that already.

**Step 1**

Create a YAML file with the following content:

```yaml infer-db-attributes.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: infer-db-attributes
  namespace: odigos-system
spec:
  actionName: Infer DB Attributes
  signals:
    - TRACES
  inferDbAttributes: {}
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f infer-db-attributes.yaml
```

## Scoped Example

To limit inference to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources.

```yaml infer-db-attributes-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: infer-db-attributes-my-app
  namespace: odigos-system
spec:
  actionName: Infer DB Attributes for my-app
  signals:
    - TRACES
  inferDbAttributes:
    scopes:
      sources:
        - name: my-app
          namespace: default
          kind: Deployment
```

> [!TIP]
>
> Pair this action with [DB Query Templatization](./dbquerytemplatization) to both infer semantic attributes and replace literals in the query text.


# Kubernetes Attributes
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/k8sattributes

This action adds kubernetes related resource attributes to spans, metrics data points and log records.

## Overview

By adding Kubernetes-specific attributes to telemetry signals, you can gain better visibility into workloads running in the cluster.

Helps in correlating traces, logs and metrics with specific Kubernetes resources, such as pods, deployments, nodes and clusters.

Adding resource attributes based on custom kubernetes annotations/labels can enrich the data with user defined aggregations and metadata corresponding to the originating pod that produced the telemetry.

## Configuration Options

The K8sAttributes action is configured using the `odigos.io/v1alpha1.Action` CRD with the `k8sAttributes` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES, LOGS, METRICS`

> \*\*k8sAttributes \*\*\*
>
> **k8sAttributes** `object` : Configuration for the K8sAttributes action.
>
> * This field is *required* for this action type
>
> > **collectContainerAttributes**
> >
> > **collectContainerAttributes** `boolean` : Collect the following container related attributes:
> >
> > * `k8s.container.name`
> > * `container.id`
> > * `container.image.name`
> > * `container.image.tag`
> >
> > This field is *optional* and defaults to `false`
>
> > **collectReplicaSetAttributes**
> >
> > **collectReplicaSetAttributes** `boolean` : Collect the following replica set related attributes:
> >
> > * `k8s.replicaset.name`
> >   if CollectWorkloadUID is set to true, the following attribute will be collected as well:
> > * `k8s.replicaset.uid`
> >
> > This field is *optional* and defaults to `false`
>
> > **CollectWorkloadUID**
> >
> > **CollectWorkloadUID** `boolean` : Collect the workload UID:
> >
> > * `k8s.deployment.uid`
> > * `k8s.daemonset.uid`
> > * `k8s.statefulset.uid`
> >
> > Only one of these attributes can appear for a given span/log/metric - based on the workload it originated from.
> > This field is *optional* and defaults to `false`
>
> > **CollectClusterUID**
> >
> > **CollectClusterUID** `boolean` : Collect the `k8s.cluster.uid` attribute, which is set to the uid of the namespace "kube-system". This allows aggregating observability based on the cluster it originated from.
> >
> > This field is *optional* and defaults to `false`
>
> > **labelsAttributes**
> >
> > **labelsAttributes** `K8sLabelAttribute[]` : Collect resource attributes based on the specified labels.
> >
> > Each label attribute can specify:
> >
> > * `labelKey`: The Kubernetes label key to extract
> > * `attributeKey`: The OpenTelemetry attribute key to create
> > * `from` (deprecated): Single source - "pod" or "namespace"
> > * `fromSources`: Array of sources to extract from, with precedence (pod > namespace > node)
> >
> > When using `fromSources` with multiple sources, if the same label exists in multiple sources, the most specific source (e.g., pod) takes precedence over less specific sources (e.g., namespace).
> >
> > This field is *optional* and can remain empty.
>
> > **annotationsAttributes**
> >
> > **annotationsAttributes** `K8sAnnotationAttribute[]` : Collect resource attributes based on the specified annotations.
> >
> > Each annotation attribute can specify:
> >
> > * `annotationKey`: The Kubernetes annotation key to extract
> > * `attributeKey`: The OpenTelemetry attribute key to create
> > * `from` (deprecated): Single source - "pod" or "namespace"
> > * `fromSources`: Array of sources to extract from, with precedence (pod > namespace > node)
> >
> > When using `fromSources` with multiple sources, if the same annotation exists in multiple sources, the most specific source (e.g., pod) takes precedence.
> >
> > This field is *optional* and can remain empty.

## Basic Example

The following example demonstrates how to collect Kubernetes attributes, including labels and annotations, and attach them to telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml k8s-attributes.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: collect-k8s-attributes
  namespace: odigos-system
spec:
  actionName: "Collect Kubernetes Attributes"
  signals:
    - TRACES
    - METRICS
    - LOGS
  k8sAttributes:
    collectContainerAttributes: true
    collectWorkloadUID: true
    collectClusterUID: true
    labelsAttributes:
      - labelKey: "app.kubernetes.io/name"
        attributeKey: "app.kubernetes.name"
        from: "pod"
    annotationsAttributes:
      - annotationKey: "kubectl.kubernetes.io/restartedAt"
        attributeKey: "kubectl.kubernetes.restartedAt"
        from: "pod"
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f k8s-attributes.yaml
```

## Multi-Source Label Extraction Example

The following example demonstrates how to extract the same label from multiple sources with proper precedence. This is useful when you want pod-level labels to override namespace-level labels when both exist.

**Step 1**

Create a YAML file with the following content:

```yaml k8s-attributes-multi-source.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: collect-environment-label
  namespace: odigos-system
spec:
  actionName: "Collect Environment Label with Precedence"
  signals:
    - TRACES
    - METRICS
    - LOGS
  k8sAttributes:
    labelsAttributes:
      - labelKey: "environment"
        attributeKey: "k8s.environment"
        fromSources:
          - pod        # Highest precedence - pod labels override namespace labels
          - namespace  # Lower precedence - used as fallback
```

In this configuration:

* If a pod has an `environment` label, that value will be used
* If the pod doesn't have the label, but the namespace does, the namespace value will be used
* This allows namespace-wide defaults while permitting pod-level overrides

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f k8s-attributes-multi-source.yaml
```


# PII Masking
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/piimasking

This action masks PII from traces using predefined categories and optional custom format or regex rules.

## Considerations

> [!WARNING]
>
> Before enabling **pii masking**, please note the following:
>
> * Predefined PII categories replace matched values with a category-specific token (e.g. `***EMAIL***`, `***CREDIT_CARD***`).
> * Custom format and regex maskings replace only the matched capture group with `****`, leaving the rest of the value intact.
> * Currently, only trace signals are supported.
> * All span attribute values in every matching span will be examined and masked accordingly.

## Use Cases

**Security**

* By default, OpenTelemetry should not record PII (Personally Identifiable Information) or sensitive data such as passwords, api tokens, etc. However, if the data is accidentally added to the telemetry signals, or if a manual instrumentation recorded it into an attribute, this action can be used to mask such data.

**Legal and Compliance**

* Ensure compliance with legal and privacy.
  * Payment Card Industry (PCI) Data Security Standards prohibit logging certain things or storing them unencrypted.

**Domain-specific secrets**

* Mask application-specific fields (e.g. SSN in JSON payloads, passwords in SQL statements, or identifiers in URL paths) using `customFormatMaskings` or `customRegexMaskings` when they are not covered by a predefined PII category.

## Configuration Options

The PiiMasking action is configured using the `odigos.io/v1alpha1.Action` CRD with the `piiMasking` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*piiMasking \*\*\*
>
> **piiMasking** `object` : Configuration for the PiiMasking action.
>
> * This field is *required* for this action type
>
> > **scopes**
> >
> > **scopes** `object` : Limits which sources this masking config applies to.
> >
> > * This field is *optional*
> >
> > * If unset or empty, the config is applied to all sources.
> >
> > > **sources**
> > >
> > > **sources** `object[]` : A list of workloads to apply this action to.
> > >
> > > * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, or `DaemonSet`).
> >
> > > **namespaces**
> > >
> > > **namespaces** `string[]` : Apply this action to all sources in the listed namespaces.
> >
> > > **languages**
> > >
> > > **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages.
>
> > **piiCategories**
> >
> > **piiCategories** `string[]` : An array of strings representing the PII categories you want to mask.
> >
> > * This field is *optional*
> > * The available PII categories are:
> >   * `CREDIT_CARD` - mask Visa and MasterCard credit card numbers
> >
> >   * `EMAIL` - mask email addresses
> >
> >   * `JWT` - mask JSON Web Tokens
> >
> >   * `UUID` - mask UUIDs
> >
> > > [!NOTE]
> > >
> > > Can't find the PII category you need? Use `customFormatMaskings` or `customRegexMaskings`, or contact us and we will add it for you.
>
> > **customFormatMaskings**
> >
> > **customFormatMaskings** `object[]` : Format-based masking rules applied in order. Each rule looks up a key inside structured attribute values and masks the matched value.
> >
> > * This field is *optional*
> >
> > > \*\*lookupKey \*\*\*
> > >
> > > **lookupKey** `string` : The field or path segment whose value should be masked (e.g. a JSON key, SQL column name, or URL path segment).
> > >
> > > * This field is *required*
> >
> > > \*\*dataFormat \*\*\*
> > >
> > > **dataFormat** `string` : The format of the data to search in.
> > >
> > > * This field is *required*
> > > * Supported values:
> > >   * `json` — matches `key: value` pairs in JSON envelopes (e.g. `{"ssn": "123-45-6789"}` or `ssn: 123-45-6789`).
> > >   * `sql` — matches `key = value` pairs in SQL statements (e.g. `WHERE password = 'hunter2'`).
> > >   * `resource_path` — matches `key/value` segments inside URL paths (e.g. `/orders/abc-123`), stopping at the next `/`, whitespace, `?`, `&`, `#`, or quote.
>
> > **customRegexMaskings**
> >
> > **customRegexMaskings** `object[]` : Regex-based masking rules applied in order.
> >
> > * This field is *optional*
> >
> > > \*\*regex \*\*\*
> > >
> > > **regex** `string` : A custom regular expression with a single capture group whose value is masked with `****`.
> > >
> > > * This field is *required*
> > > * The pattern must contain at least one capture group; only the first capture group is replaced.

## Basic Example

The following example demonstrates how to configure the Pii Masking action to mask Visa and MasterCard credit card numbers from span attribute values using the Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml pii-masking.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: pii-masking
  namespace: odigos-system
spec:
  actionName: PII Masking
  signals:
    - TRACES
  piiMasking:
    piiCategories:
      - CREDIT_CARD
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f pii-masking.yaml
```

## Custom Masking Example

The following example combines a predefined category with custom format and regex rules. Given span attributes such as:

* `{"ssn": "123-45-6789", "name": "alice"}`
* `WHERE password = 'hunter2' AND status = 'ok'`
* `/api/v1/orders/abc-123/items`
* `auth api_key=super-secret-value next`

the action produces:

| Input fragment | Masked result |
| - | - |
| `user@example.com` | `***EMAIL***` |
| `"ssn": "123-45-6789"` | `"ssn": "****"` |
| `password = 'hunter2'` | `password = '****'` |
| `/orders/abc-123/` | `/orders/****/` |
| `api_key=super-secret-value` | `api_key=****` |

**Step 1**

Create a YAML file with the following content:

```yaml pii-masking-custom.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: pii-masking-custom
  namespace: odigos-system
spec:
  actionName: Custom PII Masking
  signals:
    - TRACES
  piiMasking:
    piiCategories:
      - EMAIL
    customFormatMaskings:
      - lookupKey: ssn
        dataFormat: json
      - lookupKey: password
        dataFormat: sql
      - lookupKey: orders
        dataFormat: resource_path
    customRegexMaskings:
      - regex: 'api[_-]?key=([^\s&]+)'
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f pii-masking-custom.yaml
```

## Scoped Example

To limit masking to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources.

```yaml pii-masking-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: pii-masking-my-app
  namespace: odigos-system
spec:
  actionName: PII Masking for my-app
  signals:
    - TRACES
  piiMasking:
    scopes:
      sources:
        - name: my-app
          namespace: default
          kind: Deployment
    piiCategories:
      - CREDIT_CARD
      - EMAIL
```


# Rename Attribute
Source: https://docs.odigos.io/enterprise/pipeline/actions/attributes/renameattribute

This action will rename the specified attributes from all telemetry signals that are specified in the `signals` field, regardless of the source, or any other condition.

## Configuration Options

The RenameAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `renameAttribute` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*renameAttribute \*\*\*
>
> **renameAttribute** `object` : Configuration for the RenameAttribute action.
>
> * This field is *required* for this action type
>
> > \*\*renames \*\*\*
> >
> > **renames** `object` : A map of strings, key representing the name of the attribute to rename and values the desired name.
> >
> > * This field is *required*

## Basic Example

The following example demonstrates how to rename the deprecated `net.peer.address` attribute to stable `network.peer.address` in all telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml rename-net-to-network.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: rename-net-to-network
  namespace: odigos-system
spec:
  actionName: Rename network attributes
  signals:
    - LOGS
    - METRICS
    - TRACES
  renameAttribute:
    renames:
      net.peer.address: network.peer.address
      net.local.address: network.local.address
      net.peer.ip: network.peer.address
      net.host.ip: network.local.address
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f rename-net-to-network.yaml
```


# Customized
Source: https://docs.odigos.io/enterprise/pipeline/actions/crd

You can define custom actions using OpenTelemetry Collector Processors in Odigos.

> [!NOTE]
>
> This option is for advanced users who are familiar with the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), its [configuration](https://opentelemetry.io/docs/collector/configuration/), and with [Kubernetes CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/).

You can browse the full list of processors that Odigos includes [here](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61).

To view the configuration for each processor, it is recommended to visit the README of this component in [OpenTelemetry Collector Contrib codebase](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor).

Adding a processor to your Odigos pipeline can be done by creating a `Processor` custom resource in your Kubernetes cluster under the `odigos-system` namespace:

## Configuration Options

> **processorName**
>
> **processorName** `string` : Allows you to attach a meaningful name to the processor for convenience.
>
> * This field is *optional*
> *

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the processor for convenience.
>
> * This field is *optional*
> *

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the processor, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the processor will act on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*collectorRoles \*\*\*
>
> **collectorRoles** `string[]` : An array with the collector roles that the processor will act on.
>
> * This field is *required*
> * Supported values: `CLUSTER_GATEWAY`, `NODE_COLLECTOR`

> \*\*type \*\*\*
>
> **type** `string` : The type of the processor.
>
> * This field is *required*
> * The name of the processor you want to use, as defined in the OpenTelemetry Collector (batch, attributes, etc).

> **orderHint**
>
> **orderHint** `number` : If your processors need to run in a specific order relatively to other processors, you can hint the order by setting an integer value here.
>
> * This field is *optional*
> * The lower the value, the earlier the processor will run in the collector pipeline. If the value is missing or 0, the processor will run in an arbitrary order in the collector pipeline.

> **processorConfig**
>
> **processorConfig** `object` : A field to pass configuration to the processor.
>
> * This field is *optional*
> * The structure of this field is specific to each processor, and you can find the configuration options for each processor in the [OpenTelemetry Collector Contrib codebase](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor).

## Basic Example

The following example demonstrates how to add a resource attribute `deployment.environment` with the value `production` to all spans in the cluster.

**Step 1**

Create a YAML file with the following content:

```yaml example-processor.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Processor
metadata:
  name: example-processor
  namespace: odigos-system
spec:
  type: resource
  processorConfig:
    attributes:
      - key: deployment.environment
        value: production
        action: insert
  signals:
    - TRACES
  collectorRoles:
    - CLUSTER_GATEWAY
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f example-processor.yaml
```


# Introduction
Source: https://docs.odigos.io/enterprise/pipeline/actions/introduction

Actions are a way to modify the OpenTelemetry data recorded by Odigos Sources, before it is exported to your destinations.

## What are Processors?

Under the hood, Odigos uses the [OpenTelemetry Collector Processors](https://opentelemetry.io/docs/collector/configuration/#processors) to implement actions.

Processors are a class of OpenTelemetry Collector component. They are used to transform, filter, or enrich your OpenTelemetry data before it is sent to your [destinations](../../backends-overview).

## Benefits

* Odigos Actions are language agnostic. They run in the OpenTelemetry collectors and can act on any data type (traces, metrics, logs) no matter the source or the programming language of your service.
* Processing the data on the k8s node or the cluster level reduces the load on your services and allows you to apply the same processing rules to all your services and signals.
* Applying actions in your cluster can reduce the volume of data sent to your backend, which can reduce your costs.
* OpenTelemetry maintains a large collection of processors, which are ready to use and can be configured to your needs.

Actions are a powerful tool to help you manage your observability data and ensure that you are sending the right data to your backend.

## Action Types

### Attribute Actions

Attribute Actions are a way to modify the OpenTelemetry data recorded by Odigos Sources, before it is exported to your destinations.

* [Add Cluster Info](../../pipeline/actions/attributes/addclusterinfo)
* [Delete Attribute](../../pipeline/actions/attributes/deleteattribute)
* [Rename Attribute](../../pipeline/actions/attributes/renameattribute)
* [Pii Masking](../../pipeline/actions/attributes/piimasking)
* [Kubernetes Attributes](../../pipeline/actions/attributes/k8sattributes)
* [Extract Attribute](../../pipeline/actions/attributes/extractattribute)
* [DB Query Templatization](../../pipeline/actions/attributes/dbquerytemplatization)
* [Infer DB Attributes](../../pipeline/actions/attributes/inferdbattributes)

### Sampling Actions

Sampling Actions decide which traces are kept and which are dropped before they reach your destinations. Rules are organized into three categories, evaluated in order—**Noisy Operation → Highly Relevant → Cost Reduction**—so you can drop noise early, guarantee retention for important traffic, and cap everything else for cost.

* [Sampling overview](../../pipeline/sampling/introduction)
* [Head and tail sampling](../../pipeline/sampling/head-and-tail-sampling)
* [Rules overview](../../pipeline/sampling/rules/overview)
* [Configuration reference](../../pipeline/sampling/configuration)
* [Examples](../../pipeline/sampling/examples)

### Custom Actions

Odigos also supports adding OpenTelemetry processors with [Kubernetes CRDs](../../pipeline/actions/crd) which you can apply manually, or through a GitOps workflow.


# Troubleshooting
Source: https://docs.odigos.io/enterprise/pipeline/actions/troubleshooting

If you are experiencing issues with your actions and processors, use the checklist below to troubleshoot the issue.

## View Collector Configurations

To view the current configurations of the collectors, you can check the ConfigMap objects in the `odigos-system` namespace.

You should see the processors configured under `data/conf/processors`, and actions configured under `data/conf/pipelines`.

**Node Collector:**

```bash theme={null}
kubectl get cm -n odigos-system -o yaml odigos-data-collection
```

**Gateway Collector:**

```bash theme={null}
kubectl get cm -n odigos-system -o yaml odigos-gateway
```

## View Instrumentor Logs

The `odigos-instrumentor` controller is the component that watches for changes in the action or processor CRD and generates the collector configuration.

If you don't see your processor in the collector configuration, you can check the logs of the instrumentor for errors or clues:

```sh theme={null}
kubectl logs -n odigos-system deployments.apps/odigos-instrumentor
```

## View Collector Logs

If you do see your processor in the collector configuration, and there's an issue, the collector should log an error and will not start. To check the status of the Odigos pods, run:

```sh theme={null}
kubectl get pods -n odigos-system
```

Look for pods with STATUS `CrashLoopBackOff` or `Error`.\

To see the error, you can examine the logs of that pod:

```sh theme={null}
kubectl logs -n odigos-system <pod-name>
```

You might find an error message like this:

```
* error decoding 'processors': unknown type: "invalid" for id: "invalid/example-processor" (valid values: [datadog experimental_metricsgeneration metricstransform resource servicegraph spanmetrics sumologic batch tail_sampling cumulativetodelta resourcedetection remotetap memory_limiter filter redaction routing span attributes deltatorate groupbyattrs groupbytrace k8sattributes probabilistic_sampler])
```

If this is the case, then please follow the instructions in the error message to correct the processor configuration.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Pipeline Benchmarks
Source: https://docs.odigos.io/enterprise/pipeline/benchmarks

Choose the right ResourceSizePreset for your cluster based on telemetry volume and scale.

The [ResourceSizePreset](./configuration#1-using-sizing-configuration) profiles (`size_s`, `size_m`, `size_l`, and `size_xl`) map collector resources and scaling headroom to telemetry volume. Use this page to assist you in picking a preset.

## Total span throughput

Span throughput is the metric we use for benchmarking and estimating the resources needed to operate Odigos.

To calculate the total span throughput (bytes/second), you can either:

* Estimate your monthly span volume in bytes, which can often be inferred from your existing observability tools. If you only know the number of spans, multiply it by the average span size in bytes to estimate the total volume. Then convert the monthly volume to bytes per second.
* View the span throughput in the Odigos UI under "collectors pipeline".

> [!NOTE]
>
> Collector pipeline UI requires enabling the [Odigos Own Metrics](./odigos-own-metrics) feature.

Note that while **total span throughput** is the primary metric for determining the appropriate Odigos preset size, additional factors can affect the maximum span throughput your Odigos deployment can handle:

* **Number of instrumented pods** - each instrumented pod consumes a small amount of additional static resources in the node collector.
* **Pipeline features** - actions and destinations consume additional collector CPU and memory resources.

## Cluster preset vs. total span throughput

### Maximal single collector capacity

#### Cluster gateway (1 pod) ([see maximum capacity here](#maximum-scaled-cluster))

The cluster gateway is a Deployment that receives telemetry from node collectors, processes it, and exports it to your destinations.

![Cluster gateway benchmark throughput by ResourceSizePreset](/images/shared/pipeline/cluster-light.png)

| Size | spans/s | span throughput (MB/second) |
| - | -: | -: |
| `size_s` | **3,000** | **5.8** |
| `size_m` | **18,000** | **34.5** |
| `size_l` | **23,000** | **44.1** |
| `size_xl` | **31,500** | **60.4** |

#### Node collector

The node collector is a DaemonSet (one pod per node) that receives telemetry from apps on that node and forwards it to the cluster gateway.

![Node collector benchmark throughput by ResourceSizePreset](/images/shared/pipeline/nc-enterprise-light.png)

| Size | spans/s | span throughput (MB/second) |
| - | -: | -: |
| `size_s` | **18,000** | **34.5** |
| `size_m` | **37,000** | **71.0** |
| `size_l` | **51,500** | **98.8** |
| `size_xl` | **60,000** | **115.1** |

### Maximum scaled cluster gateway capacity

| Size | HPA Maximum Replicas | spans/s | span throughput (MB/second) |
| - | -: | -: | -: |
| `size_s` | **5** | **15,000** | **28.8** |
| `size_m` | **8** | **144,000** | **276.2** |
| `size_l` | **12** | **276,000** | **529.4** |
| `size_xl` | **15** | **472,500** | **906.3** |

\* Calculated with a span size of 1.9KB

## Troubleshooting

If you experience any of the following symptoms consistently:

* Memory limit hits or OOMKills
* CPU throttling under steady load
* Gateway HPA scaling to max replicas frequently
* Export backlogs or dropped spans

Consider selecting a larger preset or tune individual `collectorGateway` / `collectorNode` values on the [Configuration](./configuration) page.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Configuring the Pipeline
Source: https://docs.odigos.io/enterprise/pipeline/configuration

Configure the Odigos pipeline to optimize the performance and resource usage of the Odigos components.

## Configurable Components

* **Cluster Gateway Collector** - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, processes it, and exports it to the configured destinations.
* **Node Data Collection Collector** - a collector that runs as a k8s DaemonSet. It collects / scrape the OpenTelemetry data from the applications running on the nodes and forwards it to the Cluster Gateway Collector.

## Configuring the Pipeline

Odigos offers two main options for configuring the pipeline:

> [!WARNING]
>
> When configuring the pipeline, you must use **only one of the two methods** — either `profiles`, or directly editing the `odigos-configuration` ConfigMap.
>
> Using both methods will result in configurations overwriting each other, potentially leading to unexpected behavior.

#### 1. Using Sizing Configuration

To configure resource sizing at install time, set the `ResourceSizePreset` value in your Helm chart when installing Odigos—either in your `values.yaml` file or by using `--set ResourceSizePreset=<preset>` in the `helm install` command.

This approach streamlines the setup process and applies optimized defaults for common use cases.

The available sizing options—`size_s`, `size_m`, `size_l`, and `size_xl`—are predefined profiles designed to simplify pipeline configuration. Each profile specifies a defined set of parameters, as detailed below:

Don't know which size is the right one for your cluster? Read more in [Benchmarks](./benchmarks).

**Cluster Gateway Collector**:

| Size | Minimum Replicas | HPA Maximum Replicas | Request CPU (m) | Limit CPU (m) | Request Memory (Mi) | Limit Memory (Mi) |
| - | - | - | - | - | - | - |
| `size_s` | **1** | **5** | **300m** | **300m** | **300Mi** | **300Mi** |
| `size_m` | **2** | **8** | **1000m** | **1000m** | **600Mi** | **600Mi** |
| `size_l` | **3** | **12** | **1250m** | **1250m** | **850Mi** | **850Mi** |
| `size_xl` | **5** | **15** | **1500m** | **1500m** | **2000Mi** | **2000Mi** |

**Node Data Collection Collector**:

| Size | Request Memory (Mi) | Limit Memory (Mi) | Request CPU (m) | Limit CPU (m) |
| - | - | - | - | - |
| `size_s` | **150Mi** | **300Mi** | **150m** | **300m** |
| `size_m` | **250Mi** | **500Mi** | **250m** | **500m** |
| `size_l` | **500Mi** | **750Mi** | **500m** | **750m** |
| `size_xl` | **1024Mi** | **1024Mi** | **1000m** | **1000m** |

#### 2. Advanced Pipeline Configuration

For advanced control over the pipeline, you can override the default sizing and fine-tune resource and scaling parameters for the `collectorGateway` and `collectorNode`. The way you apply these advanced settings depends on how you installed Odigos:

* **If you installed Odigos using the CLI:**
  Edit the `odigos-configuration` ConfigMap directly in your cluster. This allows you to set advanced options for both `collectorGateway` and `collectorNode` (such as custom CPU/memory requests, limits, and replica counts).

* **If you installed Odigos using Helm:**
  Make your changes in the `values.yaml` file or by using the `--set` flag during installation or upgrade. **Do not** edit the `odigos-configuration` ConfigMap directly, as Helm manages this resource and will overwrite manual changes.

This method gives you full control over the pipeline configuration, letting you override any default or profile-based settings to match your specific requirements.

Below is an example of how to configure the `collectorGateway` and `collectorNode` parameters:

```yaml theme={null}
  # HYBRID CONFIGURATION APPROACH:
  # By default, all values below are automatically set based on the ResourceSizePreset values.yaml attribute describe above.
  # Uncomment and modify any value to override the automatic sizing for that specific parameter.
  #
  # IMPORTANT CONSTRAINTS:
  # 1. minReplicas must be less or equal to maxReplicas
  # 2. If you set limitMemoryMiB without requestMemoryMiB, request will equal limit
  # 3. If you set requestMemoryMiB without limitMemoryMiB, limit will equal request
  # 4. Same logic applies to CPU settings

collectorGateway:
  # the memory request for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource request
  # of the form "memory: <value>Mi".
  # default value is 600Mi
  # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value.
  # requestMemoryMiB: 600

  # the memory limit for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource limit
  # of the form "memory: <value>Mi".
  # default value is 600Mi
  # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value.
  # limitMemoryMiB: 600

  # the CPU request for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource request
  # of the form "cpu: <value>m".
  # default value is 1000m
  # If you set only requestCPUm, the limitCPUm will be set to the same value.
  # requestCPUm: 1000

  # the CPU limit for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource limit
  # of the form "cpu: <value>m".
  # default value is 1000m
  # If you set only limitCPUm, the requestCPUm will be set to the same value.
  # limitCPUm: 1000

  # The number of replicas for the cluster gateway collector deployment.
  # Also uses in MinReplicas the HPA config.
  # minReplicas: 2
  # The maxReplicas in the HPA config.
  # maxReplicas: 8

  # sets the "limit_mib" parameter in the memory limiter configuration for the collector gateway.
  # it is the hard limit after which a force garbage collection will be performed.
  # if not set, it will be 50Mi below the memory limit.
  # memoryLimiterLimitMiB: 550

  # sets the "spike_limit_mib" parameter in the memory limiter configuration for the collector gateway.
  # note that this is not the processor soft limit, but the diff in MiB between the hard limit and the soft limit.
  # if not specified, this value will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit).
  # memoryLimiterSpikeLimitMiB: 110

  # the GOMEMLIMIT environment variable value for the collector gateway deployment.
  # this is when go runtime will start garbage collection.
  # if not specified, it will be set to 80% of the hard limit of the memory limiter.
  # goMemLimitMiB: 440

collectorNode:
  # RequestMemoryMiB is the memory request for the node collector daemonset.
  # it will be embedded in the daemonset as a resource request of the form "memory: <value>Mi"
  # default value is 250Mi
  # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value.
  # requestMemoryMiB: 250

  # LimitMemoryMiB is the memory limit for the node collector daemonset.
  # it will be embedded in the daemonset as a resource limit of the form "memory: <value>Mi"
  # default value is 500Mi
  # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value.
  # limitMemoryMiB: 500

  # the CPU request for the node collector daemonset.
  # it will be embedded in the daemonset as a resource request
  # of the form "cpu: <value>m".
  # default value is 250m
  # If you set only requestCPUm, the limitCPUm will be set to the same value.
  # requestCPUm: 250

  # the CPU limit for the node collector daemonset.
  # it will be embedded in the daemonset as a resource limit
  # of the form "cpu: <value>m".
  # default value is 500m
  # If you set only limitCPUm, the requestCPUm will be set to the same value.
  # limitCPUm: 500

  # this parameter sets the "limit_mib" parameter in the memory limiter configuration for the node collector.
  # it is the hard limit after which a force garbage collection will be performed.
  # if not set, it will be 50Mi below the memory limit.
  # memoryLimiterLimitMiB: 450

  # this parameter sets the "spike_limit_mib" parameter in the memory limiter configuration for the node collector.
  # note that this is not the processor soft limit, but the diff in Mib between the hard limit and the soft limit.
  # if not set, this will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit).
  # memoryLimiterSpikeLimitMiB: 90

  # the GOMEMLIMIT environment variable value for the node collector daemonset.
  # this is when go runtime will start garbage collection.
  # if not specified, it will be set to 80% of the hard limit of the memory limiter.
  # goMemLimitMiB: 360
```

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Datastreams
Source: https://docs.odigos.io/enterprise/pipeline/datastreams

Datastreams let you split your observability data into multiple, independent pipelines - each routed to its own destination and optionally enhanced with custom actions like sampling, enrichment, or masking.

> [!WARNING]
>
> This feature is in beta. It may be subject to changes and improvements based on user feedback.

# Introduction

Odigos supports Datastreams, a powerful feature that allows you to split telemetry sources into multiple logical streams. Each stream can be independently routed to different destinations and have specific actions applied—such as [sampling](../pipeline/sampling/introduction), attribute enrichment, masking rules, and more.

# Motivation

There are many use cases for adopting Datastreams, including:

1. **Cost optimization** - Route lower-priority services (like internal or backoffice apps) to more cost-effective destinations, such as Parquet files stored in S3.

2. **Targeted processing** - Apply custom processing (e.g., [sampling](../pipeline/sampling/rules/overview), enrichment) to only a subset of services.

3. **Gradual migration** - Test a new destination on specific services before committing to a full migration.

4. **Organizational segmentation** - Split data based on development teams, product areas, or environments.

# Creating Datastreams

**UI**

After [installing Odigos CLI](../setup/installation), run the following command to start the Odigos UI:

```bash theme={null}
odigos ui
```

### In a new Odigos install

You'll be taken to the Setup page. This is where your journey with Odigos begins, and you'll be asked to provide a few initial details:

2. Provide a name for the Datastream
3. Select the telemetry [Sources](../pipeline/sources/introduction) to include
4. Choose the [destination(s)](../backends-overview) to send the data to
5. Click **Done**

### In an existing Odigos install

You can create a new Datastream from the Overview page. Click the `+ NEW` button at the top-left (under the current Datastream views) and follow the same steps as in the setup phase.

**Kubernetes Manifests**

This example demonstrates how to configure `example-source` to send telemetry to `jaeger-example` through the `example` Datastream.

1. Create or update a Source custom resource with the Datastream label, as described in the [linked documentation](../../pipeline/sources/create#workload-source).

*For example:*

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
  labels:
    odigos.io/data-stream-example: "true"
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
```

2. Create or Update a [destination](../../backends-overview) custom resource spec with the Datastream source selector.

*For example:*

```yaml theme={null}
  sourceSelector:
    dataStreams:
    - example
```

Now let's take [Jaeger](../../backends/jaeger) as complete example here:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: jaeger-example
  namespace: odigos-system
spec:
  data:
    JAEGER_URL: <Jaeger OTLP gRPC Endpoint>
  destinationName: jaeger
  signals:
  - TRACES
  type: jaeger
  sourceSelector:
    dataStreams:
    - example
```

> 💡 Tip: Even if you configure datastreams via manifests, you can still view and manage them through the Odigos UI for better visibility.

# Caveats and Considerations

1. **Trace integrity** - Distributed traces often span multiple services that participate in the same operation. If you assign these services to different datastreams, the trace will be broken across destinations. Each destination will only receive part of the trace, making it incomplete and harder to troubleshoot. To preserve trace integrity, keep all services involved in a single transaction within the same datastream.

2. **Action support** - At present, actions defined in Odigos are applied globally to all streams. Stream-specific actions (e.g., custom [sampling rules](../pipeline/sampling/rules/overview) per datastream) are not yet supported but will be in a future release.


# Odigos Own Metrics
Source: https://docs.odigos.io/enterprise/pipeline/odigos-own-metrics

Enable Odigos self-telemetry and the bundled VictoriaMetrics store for pipeline visibility in the UI.

## Overview

Odigos collects **own metrics** about its pipeline — collector throughput, export rates, eBPF instrumentation counters, and other internal telemetry. These metrics describe Odigos itself, not your application workloads.

By default, the bundled metrics store is **disabled** (`ownTelemetry.metricsStore.disabled: true`). When you enable it, Odigos deploys a VictoriaMetrics instance (`odigos-victoriametrics`) in the Odigos namespace and routes own metrics there so the UI can query them.

## What enabling the metrics store does

When `ownTelemetry.metricsStore.disabled` is set to `false`:

1. **VictoriaMetrics is deployed** — a single-replica `odigos-victoriametrics` Deployment with 1-day retention stores own metrics inside the cluster.
2. **Collectors report own telemetry** — node collectors and the cluster gateway scrape their OpenTelemetry internal `/metrics` endpoints and forward the data through the gateway to VictoriaMetrics.
3. **Odiglet instrumentation metrics are collected** — node collectors scrape eBPF instrumentation counters from odiglet and forward them through the gateway to the store.
4. **Odigos UI representation** — metrics are display under "collectors pipeline".

Separately from the metrics store, collectors always export traffic-size metrics directly to the UI server over OTLP. That path powers **per-source and per-destination throughput** (`throughputBytes`, `totalDataSent`) in the UI without requiring the metrics store.

## Enabling

The metrics store is disabled by default to keep the Odigos footprint minimal on small clusters.

**Helm yaml**

```yaml theme={null}
ownTelemetry:
  metricsStore:
    disabled: false
```

**Helm command**

```bash theme={null}
helm upgrade --install odigos odigos/odigos \
  --namespace odigos-system \
  --reuse-values \
  --set ownTelemetry.metricsStore.disabled=false
```

## Configuration options

### Export to external metrics destinations

You can also send own metrics to your existing metrics backends by enabling **Collect Odigos own metrics** on a metrics destination. When either the metrics store or at least one such destination is enabled, collectors start the own-metrics collection pipeline.

See [Odigos own metrics settings](/api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-OdigosOwnMetricsSettings) in the API reference for the full `OdigosOwnMetricsSettings` schema.

## Resource footprint

The bundled VictoriaMetrics instance requests roughly:

| Resource | Request | Limit |
| - | - | - |
| CPU | 100m | 500m |
| Memory | 512Mi | 2Gi |

Metrics are retained for **1 day** on ephemeral storage (`emptyDir`).

## Using own metrics for sizing

Own metrics help you understand how much telemetry Odigos is processing before you change [ResourceSizePreset](./configuration#1-using-sizing-configuration) or collector resources.

1. Enable the metrics store.
2. Instrument representative workloads and let traffic run for a few minutes.
3. Check **source throughput** and **destination throughput** in the Odigos UI.
4. Compare total span throughput (bytes/sec) with the [Benchmarks](./benchmarks) preset tables to pick `size_s` through `size_xl`.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Overview
Source: https://docs.odigos.io/enterprise/pipeline/overview



## High Level Architecture

![Odigos Architecture](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/odigos_architecture.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=d3eb43d4f678adfc133067a0fe684b32)

The Odigos Pipeline is composed of the following components:

* **Instrumentations** - code that records OpenTelemetry signals from your application. It can be code in your application native programming language or eBPF programs that record OpenTelemetry data from the linux kernel.
* **OpenTelemetry SDK** - a library that runs in each instrumented process. It receives the OpenTelemetry data from instrumentations and export it out of the process boundary.
* **Node Collector** - also referred to as `data-collection collector`. Each node in the cluster runs a single instance of this OpenTelemetry collector (DaemonSet). It receives the OpenTelemetry data from the OpenTelemetry SDKs, process it, and exports it to the next component in the pipeline.
* **Cluster Gateway Collector** - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, process it, and export it to the next component in the pipeline.

## Component types:

* **Kubernetes Operators** - reconcile the desired state of cluster resources and deploy / configure the necessary components.
* **DaemonSets** - perform node-level tasks, such as interacting with a process running on the node OS, accessing the Kubelet, and reading logs from the node filesystem.
* **Deployments** - run cluster-level components like the OpenTelemetry Collector and the Odigos UI.

## Component Segments

Odigos is divided into three main segments:

* Instrumentation
  * Inject OpenTelemetry agents to collect the instrumentation data from Odigos sources (instrumented applications)
  * Record and report runtime information from the process environment that is relevant to instrumentation
  * Export collected telemetry to the pipeline.

* Pipeline
  * Offload common or heavy tasks from the instrumented application runtime
  * Apply Odigos Actions on the telemetry data
  * Batch and send the telemetry data to the configured destinations

* Odigos Management
  * orchestrate the pipeline and the instrumentation based on the user's configuration and cluster state,
  * provide a UI for the odigos control plane.

## Pipeline

### Instrumentor (Pipeline Manager)

**Type**: The instrumentor is a Kubernetes operator\

**Role**: The instrumentor is responsible for managing the OpenTelemetry Collectors which implement the OpenTelemetry pipeline.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-instrumentor`

* Start and stop Collectors when needed (e.g. when sources or destinations are added or removed).
* Schedule the node collector to start only after the cluster collector is ready.
* Generate the configuration for the Collectors based on Odigos sources, destinations, and Actions.

### Cluster Collector (Gateway Collector)

**Type**: Cluster collector is a deployment of the OpenTelemetry collector\

**Role**: The cluster collector receives and processes all telemetry collected in the cluster, applies Actions, and exports data to the configured destinations.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-gateway`

* Batch telemetry data for better performance and export it to the configured destinations
* Modify the telemetry data based on the Odigos Actions.
* Apply [tail sampling](./sampling/head-and-tail-sampling) based on the Odigos [sampling rules](./sampling/rules/overview) to reduce the volume of data sent to the destinations.
* Collect metrics about the entities and report it to the UI for monitoring.
* Offload tasks from the agents, to reduce the overhead on the workload runtime environment
* Centralize any common processing that can be shared across multiple agents.

### Node Collector

**Type**: Node collector is a daemon set and will run one pod per node in the cluster\

**Role**: The Node Collector (in the context of Pipeline component) receives telemetry data from agents running on the node, enriches it with node-level attributes, and forward it efficiently and reliably to the cluster collector.\

**K8s Resource**: `kubectl get daemonsets -n odigos-system odigos-data-collection`

* Receive telemetry data from agents running on the node (Odiglet for eBPF based agents, and instrumentation processes for native OpenTelemetry agents).
* Add node-level attributes to the telemetry data (`k8s.node.name`, cloud attributes, etc).
* Batch and send telemetry data to the Cluster Collector, offloading any network overhead from the instrumented application runtime.

> [!NOTE]
>
> The Node Collector is also responsible for recording some k8s metrics, and collecting the generated logs from the node fs. Read more Under the [Instrumentation](#node-collector-2) section.

## Instrumentation

### Instrumentor (Auto Instrumentation Agent Manager)

**Type**: Instrumentor is a Kubernetes operator\

**Role**: The Instrumentor is responsible for managing auto instrumentation agents which collect telemetry data from Odigos sources.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-instrumentor`

* Create and manage `instrumentationconfig.odigos.io` objects
  * Trigger Odiglet to inspect runtime information of workloads.
  * Trigger deployment of the Odigos pipeline.
  * Merge Odigos instrumentation rules and config to a workload scope configuration for the agent.
* Inject the instrumentation device into the workload manifest resource as virtual device.
* Monitor the `source.odigos.io` CRD and cleanup any instrumentation related resources when a source is removed.
* Monitor the Odigos sources and destinations to trigger deployment of the Odigos pipeline.
* Reconcile Odigos configuration to run validations, set defaults, and apply Odigos Profiles (configuration presets) to the cluster.

### Odiglet

**Type**: Odiglet is a daemon set and will run one pod per node in the cluster\

**Role**: Odiglet sets up the node environment to run telemetry collection agents.\

**K8s Resource**: `kubectl get daemonsets -n odigos-system odiglet`

* Inspect container runtime details of instrumented sources (to determine programming language, relevant environment variables, runtime versions, etc).
* Implement the Kubernetes virtual instrumentation devices. In the presence of a supported device on the pod, Odiglet will mount the agent code from file system to the pod and add the relevant environment variables to initiate instrumentation.
* For eBPF-based instrumentation, Odiglet will inject the eBPF code into the pod, pick up telemetry from eBPF, and push it downstream to the Node Collector.
* Monitor for config changes to instrumentations and update the relevant agents accordingly at runtime.

### Node Collector

**Role**: Node collector (in the context of instrumentation) is responsible for scraping Kubernetes metrics, and collect logs from the node filesystem.

* When a metrics destination exists, the node collector will use the OpenTelemetry Collector ["kubeletstats receiver"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver) to scrape k8s related metrics from the Kubelet and export them as OpenTelemetry metrics to the cluster collector.
* When a logs destination exists, the node collector will use the OpenTelemetry Collector ["filelog receiver"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) to read the relevant pod logs from the node filesystem and export them as OpenTelemetry logs to the cluster collector.

## Odigos

### Odigos UI

**Type**: Odigos UI is a web application backed by a deployment in the cluster and accessible by the user's browser.\

**Role**: The Odigos UI is a web application that allows users to configure various aspects of the Odigos installation.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-ui`

* Serve as web server for the Odigos UI
  * Serve static files to the frontend
  * Provide a GraphQL API for the frontend, which in turn interacts with Kubernetes API to manage the Odigos installation.
* Provide a user interface for monitoring and managing the Odigos installation, including the sources, destinations, actions, and instrumentation rules.
* Examine the current state of Odigos and show any errors.


# Code Attributes
Source: https://docs.odigos.io/enterprise/pipeline/rules/codeattributes

The "Code Attributes" rule can be used to add [semantic Code attributes](https://opentelemetry.io/docs/specs/semconv/attributes-registry/code/) to spans. Code attributes can help trace back an operation in a trace to the code that generated it.

## Considerations

> [!WARNING]
>
> Before enabling **code attributes**, please note the following:
>
> * Size of `code.stacktrace` can be large and may increase the size of your spans. This can impact the performance of your application and the cost of processing, storing and analyzing traces.
> * Code attributes have various level of support, depending on the programming language and instrumentation library used.

## Configuration Options

> **column**
>
> **column** `boolean` : Indicates whether to record the `code.column` attribute for the column number in the `code.filepath` best representing the operation.
>
> * This field is *optional*, and defaults to `false`

> **filePath**
>
> **filePath** `boolean` : Indicates whether to record the `code.filepath` attribute for the source code file name that identifies the code unit.
>
> * This field is *optional*, and defaults to `false`

> **function**
>
> **function** `boolean` : Indicates whether to record the `code.function` attribute for the method or function name, or equivalent.
>
> * This field is *optional*, and defaults to `false`

> **lineNumber**
>
> **lineNumber** `boolean` : Indicates whether to record the `code.lineno` attribute for the line number in the `code.filepath` best representing the operation.
>
> * This field is *optional*, and defaults to `false`

> **namespace**
>
> **namespace** `boolean` : Indicates whether to record the `code.namespace` attribute for the “namespace” within which `code.function` is defined. Usually the qualified class or module name, such that `code.namespace` + some separator + `code.function` form a unique identifier for the code unit.
>
> * This field is *optional*, and defaults to `false`

> **stackTrace**
>
> **stackTrace** \`boolean
>
> * This field is *optional*, and defaults to `false`

> [!NOTE]
>
> Any unspecified options will be set to the default value of `false` (not recorded).

## Basic Example

The following example demonstrates how to enable code attributes for all supported workloads and instrumentation libraries in the cluster.

**Step 1**

Create a YAML file with the following content:

```yaml code-attributes.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: collect-all-payloads
  namespace: odigos-system
spec:
  ruleName: "collect all supported code attributes"
  codeAttributes:
    column: false
    filePath: true # recommended
    function: true # recommended
    lineNumber: true # recommended
    namespace: false
    stackTrace: false
```

**Step 2**

Apply the action to the cluster:

```shell theme={null}
kubectl apply -f code-attributes.yaml
```


# Custom Instrumentation
Source: https://docs.odigos.io/enterprise/pipeline/rules/custominstrumentation

The "Custom Instrumentation" rule can be used to instrument specific functions in Go, Java, or PHP applications.

## Considerations

If you wish to create custom instrumentations for Go, Java, or PHP applications, you can define a "Custom Instrumentation" rule via the Odigos InstrumentationRule CRD or the Odigos UI.

This feature is a great way to extend Odigos's auto-instrumentation capabilities to cover your specific use cases and application logic.

For Go and Java, custom instrumentation is applied dynamically via eBPF. For PHP, probes are delivered as environment variables and require a pod restart to take effect.

## Configuration Options

> **Golang**
>
> > [!WARNING]
> >
> > When `functionName` is specified, `receiverName` and `receiverMethodName` must NOT be specified, and vice versa, if `receiverName` and `receiverMethodName` are specified, `functionName` must NOT be specified.
> >
> > `packageName` is required in all cases.
>
> > **packageName**
> >
> > **packageName** `string` - The name of the package containing the function to instrument.
> >
> > * This field is *required*
>
> > **functionName**
> >
> > **functionName** `string` - The name of the function to instrument.
> >
> > * This field is *optional*
>
> > **receiverName**
> >
> > **receiverName** `string` - The name of the receiver type if the function is a method.
> >
> > * This field is *optional*
>
> > **receiverMethodName**
> >
> > **receiverMethodName** `string` - The name of the method receiver if the function is a method.
> >
> > * This field is *optional*

> **Java**
>
> > **className**
> >
> > **className** `string` - The fully qualified name of the class containing the method to instrument.
> >
> > * This field is *required*
>
> > **methodName**
> >
> > **methodName** `string` - The name of the method to instrument.
> >
> > * This field is *required*

> **PHP**
>
> > **className**
> >
> > **className** `string` - The fully qualified class name to instrument. Leave empty for a global function.
> >
> > * This field is *optional*
>
> > **functionName**
> >
> > **functionName** `string` - The name of the function or method to instrument.
> >
> > * This field is *required*

## Basic Example

The following example shows how to create a "Custom Instrumentation" rule that instruments a function named `ProcessOrder` in the `github.com/example/ecommerce` package.

**Step 1**

Create a YAML file named `custom-instrumentation.yaml` with the following content:

> **Package and function (Golang)**
>
> ```yaml theme={null}
> apiVersion: odigos.io/v1alpha1
> kind: InstrumentationRule
> metadata:
>     name: custom-instrumentation-rule
>     namespace: odigos-system
> spec:
>     ruleName: "Custom Instrumentation for ProcessOrder"
>     customInstrumentations:
>         golang:
>           - packageName: "github.com/example/ecommerce"
>             functionName: "ProcessOrder"
> ```

> **Receiver method (Golang)**
>
> ```yaml theme={null}
> apiVersion: odigos.io/v1alpha1
> kind: InstrumentationRule
> metadata:
>     name: custom-instrumentation-rule
>     namespace: odigos-system
> spec:
>     ruleName: "Custom Instrumentation for ProcessOrder Method"
>     customInstrumentations:
>         golang:
>           - packageName: "github.com/example/ecommerce"
>             receiverName: "OrderProcessor"
>             receiverMethodName: "ProcessOrder"
> ```

> **Class and method (Java)**
>
> ```yaml theme={null}
> apiVersion: odigos.io/v1alpha1
> kind: InstrumentationRule
> metadata:
>     name: custom-instrumentation-rule
>     namespace: odigos-system
> spec:
>     ruleName: "Custom Instrumentation for getCurrencyInfo"
>     customInstrumentations:
>         java:
>           - className: "com.example.CurrencyService"
>             methodName: "getCurrencyInfo"
> ```

> **Class and function (PHP)**
>
> ```yaml theme={null}
> apiVersion: odigos.io/v1alpha1
> kind: InstrumentationRule
> metadata:
>     name: custom-instrumentation-rule
>     namespace: odigos-system
> spec:
>     ruleName: "Custom Instrumentation for processOrder"
>     customInstrumentations:
>         php:
>           - className: "App\\Service\\OrderService"
>             functionName: "processOrder"
> ```

**Step 2**

Apply the action to the cluster:

```shell theme={null}
kubectl apply -f custom-instrumentation.yaml
```


# Headers Collection
Source: https://docs.odigos.io/enterprise/pipeline/rules/headerscollection

The "Headers Collection" rule can be used to add span attributes by collecting HTTP Headers data.

## Considerations

> [!WARNING]
>
> Before enabling **headers collection**, please note the following:
>
> * The support for headers collection varies between instrumentation libraries and languages.
> * HTTP headers might contain PII data (like cookies or API keys), thus it is recommended to review each requested header before configuring it to avoid sensitive information.

## Configuration Options

> **headerKeys**
>
> **headerKeys** `string[]` : Limit payload collection to specific header keys.
>
> * This field is *optional*
> * If not specified, no headers will be collected.
> * If set to `["*"]`, all supported headers will be collected.

> [!NOTE]
>
> Any unspecified headers will not be recorded.

## Basic Example

The following example demonstrates how to enable headers collection for all supported workloads and instrumentation libraries in the cluster.

**Step 1**

Create a YAML file with the following content:

```yaml headers-collection.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: collect-all-headers
  namespace: odigos-system
spec:
  ruleName: "collect all supported headers"
  headersCollection:
    headerKeys: ["*"]
```

**Step 2**

Apply the action to the cluster:

```shell theme={null}
kubectl apply -f headers-collection.yaml
```


# Introduction
Source: https://docs.odigos.io/enterprise/pipeline/rules/introduction

Instrumentation Rules control how telemetry is recorded from your application. A rule can be applied to a set of Kubernetes workloads and instrumentation libraries.

## Rule Types:

* [Code Attributes](./codeattributes)
* [Headers Collection](./headerscollection)
* [Payload Collection](./payloadcollection)
* [Custom Instrumentations](./custominstrumentation)
* [Network Metrics](./networkmetrics)

## Configuration Options

> **ruleName**
>
> **ruleName** `string` : Allows you to attach a meaningful name to the rule for convenience and documentation.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the rule for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the rule, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> **workloads**
>
> **workloads** `[namespace, name, kind]` : Allows you to apply/limit the rule to specific workloads. If not specified, the rule will be applied to all workloads.
>
> * This field is *optional*, and defaults to `nil` (all workloads)
>
> * &#x20;Empty array will make the rule ineffective
>
> > **namespace**
> >
> > **namespace** `string` - The name of the Kubernetes namespace.
> >
> > * This field is *required*
>
> > **name**
> >
> > **name** `string` - The name of the Kubernetes workload.
> >
> > * This field is *required*
>
> > **kind**
> >
> > **kind** `string` - The kind of the Kubernetes workload.
> >
> > * This field is *required*
> > * Supported values: `Deployment`, `DaemonSet`, `StatefulSet`

> **instrumentationLibraries**
>
> **instrumentationLibraries** `[name, language, spanKind]` : Allows you to apply/limit the rule to specific instrumentation libraries. If not specified, the rule will be applied to all instrumentation libraries.
>
> * This field is *optional*, and defaults to `nil` (all libraries)
>
> * &#x20;Empty array will make the rule ineffective
>
> > **name**
> >
> > **name** `string` - The unique name of the instrumentation library.
> >
> > * This field is *required*
>
> > **language**
> >
> > **language** `string` - The programming language of the instrumentation library.
> >
> > * This field is *required*
> > * Alphabetical, lowercase, and no spaces
>
> > **spanKind**
> >
> > **spanKind** `string` - The span kind of the instrumentation library.
> >
> > * This field is *required*
> > * Supported values: `client`, `server`, `producer`, `consumer`, `internal`

## Apply to Sources

Odigos allows you to apply/limit rules to specific sources.

* To apply a rule to all sources, omit the `workloads` field.
* To apply a rule to specific sources, provide an array of workload objects (identified by its namespace, name, and kind).

> [!NOTE]
>
> Odigos allows you to define multiple rules, and will merge them together to create a single rule per source. Conflicting options will be resolved by choosing to record the attribute, if it's specified by one rule (at least).

## Apply to Instrumentation Libraries

Odigos allows you to apply/limit rules to specific instrumentation libraries.

* To apply a rule to all instrumentation libraries, omit the `instrumentationLibraries` field.
* To apply a rule to specific instrumentation libraries, provide an array of instrumentation libraries (identified by its unique name, language, and span kind for golang).

> [!NOTE]
>
> Any rule for a specific instrumentation library will take precedence over a rule that does not specify a library.

> [!TIP]
>
> This is an advanced feature. It is recommended to set the same rules for all instrumentation libraries, and only use this feature when you need fine-grained control.


# Network Metrics
Source: https://docs.odigos.io/enterprise/pipeline/rules/networkmetrics

Enable network flow and TCP stats metrics for instrumented workloads via InstrumentationRule

The **Network Metrics** rule enables network flow metrics and TCP stats metrics for matching workloads. These metrics are collected in odiglet and exported over OTLP alongside other workload telemetry.

Learn more about the underlying collectors and metric families in the [OBI instrumentation guide](/oss/instrumentations/obi).

## Considerations

> [!WARNING]
>
> Before enabling **network metrics**, please note the following:
>
> * Enable the cluster infrastructure in Helm by setting `metricsSources.networkMetrics.enabled` to `true` (it defaults to disabled).

## Configuration Options

> **networkMetrics**
>
> **networkMetrics** `object` : Enable network flow and TCP stats metrics for scoped workloads. Setting this field (even as an empty object `{}`) enables collection; omit it to leave metrics disabled.

> [!NOTE]
>
> Multiple rules merge with OR semantics: if any matching rule enables network metrics, they are enabled for that source/language.

## Cluster infrastructure for network flow metrics

Network flow metrics require odiglet to run with [`hostNetwork`](https://kubernetes.io/docs/reference/kubernetes-api/core/pod-v1/#:~:text=%60hostNetwork%60), which makes the pod share the host's network namespace instead of getting its own. Configure this once at the cluster level in Helm:

```yaml theme={null}
metricsSources:
  networkMetrics:
    enabled: true
```

The InstrumentationRule alone is not enough for network flow metrics without that Helm setting.

## Basic Example

> [!NOTE]
>
> Create `InstrumentationRule` resources in the Odigos installation namespace (typically `odigos-system`). Rules in other namespaces are not picked up by Odigos.

The following example enables network metrics cluster-wide for all workloads.

**1. Enable network metrics infrastructure**

Set `metricsSources.networkMetrics.enabled` to `true` in your Odigos Helm values and upgrade the release.

**2. Create the InstrumentationRule**

Create a YAML file with the following content:

```yaml network-metrics.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: network-metrics
  namespace: odigos-system
spec:
  ruleName: "Network metrics"
  networkMetrics: {}
```

**3. Apply the rule**

```shell theme={null}
kubectl apply -f network-metrics.yaml
```

Network metrics collection will run on any new pod created after the rule is applied. Existing pods will not be affected and require a restart to initiate metrics collection.

## Scoped Example

To limit the rule to a single workload, add `scopes`. Omit `scopes` (or leave it empty) to apply the rule cluster-wide.

```yaml network-metrics-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: network-metrics-my-app
  namespace: odigos-system
spec:
  ruleName: "Network metrics for my-app"
  scopes:
    sources:
      - name: my-app
        namespace: default
        kind: Deployment
  networkMetrics: {}
```

Apply with:

```shell theme={null}
kubectl apply -f network-metrics-scoped.yaml
```

## Exported Metrics

When this rule is enabled, the following metrics are exported for matching workloads:

| Name (OpenTelemetry) | Name (Prometheus) | Type | Unit | Description |
| - | - | - | - | - |
| `odigos.network.flow.bytes` | `odigos_network_flow_bytes_total` | Counter | bytes | Bytes sent from a source network endpoint to a destination network endpoint |
| `odigos.network.flow.packets` | `odigos_network_flow_packets_total` | Counter | | Packets sent from a source network endpoint to a destination network endpoint |
| `odigos.network.inter.zone.bytes` | `odigos_network_inter_zone_bytes_total` | Counter | bytes | Bytes flowing between cloud availability zones in your cluster (Kubernetes only) |
| `odigos.stat.tcp.rtt` | `odigos_stat_tcp_rtt_seconds` | Histogram | seconds | TCP round-trip time (RTT) latency observed between network endpoints |
| `odigos.stat.tcp.failed.connections` | `odigos_stat_tcp_failed_connections_total` | Counter | 1 | Failed TCP connection attempts between endpoints, labeled by failure reason |

> [!NOTE]
>
> These metrics are collected using [OBI](https://opentelemetry.io/docs/zero-code/obi). See the [OBI instrumentation guide](/oss/instrumentations/obi) to learn more about the collectors, metric families, and available attributes.


# Payload Collection
Source: https://docs.odigos.io/enterprise/pipeline/rules/payloadcollection

The "Payload Collection" rule can be used to add span attributes by collecting payload data from various sources like HTTP requests, HTTP responses, database queries, and messaging systems.

## Considerations

> [!WARNING]
>
> Before enabling **payload collection**, please note the following:
>
> * PII (Personally Identifiable Information) and other potentially sensitive data may be present in the payload. Evaluate the risk of collecting this data, and consider using the [PII Masking Action](../../pipeline/actions/attributes/piimasking) to mask sensitive data.
> * Payload data can be large and may increase the size of your spans. This can impact the performance of your application and the cost of processing, storing and analyzing traces.
> * The support for payload collection varies between instrumentation libraries and languages. Not all libraries support payload collection, and the supported payload types and formats may differ. Consult the documentation of the instrumentation library you are using for more information.

## Configuration Options

> **httpRequest**
>
> **httpRequest** `object` : Collect HTTP request payload data when available. Can be a client (outgoing) request or a server (incoming) request, depending on the instrumentation library.
>
> * This field is *optional*
>
> > **mimeTypes**
> >
> > **mimeTypes** `string[]` : Limit payload collection to specific mime types based on the content-type header.
> >
> > * This field is *optional*, and defaults to `nil` (all mime types)
> > * &#x20;Empty array will make the rule ineffective
>
> > **maxPayloadLength**
> >
> > **maxPayloadLength** `number` : Maximum length of the payload to collect. If the payload is longer than this value, it will be truncated or dropped, based on the value of `dropPartialPayloads`.
> >
> > * This field is *optional*
>
> > **dropPartialPayloads**
> >
> > **dropPartialPayloads** `boolean` : If the payload is larger than the `maxPayloadLength`, this parameter will determine if the payload should be partially collected up to the allowed length, or not collected at all. This is useful if you require some decoding of the payload (like json) and having it partially is not useful.
> >
> > * This field is *optional*, and defaults to `false`

> **httpResponse**
>
> **httpResponse** `object` : Collect HTTP response payload data when available. Can be a client (incoming) response or a server (outgoing) response, depending on the instrumentation library
>
> * This field is *optional*
>
> > **mimeTypes**
> >
> > **mimeTypes** `string[]` : Limit payload collection to specific mime types based on the content-type header.
> >
> > * This field is *optional*, and defaults to `nil` (all mime types)
> > * &#x20;Empty array will make the rule ineffective
>
> > **maxPayloadLength**
> >
> > **maxPayloadLength** `number` : Maximum length of the payload to collect. If the payload is longer than this value, it will be truncated or dropped, based on the value of `dropPartialPayloads`.
> >
> > * This field is *optional*
>
> > **dropPartialPayloads**
> >
> > **dropPartialPayloads** `boolean` : If the payload is larger than the `maxPayloadLength`, this parameter will determine if the payload should be partially collected up to the allowed length, or not collected at all. This is useful if you require some decoding of the payload (like json) and having it partially is not useful.
> >
> > * This field is *optional*, and defaults to `false`

> **dbQuery**
>
> **dbQuery** `object` : Collect database query payload info when available.
>
> * This field is *optional*
>
> > **mimeTypes**
> >
> > **mimeTypes** `string[]` : Limit payload collection to specific mime types based on the content-type header.
> >
> > * This field is *optional*, and defaults to `nil` (all mime types)
> > * &#x20;Empty array will make the rule ineffective
>
> > **maxPayloadLength**
> >
> > **maxPayloadLength** `number` : Maximum length of the payload to collect. If the payload is longer than this value, it will be truncated or dropped, based on the value of `dropPartialPayloads`.
> >
> > * This field is *optional*

> **messaging**
>
> **messaging** `object` : Collect messaging system operation payload info when available.
>
> * This field is *optional*
>
> > **mimeTypes**
> >
> > **mimeTypes** `string[]` : Limit payload collection to specific mime types based on the content-type header.
> >
> > * This field is *optional*, and defaults to `nil` (all mime types)
> > * &#x20;Empty array will make the rule ineffective
>
> > **maxPayloadLength**
> >
> > **maxPayloadLength** `number` : Maximum length of the payload to collect. If the payload is longer than this value, it will be truncated or dropped, based on the value of `dropPartialPayloads`.
> >
> > * This field is *optional*

> [!NOTE]
>
> Any unspecified options will fallback to a reasonable default value provided by the instrumentation library (recommended).

## Basic Example

The following example demonstrates how to enable payload collection for all supported workloads and instrumentation libraries in the cluster.

**Step 1**

Create a YAML file with the following content:

```yaml payload-collection.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: collect-all-payloads
  namespace: odigos-system
spec:
  ruleName: "collect all supported payloads"
  payloadCollection:
    httpRequest: {}
    httpResponse: {}
    dbQuery: {}
    messaging: {}
```

**Step 2**

Apply the action to the cluster:

```shell theme={null}
kubectl apply -f payload-collection.yaml
```

## Full Example

The following example is a demonstration of all the options available in the "Payload Collection" Rule.
It is not meant to be used "as is", but rather as a reference to customize the rule to your needs.

**Step 1**

Create a YAML file with the following content:

```yaml payload-collection.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: full-payload-collection-example
  namespace: odigos-system
spec:
  ruleName: "Full example for payload collection"
  disabled: false
  notes: "This rule showcase all the options available for payload collection rule"
  workloads:
    - kind: Deployment
      name: example-deployment
      namespace: default
    - kind: DaemonSet
      name: example-ds
      namespace: default
  instrumentationLibraries:
    - language: go
      name: "net/http"
      spanKind: server
    - language: go
      name: "database/sql"
      spanKind: client
  payloadCollection:
    httpRequest:
        mimeTypes:
        - "application/json"
        maxPayloadLength: 2048
        dropPartialPayloads: true
    httpResponse:
        mimeTypes:
        - "application/json"
        - "text/plain"
        maxPayloadLength: 8096
        dropPartialPayloads: true
    dbQuery:
        maxPayloadLength: 1024
        dropPartialPayloads: true
    messaging:
        maxPayloadLength: 512
        dropPartialPayloads: false
```

**Step 2**

Apply the action to the cluster:

```shell theme={null}
kubectl apply -f payload-collection.yaml
```


# Configuration
Source: https://docs.odigos.io/enterprise/pipeline/sampling/configuration

Configure sampling actions in Odigos Enterprise.

You can configure sampling rules in the **UI** or as **YAML** (Kubernetes **CRDs**). The schema matches the same concepts as the UI, so GitOps workflows can express the same behavior as in the product.

Every rule has three parts: a **Source scope** that defines which workloads the rule applies to, an **Operation** that picks the specific operations within that scope, and a **Keep Percentage** that controls how much matching traffic is retained. The Source scope options are identical across all three categories; the Operation and Keep Percentage options differ by category. Each rule can also carry an optional **Name** and **Notes** for easier identification.

> [!TIP]
>
> This page is the **field reference**. For end-to-end walkthroughs—UI steps and the equivalent YAML side by side—see [Examples](./examples).

## Source scope

A rule can be scoped to one of:

* **Entire cluster** — the rule applies to every source in the cluster.
* **Specific Source** — one or more named sources (services).
* **Specific Namespace** — one or more Kubernetes namespaces.
* **Specific Programming Language** — one or more runtimes: **Python**, **Java**, **NodeJS**, **.Net**, **PHP**, **Ruby**, or **Go**.

You can select multiple values within a scope type—for example several namespaces, or several languages.

> [!WARNING]
>
> When you populate more than one scope type on the same rule, the selections are combined with **AND**, not **OR**. A rule scoped to "Specific Namespace = `prod`" plus "Specific Programming Language = Java" matches **only** Java workloads in the `prod` namespace; it does not match every workload in `prod` or every Java workload across the cluster.

## Operation

The Operation options depend on the rule's category.

**Noisy Operation**

* **All Operations** — every operation in scope.
* **HTTP Server** — match HTTP requests received by a server. Configure:
  * An **HTTP route** (exact path) or **HTTP route prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **HTTP Client** — match outbound HTTP calls made by the source. Configure:
  * A **server address** (the remote host being called).
  * A **templated path** (exact) or **templated path prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **gRPC Server** — match incoming gRPC calls received by a server. Configure:
  * A **service** — the fully-qualified gRPC service name (e.g. `acme.inventory.v1.InventoryService`). Set only the service to match every method on that service.
  * A **method** — the bare method name (e.g. `ListItems`). Combine with **service** for a single specific RPC.
* **gRPC Client** — match outbound gRPC calls made by the source. Configure:
  * A **service** — the fully-qualified gRPC service name (e.g. `acme.inventory.v1.InventoryService`). Set only the service to match every method on that service.
  * A **method** — the bare method name (e.g. `ListItems`). Combine with **service** for a single specific RPC.
  * A **server address** (the remote host being called).

**Highly Relevant**

* **All Operations** — every operation in scope.
* **HTTP Server** — match HTTP requests received by a server. Configure:
  * An **HTTP route** (exact path) or **HTTP route prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **Kafka Consumer** — match traces from a Kafka consumer. Configure the **Kafka topic**.
* **Kafka Producer** — match traces from a Kafka producer. Configure the **Kafka topic**.

HTTP Client is **not** available in this category.

**Cost Reduction**

* **All Operations** — every operation in scope.
* **HTTP Server** — match HTTP requests received by a server. Configure:
  * An **HTTP route** (exact path) or **HTTP route prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **Kafka Consumer** — match traces from a Kafka consumer. Configure the **Kafka topic**.
* **Kafka Producer** — match traces from a Kafka producer. Configure the **Kafka topic**.

HTTP Client is **not** available in this category.

## Keep Percentage

Each rule defines what fraction of matching traffic is retained. The available options differ by category:

| Category | Options |
| - | - |
| Noisy Operation | **Drop All**, or **Keep at most** *N*% |
| Highly Relevant | **Keep All**, or **Keep at least** *N*% |
| Cost Reduction | **Keep at most** *N*% (set *N* = 0 to drop everything) |

**Keep at most** is a cap; **Keep at least** is a floor. See [Rule evaluation](./rules/overview#rule-evaluation) for how these percentages interact within a category and between Highly Relevant and Cost Reduction.

## Enabled

Every rule has an **Enabled** toggle that controls whether it participates in sampling decisions. Disabled rules stay in your configuration but are excluded from evaluation, which makes the toggle useful for several patterns:

* **Temporary suspension.** Turn a rule off during an incident, debugging session, or evaluation period when you need broader visibility—without losing the rule's configuration.
* **Disable vs. delete.** Disabling preserves the matcher, scope, and Keep Percentage so you can re-enable later without re-authoring. Use it when a rule may still be useful later but isn't needed right now.
* **"What-if" metrics.** Disabled rules continue to record matching metrics: how many traces *would have* matched, and how many would have been kept or dropped if the rule were active. Useful for sizing a candidate rule's impact before committing to it.

See [Disabling a rule](./examples#disabling-a-rule) for UI screenshots and the equivalent YAML.

## Name and Notes

Both fields are optional and exist purely to make rules easier to identify and maintain—they have **no effect** on evaluation or matching.

* **Name** — a short, free-text label for the rule. The name appears in the rule list in the UI and may also surface on metrics and span attributes, so use something descriptive that makes the rule recognizable in dashboards and queries.
* **Notes** — free-text describing **why** the rule exists. Use this to capture context that isn't obvious from the matcher itself: the incident that triggered the rule, the team that owns it, or the cost target it's serving. Especially useful when auditing rules or onboarding new operators.

> [!TIP]
>
> Treat Name and Notes like a commit message for your sampling policy. A good Name + Notes pair turns "what does this rule do?" into a one-line answer at a glance.


# Considerations
Source: https://docs.odigos.io/enterprise/pipeline/sampling/considerations

Operational and design considerations for sampling in Odigos Enterprise.

Operational and design considerations to keep in mind when running sampling in production.

## Gateway load, OOM, and scaling

When the gateway accepts a trace for tail evaluation, it allocates memory to hold spans until the [aggregation window](./head-and-tail-sampling#aggregation-window) closes. If the gateway isn't sized correctly, bursts of traffic can cause **OOM** restarts. Symptoms include **dropped spans** or collector crashes that aren't caused by your sampling rules.

For tail sampling, prefer **scaling up** (bigger collector pods) over **scaling out** (more replicas). All spans of a single trace must land on the same pod for aggregation, so fewer, larger pods simplify routing and load balancing. Validate sizing against your own load—there's no one-size-fits-all.

## When any sampler enables tail mode

> [!WARNING]
>
> Any sampling rule that requires tail evaluation activates **tail mode** for the entire cluster, even rules that look inactive or minimal—for example a placeholder service name or a broad `/` route. Once tail mode is on, the gateway buffers each trace for the [aggregation window](./head-and-tail-sampling#aggregation-window), which increases memory use, export latency, and the risk of dropped spans. If you don't need tail behavior, either remove any unused or test sampling rules from the cluster, or disable tail sampling explicitly in Odigos configuration (Helm values, or the UI **Settings** page).


# Examples
Source: https://docs.odigos.io/enterprise/pipeline/sampling/examples

End-to-end walkthroughs of common sampling rules in Odigos Enterprise, with UI steps and equivalent YAML.

End-to-end walkthroughs of common sampling rules. Each example shows the rule in the **UI** and as a **YAML** `Sampling` resource (the same CRD applied via `kubectl apply` or GitOps). Use these as starting points and adjust scope, operations, and percentages to match your environment.

> [!NOTE]
>
> YAML examples are accurate against the current `Sampling` CRD schema. The screenshots for each scenario reflect the live UI; minor styling may shift between releases.

For the field reference behind each example—every Source scope option, every Operation option, every Keep Percentage choice—see [Configuration](./configuration). For the rules engine itself, see [Rules](./rules/overview).

> [!NOTE]
>
> **YAML examples on this page** conform to the `Sampling` CRD (`odigos.io/v1alpha1`). For every field, type, validation rule, and default—including options not covered by the examples below—see the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling).

> [!TIP]
>
> **Scope rules to only the workloads they need to evaluate.** Cluster-wide rules are checked against every trace; tightly scoped rules (Specific Source, Namespace, or Language) are skipped entirely for unrelated workloads, which avoids wasted matching work in the gateway. Several examples below illustrate this pattern.

## 1. Drop a noisy custom endpoint

**Goal:** Drop traces for a high-volume polling endpoint (`/api/v1/poll`) that produces many traces but is not useful for debugging or SLOs. This is a typical **Noisy** rule for traffic that's neither a Kubernetes probe nor a metrics scrape, but is still pure noise.

**UI**

Create a Noisy rule with:

* **Name:** `Drop polling endpoint`
* **Notes:** `High-volume poll endpoint with no SLO/debug value.`
* **Source scope:** Specific Source = `odimall / Deployment / notification-service`
* **Operation:** HTTP Server, route prefix `/api/v1/poll`, method **GET**
* **Drop Percentage:** Drop All

![UI screenshot of a Noisy rule that drops GET traces for /api/v1/poll, scoped to the notification-service Deployment in the odimall namespace.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/drop-poll-endpoint-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=102aa70d7ece6368d65564256293c7f0)

**YAML**

```yaml drop-poll-endpoint.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: drop-poll-endpoint
  namespace: odigos-system
spec:
  name: Drop /api/v1/poll
  noisyOperations:
    - name: Drop polling endpoint
      notes: High-volume poll endpoint with no SLO/debug value.
      percentageAtMost: 0
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: notification-service
      operation:
        httpServer:
          routePrefix: /api/v1/poll
          method: GET
```

## 2. Keep slow requests at 100%

**Goal:** Always retain traces for `GET /api/products` whenever they take longer than 500 ms. This is a **Highly Relevant** duration rule—a complement to the built-in [Keep All Error Traces](./rules/highly-relevant#built-in-rule-keep-all-error-traces) rule, focused on latency rather than errors.

**UI**

Create a Highly Relevant rule with:

* **Name:** `Slow GET /api/products`
* **Notes:** `SLO-relevant path; retain anything over 500ms for investigation.`
* **Rule Type:** Duration
* **Source scope:** Specific Source = `odimall / Deployment / frontend`
* **Operation:** HTTP Server, route `/api/products`, method **GET**
* **Keep Percentage:** Keep All
* **Duration:** 500ms

![UI screenshot of a Highly Relevant rule that keeps 100% of slow GET /api/products traces, scoped to the frontend Deployment in the odimall namespace.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/keep-slow-products-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=a20b0f51a16027c4a32791d855a3ff0e)

**YAML**

```yaml keep-slow-products.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: keep-slow-products
  namespace: odigos-system
spec:
  name: Keep slow product reads
  highlyRelevantOperations:
    - name: Slow GET /api/products
      notes: SLO-relevant path; retain anything over 500ms for investigation.
      durationAtLeastMs: 500
      percentageAtLeast: 100
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: frontend
      operation:
        httpServer:
          route: /api/products
          method: GET
```

## 3. Apply a cluster-wide 5% baseline

**Goal:** Keep 5% of all cluster traffic as a statistical baseline, regardless of source or operation. This is the same shape as the built-in [Drop Traces Cluster-Wide](./rules/cost-reduction#built-in-rule-drop-traces-cluster-wide) rule and is the simplest way to cut volume after Noisy and Highly Relevant rules have run.

**UI**

Create a Cost Reduction rule with:

* **Name:** `5% baseline`
* **Notes:** `Statistical baseline for non-noisy, non-important traffic.`
* **Source scope:** Entire cluster
* **Operation:** All Operations
* **Drop Percentage:** 5

![UI screenshot of a Cost Reduction rule that caps cluster-wide retention at 5%.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/baseline-cluster-sample-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=50cc53df3883927c0a1702c856c02fe8)

**YAML**

```yaml baseline-cluster-sample.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: baseline-cluster-sample
  namespace: odigos-system
spec:
  name: Cluster baseline 5%
  costReductionRules:
    - name: 5% baseline
      notes: Statistical baseline for non-noisy, non-important traffic.
      percentageAtMost: 5
```

> [!TIP]
>
> Cost Reduction rules with **no** Operation block apply to every operation in the scope. Combine with Highly Relevant rules to make sure error and latency traces survive the cap—Highly Relevant always wins over Cost Reduction.

## 4. Scope a rule to namespace and language

**Goal:** Drop scrape traffic to `/internal/metrics` for **Java** workloads in the `prod` namespace only—leave other namespaces and other languages untouched. This example demonstrates the **AND** semantics across scope types: a single source scope item with both `workloadNamespace` and `workloadLanguage` set.

**UI**

Create a Noisy rule with:

* **Name:** `Java metrics scrape in prod`
* **Notes:** `Internal Prometheus scrape endpoint; the scrape itself isn't useful for tracing.`
* **Source scope:** Specific Namespace = `prod` **and** Specific Programming Language = **Java**
* **Operation:** HTTP Server, route `/internal/metrics`, method **GET**
* **Drop Percentage:** Drop All

![UI screenshot of a Noisy rule scoped with namespace prod AND language Java.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/drop-metrics-java-prod-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=275e5a1e330c052c4c75d27895afbe88)

**YAML**

```yaml drop-metrics-java-prod.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: drop-metrics-java-prod
  namespace: odigos-system
spec:
  name: Drop /internal/metrics (Java in prod)
  noisyOperations:
    - name: Java metrics scrape in prod
      notes: Internal Prometheus scrape endpoint; the scrape itself isn't useful for tracing.
      percentageAtMost: 0
      sourceScopes:
        - workloadNamespace: prod
          workloadLanguage: java
      operation:
        httpServer:
          route: /internal/metrics
          method: GET
```

> [!NOTE]
>
> Source scope combines in two layers:
>
> * **Different categories must all match.** This rule sets Specific Namespace = `prod` together with Specific Programming Language = `java`, so a workload has to be in `prod` **and** be a Java workload. In YAML that maps to a single `sourceScopes` item with both fields set on it.
> * **Multiple values inside one category are alternatives.** Setting the namespace category to `prod` and `staging` would match a Java workload in either one. In YAML that maps to multiple `sourceScopes` items, each carrying just the namespace field.

## 5. Combine Highly Relevant and Cost Reduction

**Goal:** A `payment-service` where **every error trace** must be kept (Highly Relevant) and **all other traffic** is capped at 1% (Cost Reduction). This is the canonical "keep what matters, sample the rest" pattern, expressed in a single `Sampling` object that bundles both categories together.

**UI**

Create two rules:

**1. Highly Relevant**

* **Name:** `Keep payment-service errors`
* **Notes:** `Errors from payment-service are always investigated; never sample them away.`
* **Rule Type:** Error
* **Source scope:** Specific Source = `odimall / Deployment / payment-service`
* **Operation:** All Operations
* **Keep Percentage:** Keep All

![UI screenshot of a Highly Relevant rule with Rule Type Error scoped to payment-service, Keep at least 100%.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/payment-service-policy-relevant-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=f4352d98a24c24594c9249eb60ceb281)

**2. Cost Reduction**

* **Name:** `payment-service 1% cap`
* **Notes:** `Cap healthy payment-service traffic to keep cost in check; HR errors above always win.`
* **Source scope:** Specific Source = `odimall / Deployment / payment-service`
* **Operation:** All Operations
* **Drop Percentage:** 1%

![UI screenshot of a Cost Reduction rule on payment-service with Keep at most 1%.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/payment-service-policy-cost-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=4948667e517c18b33fe0bf68c0522ad5)

**YAML**

```yaml payment-service-policy.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: payment-service-policy
  namespace: odigos-system
spec:
  name: payment-service policy
  notes: Always keep errors; cap healthy traffic at 1%.
  highlyRelevantOperations:
    - name: Keep payment-service errors
      notes: Errors from payment-service are always investigated; never sample them away.
      error: true
      percentageAtLeast: 100
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: payment-service
  costReductionRules:
    - name: payment-service 1% cap
      notes: Cap healthy payment-service traffic to keep cost in check; HR errors above always win.
      percentageAtMost: 1
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: payment-service
```

> [!NOTE]
>
> A trace from `payment-service` that contains any error span is kept at 100% by the Highly Relevant rule. A trace with no errors falls through to the Cost Reduction cap of 1%. Both rules are evaluated for every matching trace; Highly Relevant's keep percentage wins when both apply. See [Evaluation between Highly Relevant and Cost Reduction](./rules/overview#evaluation-between-highly-relevant-and-cost-reduction-categories).

## 6. Drop noisy gRPC health checks

**Goal:** Drop standard gRPC liveness/readiness checks (`grpc.health.v1.Health.Check`, `Watch`, ...) on the `inventory` Deployment. This is the gRPC counterpart of [Example 1](#1-drop-a-noisy-custom-endpoint), and shows how setting `service` alone scopes a rule to every method of a whole gRPC service.

```yaml drop-grpc-health-checks.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: drop-grpc-health-checks
  namespace: odigos-system
spec:
  name: Drop gRPC health checks
  noisyOperations:
    - name: Drop gRPC health checks
      notes: Standard liveness/readiness pings; not useful for tracing.
      percentageAtMost: 0
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: inventory
      operation:
        grpcServer:
          service: grpc.health.v1.Health
```

> [!TIP]
>
> Because `service` and `method` are independent, a single `service:` line filters every method on a service. Add `method: ListItems` to scope to one specific RPC instead, or combine with `grpcClient.serverAddress` to drop outgoing calls to a particular backend.

## YAML reference notes

A few details that aren't obvious from YAML alone but matter when authoring rules. The full schema—every field, type, and validation rule—is in the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling).

> **Source scope: language enum and workloadKind casing**
>
> The YAML enum for `workloadLanguage` uses lowercase identifiers: `java`, `python`, `go`, `dotnet`, `javascript`, `php`, `ruby`. The UI displays the friendly names (Java, Python, Go, .Net, NodeJS, PHP, Ruby).
>
> `workloadKind` must match the Kubernetes resource kind exactly (case-sensitive PascalCase): `Deployment`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob`. Lowercase or pluralized values will silently fail to match.

> **Defaults when fields are omitted**
>
> Empty or missing fields don't mean "match nothing"—each one has a default that often biases toward broader matching:
>
> * **`sourceScopes` empty or absent** — the rule matches **every** source.
> * **`operation` omitted** — the rule matches **every** operation in the scope.
> * **Noisy `percentageAtMost` omitted** — defaults to **0%** (drops everything).
> * **Highly Relevant `percentageAtLeast` omitted** — defaults to **100%** (keeps everything).
> * **Cost Reduction `percentageAtMost`** — **required**. There is no default; the resource will fail validation if the field is absent.

> **HTTP matchers: route, method, and templated paths**
>
> * On HTTP Server matchers, set **either** `route` (exact match) **or** `routePrefix` (prefix match)—not both. Both match the framework's **templated path** (e.g., `/users/{id}`, `/users/:id`, `/users/*`), not the resolved URL.
> * `method` is a free-form string in YAML. The UI restricts it to GET, POST, PATCH, DELETE, and PUT, but YAML accepts any method (e.g., HEAD, OPTIONS). Leave it empty to match any method.
> * HTTP Client matchers (Noisy only) use `templatedPath` / `templatedPathPrefix`—the framework's parameterized path (e.g., `/users/{id}`), **not** the resolved URL (e.g., `/users/42`). Match against the route as your HTTP library reports it.

> **gRPC matchers: service and method are independent fields**
>
> * `service` matches the fully-qualified gRPC service name (e.g., `acme.inventory.v1.InventoryService`, `grpc.health.v1.Health`); `method` matches the bare method name (e.g., `ListItems`, `Check`). Both fields are **exact match** and **case-sensitive**, mirroring proto naming.
> * The canonical "drop every call to this service" rule is one line: set `service` and leave `method` empty. To target a single RPC, set both. To match every gRPC call regardless of service, leave both empty.
> * Both OpenTelemetry semconv conventions are supported transparently: the older split form (`rpc.service` + bare `rpc.method`) and the newer fully-qualified form (`rpc.method` = `"Service/method"`, with `rpc.service` deprecated). For the latter the matcher splits `rpc.method` on the first `/` to derive the service/method, so the same rule matches spans from agents on either version.
> * The `rpc.*` attribute namespace is shared with other RPC frameworks (Apache Dubbo, Connect RPC, JSON-RPC, .NET WCF, Java RMI, ONC RPC). The matcher reads `rpc.system` / `rpc.system.name` and, when present, only matches spans whose system equals `"grpc"`; non-gRPC RPC spans never accidentally fire a gRPC rule. Spans that omit `rpc.system` entirely are still considered (permissive default for older instrumentations).
> * A span must carry at least `rpc.service` **or** `rpc.method` to be considered a gRPC span; rules on an attribute that can't be derived from the span will not match (e.g. a `service:` rule against a span whose only RPC attribute is the bare-method `rpc.method` or the `_OTHER` sentinel).
> * gRPC Client matchers (Noisy only) additionally support `serverAddress` for matching the remote host (the gRPC channel target). Leave it empty to match any address.

> **Kafka matchers: empty topic matches all topics**
>
> Leaving `kafkaTopic` empty on a `kafkaConsumer` or `kafkaProducer` matcher matches **every** topic for that direction. Set the topic explicitly to scope a Kafka rule to a specific stream.

> **Highly Relevant: combining `error` and `durationAtLeastMs`**
>
> Setting both `error: true` and `durationAtLeastMs` on a single `highlyRelevantOperations` item is an **AND**—the span must be an error **and** longer than the threshold to match. For "errors **or** slow," use two separate items in the `highlyRelevantOperations` array.

> **`name` and `disabled` surface in observability data**
>
> The rule `name` is written as a **metric attribute** and stored as a **span attribute** on participating spans, so use a value that's safe for dashboards and queries—avoid PII or characters your backend doesn't index well.
>
> `disabled: true` on a rule removes it from sampling decisions but **still emits metrics** for it. That's useful for tuning a rule and observing its would-be impact before re-enabling it, rather than deleting it outright.

## Applying YAML rules

You can apply any of the YAML examples above with `kubectl`, or commit them to your GitOps repository alongside the rest of your Odigos configuration:

```bash theme={null}
kubectl apply -f drop-poll-endpoint.yaml
```

Each `Sampling` resource is a self-contained group of rules. You can split rules across multiple `Sampling` objects—by team, environment, or use case—and they will be **joined together** to form the global sampling policy.

```bash theme={null}
kubectl get samplings.odigos.io -n odigos-system
```

### Disabling a rule

To take a rule out of sampling decisions without deleting it, you can either flip the toggle in the UI or set `disabled: true` in YAML. Disabled rules **still emit sampling metrics**, so you can observe what the rule would have done before re-enabling it—useful as a "soft delete" while tuning, rather than removing the rule outright. See [Enabled](./configuration#enabled) for when to reach for this versus deleting the rule.

**UI**

Each rule's detail view has a toggle in the top-right corner that flips its label between **Rule Enabled** and **Rule Disabled**. Slide it off to take the rule out of sampling decisions immediately; slide it back on to re-enable. The toggle is **per rule**—other rules in the same `Sampling` object continue to apply.

![UI screenshot of the Rule Enabled toggle in the on (green) position.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/ui-enable-rule.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=4e5f6ce5034b996a1d932fd2183ded88)

![UI screenshot of the Rule Disabled toggle in the off (red) position.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/ui-disable-rule.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=8e0623bb7fee782f80f0473c69046805)

**YAML**

Set `disabled: true` on an individual entry in `noisyOperations`, `highlyRelevantOperations`, or `costReductionRules`. The other rules in the same `Sampling` object continue to apply. This is what the UI toggle writes.

```yaml disable-a-rule.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: payment-service-policy
  namespace: odigos-system
spec:
  name: payment-service policy
  costReductionRules:
    - name: payment-service 1% cap
      notes: Cap healthy payment-service traffic to keep cost in check; HR errors above always win.
      disabled: true
      percentageAtMost: 1
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: payment-service
```

> [!TIP]
>
> The examples on this page cover the most common shapes; the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling) is the source of truth for every field, type, validation rule, and default on the CRD. Use it when you're extending an example or building a rule that doesn't fit one of the patterns above.


# FAQs
Source: https://docs.odigos.io/enterprise/pipeline/sampling/faqs

Frequently asked questions about sampling in Odigos Enterprise.

Frequently asked questions about sampling in Odigos.

## Concepts and terminology

> **Does 'sampling' mean dropping traces or keeping them?**
>
> Both, depending on who you ask. Some teams describe sampling as **what gets dropped** (e.g. "drop 90%"), others as **what gets kept** (e.g. "keep 10%"). Odigos uses the **keep** framing throughout the UI and docs because it aligns with how rules are written (each rule has a **Keep Percentage**). The two phrasings describe the same outcome; we standardize on "keep" to avoid confusion. See [Introduction to sampling](./introduction#ambiguity-of-terminology).

> **Why is dropping health checks called 'sampling' if I'm dropping 100% of them?**
>
> "Sampling" is the name of the **pipeline mechanism** that makes a per-trace keep/drop decision—not a description of "always keep some, always drop some." Dropping all health probes is just sampling at 0%. The same engine that drops 100% of probe traffic also retains 25% of `/api/products` traffic; both are sampling decisions, just at different keep percentages.

> **Why does Odigos always keep or drop the entire trace?**
>
> Odigos applies a single keep/drop decision per trace, not per span. With **head sampling**, the decision is made at the root span and propagated to every child span (parent-based sampling). With **tail sampling**, the Odigos Gateway buffers all spans for a trace ID and makes one decision after the [aggregation window](./head-and-tail-sampling#aggregation-window). In both cases, splitting a trace—keeping some spans and dropping others—would break parent-child links, end-to-end timing, and any analysis that depends on the full request path, so a kept trace is always complete from root through children. See [Sampling level](./introduction#sampling-level).

## How rules are evaluated

> **If a trace matches rules in more than one category, which category wins?**
>
> Categories are evaluated in a fixed sequence: **Noisy → Highly Relevant → Cost Reduction**. A **Noisy** match short-circuits the rest—the trace is dropped (or sampled down) and the other categories are skipped. A **Highly Relevant** match does **not** short-circuit—Cost Reduction rules are still evaluated, but the Highly Relevant Keep Percentage wins. Cost Reduction caps only take effect on traces with no Highly Relevant match. See [Order of operations between categories](./rules/overview#order-of-operations-between-categories).

> **Within a category, do rules have a priority order?**
>
> No. Within a single category, rules have **no hierarchy**—they're all equal. When a trace matches multiple rules in the same category, the engine satisfies all matches at once: the **highest** Keep Percentage wins for **Highly Relevant** (floors, "keep at least"), and the **lowest** Keep Percentage wins for **Noisy Operation** and **Cost Reduction** (caps, "keep at most"). This is a deliberate design choice: ordered rule lists (like firewall rules) become unmaintainable when you have dozens of rules and have to mentally simulate the order to predict behavior.

> **Why does 'Keep at most 25%' not always mean exactly 25%?**
>
> Cost Reduction percentages are **caps**, not exact rates, because the engine considers **all** matching rules together. If two Cost Reduction rules match the same trace—one capped at 25% and another at 1%—the lower cap (1%) wins. This makes it safe to add new rules without worrying that they'll silently override existing ones. The same logic runs in reverse for Highly Relevant: percentages there are **floors** ("keep at least"), and the highest floor wins.

> **If I only create Highly Relevant rules and no Cost Reduction rules, what gets dropped?**
>
> Almost nothing—and that's usually not what you want. Highly Relevant rules mark important traces, but they don't reduce volume on their own; non-matching traces still flow through the pipeline. Worse, the gateway still buffers traces for tail evaluation, which **adds load without cutting volume**. To control volume, pair Highly Relevant rules with at least one Cost Reduction rule (commonly the built-in Drop Traces Cluster-Wide rule). See [Highly Relevant](./rules/highly-relevant#interaction-with-cost-reduction).

> **Can I combine multiple scope types (e.g. namespace + language) in a single rule?**
>
> Yes. Within a single scope type, multiple selections are combined with **OR** (e.g. namespaces `prod` OR `staging`). Across different scope types, selections are combined with **AND** (e.g. namespace `prod` AND language Java matches **only** Java workloads in `prod`—not every workload in `prod` and not every Java workload across the cluster). See [Source scope](./configuration#source-scope).

## Operational impact

> **Does adding any sampling rule increase load on the gateway?**
>
> It depends on the rule. **Noisy** rules use head sampling and run in the instrumentation agent where supported—they reduce gateway load. **Highly Relevant**, **Cost Reduction** and **Noisy** rules in agents with no head-sampling support, require tail evaluation, which means the gateway buffers each affected trace for the [aggregation window](./head-and-tail-sampling#aggregation-window). Even a single tail-mode rule activates buffering for all traffic, raising memory and CPU on the gateway.

> **Why do gateways sometimes crash instead of just dropping data?**
>
> Once the gateway accepts a trace for tail evaluation, it allocates memory to hold spans until the aggregation window closes. During traffic bursts, many traces can be accepted concurrently before backpressure kicks in—each one keeps allocating memory until the pod hits its limit and OOMs. This is why bursty workloads can produce dropped spans or collector restarts even when your sampling rules look correct. Sizing the gateway for peak burst capacity (not steady-state) is the most reliable fix. See [Gateway load, OOM, and scaling](./considerations#gateway-load-oom-and-scaling).

> **Should I scale up (bigger pods) or scale out (more replicas) for tail sampling?**
>
> Both arithmetically yield similar capacity, but **scaling up is usually preferable for tail sampling**. All spans of a single trace must land on the same gateway pod for aggregation, so each scale event re-shuffles the trace-ID hash routing. Fewer, larger pods scale less often and produce less routing churn. There's no one-size-fits-all answer—validate against your own load.

> **What happens to in-flight traces if a gateway crashes?**
>
> They are lost. Odigos does not currently use persistent queues or full backpressure between the data collector and the gateway, so a crash drops everything the gateway was buffering. **Graceful** shutdowns (rolling updates, scale-downs) flush their buffers before exit; only ungraceful crashes lose data.

> **My sampler looks empty or inactive—why is it still affecting my pipeline?**
>
> The presence of **any** sampling rule that requires tail evaluation activates tail mode for matching traffic—even rules that look minimal, such as a placeholder service name or a broad `/` route. Once tail mode is on, the gateway buffers traces for the aggregation window, which raises memory and latency to export. If you don't need tail behavior, either remove unused or test sampling rules from the cluster, or disable tail sampling explicitly in Odigos configuration (Helm values, or the UI **Settings** page). See [When any sampler enables tail mode](./considerations#when-any-sampler-enables-tail-mode).

## Tail sampling specifics

> **Why is the aggregation window 30 seconds, and can I change it?**
>
> There is no reliable signal for "this trace is finished," so the engine uses a fixed timeout instead. **30 seconds** is the Odigos default, matching the OpenTelemetry community default. You can change it through the Helm value `sampling.tailSampling.traceAggregationWaitDuration` (for example `'45s'` or `'15s'`); the same value applies to every trace in the cluster. Larger windows hold more memory and delay traces appearing in your backend; smaller windows risk evaluating partial traces and fragmenting a single trace across multiple sampling decisions. See [Aggregation window](./head-and-tail-sampling#aggregation-window).

> **Does the 30-second timer start at the first span or the last span of a trace?**
>
> The **first** span the gateway sees for a trace ID, following the OpenTelemetry tail-sampling standard. Practical consequence: requests longer than 30 seconds may be evaluated on a partial trace.

> **What is trace-based load balancing, and why does tail sampling need it?**
>
> Tail evaluation requires every span of a single trace to land on the same gateway pod—otherwise the gateway can't aggregate the trace before deciding. Odigos's data collectors use a **trace-based load balancing exporter** that hashes the trace ID and routes all spans for that trace to the same gateway. Plain round-robin (the default without tail sampling) would scatter spans across pods and break aggregation.

## Built-in rules

> **What exactly does the built-in Kubernetes Health Probes rule cover?**
>
> It drops traces for the **liveness**, **readiness**, and **startup** probe paths discovered on each workload's Kubernetes manifest. The rule's Keep Percentage defaults to **0%** (drop everything) and is configurable. It only matches **HTTPGet** probes—`exec` and `tcpSocket` probes, or probes routed through ingress rewrites or service-mesh wrappers, are not auto-handled and need an explicit Noisy rule. See [Built-in rule: Kubernetes Health Probes](./rules/noisy-operation#built-in-rule-kubernetes-health-probes).

> **Can I disable or change the built-in 'Keep All Error Traces' rule?**
>
> The built-in rule itself is fixed at 100% retention and is not configurable. If you need a different keep percentage for error traces—for example because your error volume is too high to retain all of it—author your own Highly Relevant rule on errors with the Keep Percentage you want. Custom rules are how you express any intent other than the built-in 100%. See [Built-in rule: Keep All Error Traces](./rules/highly-relevant#built-in-rule-keep-all-error-traces).

> **Why isn't there a built-in rule for high-latency traces?**
>
> Latency thresholds depend on the application: 200 ms is fast for one service and unacceptable for another, so there's no sensible default the product can ship. Errors are different—an error span is universally a signal worth keeping—which is why the error rule is built in but a duration rule is not. Define your own Highly Relevant duration rule with the threshold and Keep Percentage that fit your service.

> **Are there built-in operations beyond HTTP and Kafka (e.g. SQS, gRPC, database calls)?**
>
> Today's Operation options are **HTTP Server**, **HTTP Client** (Noisy only), and **Kafka Consumer / Kafka Producer** (Highly Relevant and Cost Reduction). Odigos will add support for additional protocols as users needs demand it.


# Head and tail sampling
Source: https://docs.odigos.io/enterprise/pipeline/sampling/head-and-tail-sampling

How head-based and tail-based trace sampling differ in Odigos and when each approach is appropriate.

## Types of sampling implementation

This page contrasts head sampling (decisions in the **agent** when a trace starts) and tail sampling (decisions at the **Odigos Gateway** after spans are brought together). For how Odigos treats whole traces and root-span decisions, see [Introduction to Sampling](./introduction).

### Head sampling (Agent)

Head sampling is the most basic and efficient form of sampling. The decision is made at the root span before child spans are processed; the trace in its entirety is either kept or dropped to match that decision.

* **Decision point** — At the root span as the trace starts, inside the supported instrumentation agents. The same keep or drop applies to the entire trace (parent-based).
* **Context** — Limited to root-level fields: service name, entry endpoint, and similar. Typical rule inputs are things like service, root span details, and specific routes (e.g. `foo/about`). You cannot base the decision on errors or latency that appear only on child spans, or at span end time (http response codes, etc)
* **Efficiency** — Spans in a dropped trace are skipped at span creation time, so almost no work is done on them: no agent processing, no serialization into eBPF maps, no handling at the Odiglet or data collection stage, and no network or I/O cost. Volume is cut before any downstream pipeline stage sees it.
* **Accuracy** — The choice is early: you do not yet know how the trace will finish, so you cannot guarantee keeping traces based on downstream failures or slowness. Percentage-based cuts behave statistically—rare one-offs may be missed, while frequent behavior still tends to show up in the retained sample over time.
* **Best for** — Noise you can recognize at the start (health checks, scrapes, probes, low-value traffic); percentage-based sampling when Odigos Gateway or destination capacity is tight; cost and volume control in environments where root-only policy is enough.

### Tail sampling (Odigos Gateway)

Tail sampling defers the keep/drop decision until spans for a trace are buffered and aggregated into a single unit the evaluator can read. The decision uses the full span set, not only the root span, so it can reflect the whole request path before anything is exported downstream. Timing, buffering, and delay are covered in [Aggregation window](#aggregation-window).

* **Decision point** — After spans for a trace are buffered and aggregated, in the Odigos Gateway.
* **Context** — Full trace: errors, duration, services and endpoints anywhere in the tree, including off the critical path. Rules can target behavior that only shows up on nested spans.
* **Efficiency** — CPU and memory requirements on the Odigos Gateway are increased compared with head sampling: buffering and per-trace evaluation need more resources. Most volume is still dropped before destinations, which can ease storage and downstream cost.
* **Accuracy** — Full-trace context lets retention match policy on errors, latency, or any span—but evaluation runs only after the [aggregation window](#aggregation-window), so traces show up later in UIs, and very long requests may be judged on a partial trace.
* **Best for** — Debugging errors and slow requests end-to-end; full-trace sampling goals; percentage-style or volume cuts after you have full-path context; policies that need services or routes off the entry span.

#### Aggregation window

The pipeline does not wait for a definitive “end of trace” event, because that would require an indefinite wait. Instead, spans are held up to a bounded window to bound memory and latency. The implementation uses a **default 30-second** interval from the first observed span for a trace ID, consistent with OpenTelemetry-style tail processing, then forwards the trace to the sampler with the spans received by then. Evaluation can therefore run on partial traces for very long requests.

> [!NOTE]
>
> The aggregation window is configurable through the Helm value `sampling.tailSampling.traceAggregationWaitDuration` (for example `'45s'` or `'2m'`). The same value applies to every trace in the cluster. Larger windows hold more memory and delay traces appearing in your backend; smaller windows risk evaluating partial traces and fragmenting a single trace across multiple sampling decisions.

> [!NOTE]
>
> Observability UIs typically show tail-sampled data delayed by the configured aggregation window (about 30 seconds by default). Export, queueing, storage, and indexing add further delay on top of that gap.

## Head vs. tail: quick comparison

| | Head | Tail (Odigos Gateway) |
| - | - | - |
| **Decision point** | Start of trace (root span) | After aggregation |
| **Context** | Root only (service, endpoint, …) | Full trace (errors, latency, any span) |
| **Efficiency** | High; low gateway load for the decision | Higher CPU and memory on the gateway |
| **Accuracy** | Early decision; percentage cuts are statistical | Full-trace policy; aggregation window tradeoffs |
| **Best for** | Noise, cost control, root-level rules | Errors, latency, nested routes, full-path rules |

## Choosing the Right Sampling Strategy: Head vs. Tail

Sampling is primarily a tool for cost and volume management. While both head and tail sampling aim to reduce the amount of data sent to your backend, they differ significantly in when the decision is made and what information is used to make it.

### Head-Based Sampling

Head-based sampling makes a decision at the very beginning of a trace—usually at the root span. This decision is then propagated to all downstream services (often called "parent-based sampling"), ensuring that either the entire trace is kept or the entire trace is dropped.

**When to use it:**

* **High-Volume Environments:** If your system generates an overwhelming number of traces, head sampling is the most efficient way to drop data early, saving CPU and memory on your application agents and reducing network traffic immediately.

* **Simple Logic:** Use it when you only need to sample based on known "head" attributes, such as the service name, the initial endpoint (e.g., /health vs. /api/orders), or a simple percentage (e.g., "keep 5% of all traffic").

* **Resource Constraints on Collectors:** Since decisions happen at the source, your collectors/gateways don't need to buffer or process data that is already marked to be dropped.

**The Trade-off:** \

It is a "blind" decision. Because the decision is made at the start, you cannot know if that specific trace will eventually encounter an error or experience high latency. You risk losing the exact "needle in the haystack" traces you might need for debugging.

> [!NOTE]
>
> OpenTelemetry formalizes this as **representativeness**: a smaller sample can mathematically represent the larger population, so percentage-based head sampling preserves *aggregate* signal even when individual traces are dropped. See [OpenTelemetry: Why sampling?](https://opentelemetry.io/docs/concepts/sampling/#why-sampling).

### Tail-Based Sampling

Tail-based sampling waits until the trace is complete (or a timeout is reached) before deciding whether to keep it. This is handled at the gateway collector level.

**When to use it:**

* **Ensuring Visibility of Errors:** If your goal is to "keep 100% of traces with errors," tail sampling is essential. You can inspect every span in the trace for an error status before making the final call.

* **Latency-Based Sampling:** Use this when you want to capture traces that exceed a specific duration (e.g., "keep any trace longer than 500ms").

* **Complex Business Rules:** When the importance of a trace depends on something that happens deep in the call stack—such as hitting a specific database or a legacy third-party API—tail sampling provides the necessary full context.

**The Trade-off:** \

It is resource-intensive. The gateway must buffer all spans for a trace (often for 30 seconds or more) to ensure it has the "full picture" before deciding. This increases memory usage at the gateway and introduces a delay before traces appear in your UI. Furthermore, it requires "trace-based load balancing" to ensure all spans for a single trace arrive at the same collector.


# Introduction to Sampling
Source: https://docs.odigos.io/enterprise/pipeline/sampling/introduction

Start here for trace sampling in Odigos: why sampling is used, how cost and visibility are balanced, and the core terminology behind the product.

## What is sampling

**Sampling** is how you choose which traces are **kept** in the pipeline and which are **dropped** before they reach backends and UIs. It trims volume so ingest, storage, and search stay manageable, while policies still aim to leave enough signal for debugging and reliability work.

In Odigos, sampling is **per trace**: the entire trace is either retained end to end or not exported, so parent–child structure and timing stay coherent when something is kept. How and when that choice is made—early in the agent at the time of creation (**head** sampling), or at the **Odigos Gateway** after spans are assembled (**tail** sampling)—is covered on [Head and tail sampling](./head-and-tail-sampling).

## Core motivations and definitions

### Primary driver: cost reduction

The main reason for sampling is **cost reduction**. Unsampled traces can flood collectors, backends, and analysis tools with volume that drives up ingest fees, storage, and compute. Sampling also helps prevent the observability stack from being overwhelmed by **redundant or low-value (“uninteresting”) data**, so capacity stays available for signals that support debugging and understanding production behavior.

### Reducing noise from uninteresting traces

Sampling is also commonly used as a **sanitation layer** to drop traces that add little debugging value. Typical examples include:

* **Health-check probes** and readiness/liveness endpoints
* **Metrics scraping** requests (for example, `/metrics` endpoints polled by Prometheus)
* **Infrastructure traffic** such as service-mesh sidecar chatter, internal control-plane calls, and routine background jobs

Removing this noise keeps the observability stack focused on application behavior and prevents redundant data from drowning out meaningful signals.

### Ambiguity of terminology

In many systems, “sampling” is described either as **what is dropped** or **what is kept**. Those views are equivalent mathematically, but they express different policy intent—retention versus reduction. Odigos is oriented around **prioritized signal**: defining and keeping **important** traces so that critical evidence remains available for investigation and problem-solving.

> [!NOTE]
>
> Phrasings such as “drop 90%” and “keep 10%” describe the same outcome. Odigos emphasizes **prioritization** and **clear intent** rather than opaque drop semantics alone.

## Introduction to sampling

### Why use sampling? Cost versus visibility

**Without sampling**

Visibility is maximized—every trace is available for analysis—but that comes with cost, infrastructure load, and noise. At high volume, the stack and ingest budget become the limiting factors.

**With sampling**

Not every trace is stored; the aim is still to preserve enough signal to detect errors, latency regressions, and dependency issues. The balance is deliberate: invest ingest and storage in traces that matter instead of exhaustive duplication of routine, successful traffic.

### The Odigos philosophy

Traditional framing often starts from dropping data to save money. Odigos shifts the emphasis to **prioritizing what matters**: configuration expresses what should be treated as important (errors, slow paths, specific services or attributes, and so on), and the pipeline retains those traces preferentially. Cost control follows from that prioritization rather than from arbitrary bulk deletion alone.

> [!TIP]
>
> When designing sampling policies, **define “important” first** (failures, SLO risks, critical services), then set fallbacks or rates for everything else. That mirrors how Odigos surfaces configuration and avoids tuning only by a global drop percentage.

### Sampling level

Sampling in Odigos is per trace: a single keep-or-drop decision is applied to every span. That decision is made at the root span for head sampling, or at the Odigos Gateway for tail sampling. Partial retention would break parent–child links, timing, and attributes that span the whole request, so a kept trace is always **complete** from the root span through its children.

### Key terminology

These terms describe how traces are classified in the sampling model:

> **Important traces**
>
> Traces that match configured rules for high value—for example, failures, high latency, or explicitly selected attributes. The system is designed to **keep** these preferentially so they remain available for investigation.

> **Regular traces**
>
> Typical traffic that does not match those priority rules. These may be **sampled at a lower rate** (or dropped according to configuration) while still allowing a statistical view of normal behavior where policies allow it.

> **Noisy traces**
>
> Low-value traffic such as **health checks, readiness/liveness probes, metrics scraping, and routine infrastructure calls**. These are typically **dropped on sight** as a sanitation step, since they add volume without aiding investigation.

> **Dropped traces**
>
> Traces that are **not retained** for export to destinations after sampling decisions—whether classified as noisy, deprioritized as regular, or excluded by rate limits. They are not available in the backend for that retention window; this is how volume and cost are reduced.

Together, this vocabulary ties configuration to outcomes: policies define what is important, drop what is noisy, treat everything else as regular unless additional rules apply, and dropped traces represent the tradeoff for sustainable cost and signal-to-noise ratio.


# Cost Reduction
Source: https://docs.odigos.io/enterprise/pipeline/sampling/rules/cost-reduction

Cap how much of the remaining traffic is exported to control trace volume and ingest cost.

**Cost Reduction** rules cap how much traffic is retained for export, after Noisy and Highly Relevant rules have run. They are the **third and final** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories) and are how you control **volume and ingest cost** for traffic that is neither obviously noisy nor specifically marked as important.

## How they're evaluated

* **Applied to traces that fall through earlier categories.** Traces that matched a Noisy drop never reach this stage; traces that matched a Highly Relevant rule are governed by that rule instead (see [Interaction with Highly Relevant](#interaction-with-highly-relevant)).
* **Percentage-based.** Each rule has a **Keep Percentage** that defines the fraction of matching traffic to retain.
* **Scoped.** A Cost Reduction rule can apply to the whole cluster, one or more namespaces, services, or programming languages, and can target all operations or a specific HTTP route or Kafka topic. See [Configuration](../configuration) for the full authoring options.

## Built-in rule: Drop Traces Cluster-Wide

Odigos ships a single built-in Cost Reduction rule, **Drop Traces Cluster-Wide**, which applies one **Keep Percentage** uniformly across every trace in the cluster. You configure the percentage; the engine then makes an independent, probabilistic keep/drop decision per trace.

At high trace volume, the **law of large numbers** applies and sample proportions—error rates, latency percentiles, per-service and per-route traffic shares—converge on the underlying population. The retained traces preserve the **shape** of your traffic, so common scenarios remain visible at their original relative frequencies.

> [!WARNING]
>
> Uniform sampling preserves **statistics**, not individual events. At low keep percentages, or for rare scenarios such as low-frequency error paths or occasional regressions, specific occurrences can be missed entirely. To **guarantee** retention for traces you care about, use a [Highly Relevant](./highly-relevant) rule—which always wins over Cost Reduction—rather than relying on this rule alone.

## Interaction with Highly Relevant

Highly Relevant **always wins** over Cost Reduction on traces that match both, regardless of which keep percentage is higher. That means a Cost Reduction cap of 10% on `/api/orders` does **not** suppress a Highly Relevant rule that keeps 100% of error traces on the same route. See [Evaluation between Highly Relevant and Cost Reduction](./overview#evaluation-between-highly-relevant-and-cost-reduction-categories) for worked examples.

## Typical use cases

* Keep 5% of all cluster traffic as a baseline statistical sample
* Keep 50% of `/api/products` traffic to control a high-volume endpoint
* Keep 1% of traces from a chatty internal service


# Highly Relevant
Source: https://docs.odigos.io/enterprise/pipeline/sampling/rules/highly-relevant

Mark and retain important traces using RED-style signals like errors and high duration, with a per-rule keep percentage.

**Highly Relevant** rules mark traces you want to treat as **important**, aligned with **RED**-style signals (request, **error**, **duration**). They are the **second** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories), evaluated only on traces that did **not** match any Noisy rule.

## How they're evaluated

* **Tail-aware.** Importance often depends on something that only appears mid-trace—an error on a child span, or a long downstream call—so the evaluator typically needs the assembled trace. See [Head vs. tail sampling](../head-and-tail-sampling).
* **One error span marks the entire trace.** When a Highly Relevant error rule applies, a single error span anywhere in the tree is enough to classify the whole trace as important.
* **100% retention by default.** Highly Relevant rules keep every matching trace by default—as the built-in error rule does—which works for most workloads. If a specific operation produces too much volume, lower its Keep Percentage on a custom rule to retain only a fractional sample, while still benefiting from Highly Relevant priority.

## Built-in rule: Keep All Error Traces

Odigos ships a single built-in Highly Relevant rule, **Keep All Error Traces**, which retains **100%** of traces that contain at least one span with **error** status. The built-in rule's keep percentage is fixed at **100%** and is **not configurable**.

Because the evaluator inspects every span in the trace, a single error anywhere in the call tree is enough to keep the whole trace—even when the root span itself succeeds.

> [!NOTE]
>
> If 100% retention of error traces is more than you need—for example because your error volume is high—author your own Highly Relevant rule on errors with the **Keep Percentage** you want. Custom rules are how you express any intent other than the built-in 100%.

## Interaction with Cost Reduction

When a trace matches rules in **both** Highly Relevant and Cost Reduction, the Highly Relevant Keep Percentage **always wins**, whether higher or lower than the Cost Reduction cap. See [Evaluation between Highly Relevant and Cost Reduction](./overview#evaluation-between-highly-relevant-and-cost-reduction-categories) for worked examples.

> [!WARNING]
>
> Highly Relevant rules without any Cost Reduction rules **do not aggressively reduce volume** on their own. Unimportant traffic still flows through the gateway and may be evaluated as part of full-trace policy. Pair Highly Relevant with Cost Reduction when you need both signal and volume control.

## Typical use cases

* Keep all (or most) traces with at least one error span
* Keep a higher percentage of traces over a duration threshold (for example > 500 ms)
* Always keep a sample from a critical service or route


# Noisy Operation
Source: https://docs.odigos.io/enterprise/pipeline/sampling/rules/noisy-operation

Drop or down-sample low-value traffic — health probes, scrapes, heartbeats — early at the root span with head sampling.

**Noisy Operation** rules drop or down-sample traffic that you treat as having little or no value—for example Kubernetes liveness and readiness probes, periodic health checks, or scraper traffic. They are the **first** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories): a trace that matches a Noisy rule is **not** evaluated against Highly Relevant or Cost Reduction rules.

## How they're evaluated

* **Head sampling at the root span.** The keep/drop decision is made on the entry span, before child spans are processed.
* **No trace buffering.** Because the decision happens at the start of the trace, the gateway does not need to wait for an [aggregation window](../head-and-tail-sampling#aggregation-window).
* **Lowest overhead.** Head sampling is the **most efficient** sampling type, with the **least CPU and memory** impact on collectors and gateway.

> [!WARNING]
>
> Head-sampled Noisy rules require an instrumentation agent that supports head sampling. Today this includes **Node.js**, **Python**, **Java**, and **Go**. For other languages or runtimes, Noisy rules are **evaluated at tail-sampling time** instead—the trace is buffered for the full [aggregation window](../head-and-tail-sampling#aggregation-window) before the drop happens, so the efficiency advantages above (no buffering, lowest overhead) **do not apply**.

## Possible outcomes

A matching trace is either:

* **Dropped** — never exported and never evaluated against Highly Relevant or Cost Reduction rules.
* **Sampled down** — kept at a configured percentage. The kept fraction also skips later categories.

> [!NOTE]
>
> Dropped traces do not continue to other categories. If you need a noisy route to remain visible when something goes wrong (for example a failing health check), replace the Noisy rule with a **Highly Relevant** rule on errors and use a **Cost Reduction** rule to cap the successful traffic.

## Built-in rule: Kubernetes Health Probes

Odigos ships a single built-in Noisy rule, **Kubernetes Health Probes**, which drops traces for the **liveness**, **readiness**, and **startup** probe paths discovered on each workload's Kubernetes manifest.

The rule's **Keep Percentage** controls how much probe traffic is retained. Common configurations:

* **0% — drop everything (default).** Probes carry no signal about application behavior, so dropping them keeps the pipeline focused on real traffic and minimizes cost.
* **\~2% — sparse liveness signal.** For services with little or no real traffic, a small permanent value keeps a handful of probe traces flowing. The retained samples act as a "service is up" indicator on dashboards that would otherwise look empty, helping distinguish a *quiet* service from a *silently broken* one.
* **100% — debug probe failures.** Temporarily raise while investigating failing or flaky probes, then reset to your normal value once resolved.

> [!TIP]
>
> Only **HTTPGet** probes declared on the manifest are matched. Probes defined as `exec` or `tcpSocket`, or probes routed through different paths (custom configuration, ingress rewrites, service-mesh wrappers), are not auto-handled; add an explicit Noisy rule for those paths if needed.

## Typical use cases

* Kubernetes `liveness`, `readiness`, and `startup` probes
* Internal `/health` and `/healthz` endpoints
* Prometheus `/metrics` scrape traffic
* Background heartbeats and keep-alive pings
* High-volume, low-value endpoints — for example a static asset route, a polling endpoint, or an autocomplete API that produces many traces without contributing to debugging or SLOs


# Overview
Source: https://docs.odigos.io/enterprise/pipeline/sampling/rules/overview

How Odigos evaluates sampling rules: categories, order between Noisy, Highly Relevant, and Cost Reduction, conflicts inside a category, and head vs. tail sampling.

This page describes how **sampling rules** are evaluated in Odigos: the three **rule categories** (Noisy Operation, Highly Relevant, Cost Reduction), the **order between categories**, and how **conflicts inside a category** resolve. Each category has its own page; this overview focuses on how the categories interact.

## Rule categories

> **[Noisy Operation](./noisy-operation)**
>
> Traffic with little or no value (for example Kubernetes health probes). Decided early at the **root span** with **head sampling**.

> **[Highly Relevant](./highly-relevant)**
>
> **Important** traffic aligned with **RED**-style signals (errors, duration). Each rule's **Keep Percentage** controls how much is retained.

> **[Cost Reduction](./cost-reduction)**
>
> Caps applied to remaining traffic to control volume and cost (for example a percentage across the cluster or a specific route).

> [!NOTE]
>
> See [Head vs. tail sampling](../head-and-tail-sampling) for how head and tail sampling differ in Odigos (root-span decisions versus aggregation at the gateway).

If you only define **Highly Relevant** rules and **no Cost Reduction** rules, unimportant traffic is not aggressively reduced by those rules alone—policy may still require full-trace context to evaluate importance, which increases gateway work without cutting volume. Use **Highly Relevant** and **Cost Reduction** together when you need both signal and volume control.

## Order of operations between categories

Categories are **not** all evaluated for every trace in parallel. Odigos applies them in this fixed **sequence**:

> **Noisy Operation**
>
> If the trace **matches** a Noisy rule, it is **dropped or sampled down** as configured. The trace exits the pipeline at this stage; **Highly Relevant** and **Cost Reduction** are **not** evaluated for that trace.

> **Highly Relevant**
>
> Only traces that did **not** match any Noisy rule reach this stage. Rules here classify **important** traffic (for example errors and duration); **important** matches **can still be sampled** per each rule's **Keep Percentage**. Traces **continue** to **Cost Reduction** after this stage.

> **Cost Reduction**
>
> For traces that **match** rules in this **category** after the earlier stages, this stage applies caps to control volume and cost.

The pipeline runs the categories in this order. At each stage, every rule in that category is considered; how multiple matches resolve is covered in [Rule evaluation](#rule-evaluation).

## Rule evaluation

### Evaluation inside a category

Within a single category, rule order does not matter. When multiple rules match the same trace, the engine combines them so all of them are satisfied at once.

For Highly Relevant, percentages are floors — the highest keep percentage wins, so any retention guarantee is honored. For Noisy Operation and Cost Reduction, percentages are caps — the lowest keep percentage wins, so the strictest cap is honored. This makes it safe to add rules without fear of one silently overriding another.

### Evaluation between Highly Relevant and Cost Reduction categories

When a trace matches rules in **both** the **Highly Relevant** and **Cost Reduction** categories, the **Highly Relevant** keep percentage **always wins** — regardless of whether it is higher or lower than the **Cost Reduction** cap.

> **Highly Relevant rate higher than Cost Reduction**
>
> **Trace:** `POST /api/orders` that returned a `500` error.
>
> **Matching rules:**
>
> * **Highly Relevant** — keep **100%** of error traces.
> * **Cost Reduction** — keep at most **10%** of `/api/orders` traffic.
>
> **Outcome:** the trace is **kept at 100%**. Highly Relevant overrides the Cost Reduction cap.

> **Highly Relevant rate lower than Cost Reduction**
>
> **Trace:** `GET /api/products` that took **750 ms**.
>
> **Matching rules:**
>
> * **Highly Relevant** — keep **25%** of traces longer than **500 ms**.
> * **Cost Reduction** — keep at most **50%** of `/api/products` traffic.
>
> **Outcome:** the trace is **kept at 25%**. Highly Relevant still wins, even though its keep rate is lower than the Cost Reduction cap.

## Why full traces still matter

For traces that are evaluated on the **tail** path, the evaluator needs the assembled trace (within the aggregation window) so it can apply **Highly Relevant** conditions that depend on any span. That is why tail sampling and aggregation are part of the pipeline for those policies, even when the category order is Noisy → Highly Relevant → Cost Reduction.


# Service Graph
Source: https://docs.odigos.io/enterprise/pipeline/service-graph

Visualize how your services connect to each other and to uninstrumented dependencies like Redis or Kafka.

## Overview

The Service Graph is a live topology map that shows which services call each other and how those calls are performing.
It is built entirely from your distributed traces — no additional instrumentation or agents are required.

Odigos generates the service graph automatically when traces are enabled and at least one tracing destination is configured.

## How It Works

Odigos uses the OpenTelemetry [Service Graph Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector) inside the Cluster Gateway Collector. The pipeline looks like this:

```
Instrumented services
       │ (OTLP traces)
       ▼
 Node Collector (DaemonSet)
       │ (OTLP traces)
       ▼
 Cluster Gateway Collector
       ├──► servicegraph connector ──► Odigos UI (topology map)
```

1. The **servicegraph connector** receives every trace that passes through the gateway.
   It matches CLIENT spans with their corresponding SERVER spans to detect service-to-service calls.
2. For each matched pair it emits a metric labelled with the `client` and `server` service names plus any configured extra dimensions, which the Odigos UI uses to render the interactive topology map.

### Virtual Nodes (Uninstrumented Dependencies)

When a service calls an uninstrumented dependency (Redis, Kafka, a third-party API, …) there is no SERVER span.
The connector handles this with **virtual nodes**: it reads certain attributes from the CLIENT span to infer the name of the remote endpoint and creates a synthetic node in the graph.

The attributes checked — in priority order — are configured via `serviceGraphVirtualNodePeerAttributes`.
The connector picks the **first attribute that is present** on the CLIENT span.

> [!NOTE]
>
> Only **span-level attributes** are read when resolving virtual node identities (e.g. `server.address`, `db.system`, `peer.service`).

## Enabling and Disabling

The service graph is **enabled by default** when traces are active.
Disabling it removes the `servicegraph` connector from the gateway pipeline and stops emitting the related metrics.

> [!WARNING]
>
> Disabling the service graph also removes the load balancing behavior between the node collectors and the gateway, because the gateway no longer needs to aggregate spans from the same trace.

```yaml theme={null}
# values.yaml
collectorGateway:
  serviceGraphDisabled: true   # set to true to disable
```

## Configuration Options

All options live under `collectorGateway` in your `values.yaml` (Helm) or by manually editing the `odigos-configuration` ConfigMap in your cluster.

### Extra Dimensions

By default the graph distinguishes services by `service.name` only.
`serviceGraphExtraDimensions` adds more labels to the edge metrics, creating a separate node for each unique combination of values.

**When to use it:** multiple deployments share the same `service.name` (e.g. the same service deployed in `staging` and `production` namespaces).

```yaml theme={null}
collectorGateway:
  serviceGraphExtraDimensions:
    - k8s.namespace.name   # separate nodes per namespace
```

> [!NOTE]
>
> Both span-level and resource-level attributes are supported for extra dimensions — the connector reads from the full span resource context.
> Avoid high-cardinality attributes such as `k8s.pod.name`; each unique value creates a separate time series in Prometheus.

### Virtual Node Peer Attributes

`serviceGraphVirtualNodePeerAttributes` controls which span attribute is used to name uninstrumented dependencies.
The connector evaluates the list in order and uses the first attribute that has a non-empty value.

**When to use it:** multiple instances of the same technology (e.g. a Redis Cluster with several shards, or two separate Redis caches) are collapsing into a single node because they all resolve to the same value (e.g. `db.system=redis`).

```yaml theme={null}
collectorGateway:
  serviceGraphVirtualNodePeerAttributes:
    - peer.service      # explicit logical name — highest priority
    - server.address    # hostname of the specific instance (e.g. redis-cache.default.svc.cluster.local)
    - net.peer.name     # older convention, wide SDK support
    - db.name           # database index (rarely useful for Redis — most apps use index 0)
    - db.system         # technology name (e.g. "redis") — last resort, not instance-specific
```

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Add Sources
Source: https://docs.odigos.io/enterprise/pipeline/sources/create

Add a Source to enable telemetry collection for specific Kubernetes workloads, or a whole namespace.

## With Odigos UI

The easiest way to add a source to Odigos is to use the Odigos UI.\

After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI:

```shell theme={null}
odigos ui
```

And then, in the setup page or the sources page, select the workloads / namespaces for which you want telemetry collection to be enabled.

To include all the workloads in a namespace as sources, and automatically add future-new workloads, enable the `Future Apps` toggle in the UI.
Otherwise, you can enable the `Select All` toggle to include all the workloads in the namespace as sources, excluding future-new workloads.

## With Odigos CLI

See the [CLI documentation on Sources](../../cli/odigos_sources) for detailed command documentation for manipulating Sources.

## Kubernetes Manifests

If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can add a source to Odigos by creating a Source object.

> [!WARNING]
>
> Only one Source object may exist for a given workload, except in namespace instrumentation.

### Workload Source

The following Source object enables instrumentation for a Deployment named `frontend` in the `default` namespace and belong to Datastream `example`.

> **Note:** Sources that do not have a Datastream label will be automatically attached to the default Datastream.

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
  labels:
    odigos.io/data-stream-example: "true"
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
```

### Namespace Source

The following Source object enables instrumentation for all the workloads in the `default` namespace:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: namespace-source
  namespace: default
spec:
  workload:
    name: default
    namespace: default
    kind: Namespace
```

Namespace Sources must have the following fields:\


* `spec.workload.name == spec.workload.namespace`
* `spec.workload.kind: Namespace`

### Disable Workload Source in Namespace

You can have a namespace Source and a workload Source that both enable instrumentation for a workload. In this case, if one is deleted, the other will persist instrumentation for the workload.

To exclude specific workloads from namespace instrumentation, set `disableInstrumentation: true` on the workload Source. For example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: namespace-source
  namespace: default
spec:
  workload:
    name: default
    namespace: default
    kind: Namespace
---
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
spec:
  disableInstrumentation: true
  workload:
    name: frontend
    namespace: default
    kind: Deployment
```

In this example, all workloads in the `default` namespace will be instrumented except for the `frontend` Deployment. If `example-source` is deleted (or `disableInstrumentation` is set to `false`), the `frontend` Deployment will be instrumented.


# Remove Sources
Source: https://docs.odigos.io/enterprise/pipeline/sources/delete

Remove a Source to disable telemetry collection for specific Kubernetes workloads, or a whole namespace.

> [!NOTE]
>
> Sources are persistent and decoupled from the workloads they instrument.\\
>
> If a workload is deleted without deleting the Source, the Source will resume its effects if the workload is re-created later.

## With Odigos CLI

The Odigos CLI supports instrumenting and uninstrumenting workloads with the [`odigos sources enable`](../../cli/odigos_sources_enable)
and [`odigos sources disable`](../../cli/odigos_sources_disable) commands.

Each of these commands accepts a source type, name, and namespace.

For example, to remove a source from odigos, run:

```shell theme={null}
odigos sources disable deployment <deployment-name> -n <deployment-namespace>
```

This command will rollout existing pods to restart without any odigos additions.

It can be undone (re-instrumented) by running: "

```shell theme={null}
odigos sources enable deployment <deployment-name> -n <deployment-namespace>
```

## With Odigos UI

The easiest way to remove a source from Odigos is to use the Odigos UI.\

After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI:

```shell theme={null}
odigos ui
```

And then, in the sources page, unselect the workloads / namespace for which you want telemetry collection to be disabled.

## Kubernetes Manifests

If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can remove a source from Odigos by deleting the Source object, for example:

**Step 1**

Run:

```shell theme={null}
kubectl get sources -n default
```

Output:

```shell theme={null}
NAME           WORKLOAD     KIND         NAMESPACE
source-7sxsb   frontend     Deployment   default
source-b42rn   inventory    Deployment   default
source-b7zn9   coupon       Deployment   default
source-v6sxs   membership   Deployment   default
source-v9jjw   pricing      Deployment   default
```

**Step 2**

Run:

```shell theme={null}
kubectl delete source source-7sxsb -n default
```

Output:

```shell theme={null}
source.odigos.io "source-7sxsb" deleted
```


# Introduction
Source: https://docs.odigos.io/enterprise/pipeline/sources/introduction

Sources are a Custom Resource that define which Kubernetes workloads should be auto-instrumented by Odigos.

## How Sources work

Source custom resource objects are created as a reference to a workload or namespace ("entities").

Odigos reacts to changes in these custom resource objects to configure instrumentation for the referenced entity.

On this page, "Sources" refers to the custom resource object, while "workload", "namespace", or "entity" refers to the application
being instrumented.

See the [API Reference](../../api-reference/odigos.io.v1alpha1#source) for the full structure of `Source` objects and their fields.

## Supported Entities

Odigos supports the following Kubernetes entities:

* Kubernetes **Deployment**
* Kubernetes **DaemonSet**
* Kubernetes **StatefulSet**
* Kubernetes **Namespace**

## Types of Sources

A Source object can be either:

* A **Workload Source**, which controls instrumentation for a single workload (Deployment, DaemonSet, or StatefulSet)
* A **Namespace Source**, which controls instrumentation for every workload in a namespace.

### Workload Sources

Workload Sources specify a single workload, and the Source represents the instrumentation configuration for that workload.

Use Workload Sources to control instrumentation and configuration for individual Deployments, StatefulSets, or DaemonSets
with unique requirements.

Example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-workload
  namespace: default
spec:
  workload:
    kind: DaemonSet
    name: sample-app
    namespace: default
```

### Namespace Sources

Namespace Sources represent instrumentation settings for an entire namespace. Namespace Sources are created by setting
`spec.workload.kind: Namespace` along with `spec.workload.name` and `spec.workload.namespace` equal to the name of the
namespace to control.

Use Namespace Sources to control instrumentation for multiple Deployments, StatefulSets, and DaemonSets from a
single spot.

Example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-namespace
  namespace: default
spec:
  workload:
    kind: Namespace
    name: default
    namespace: default
```

## Using multiple Sources

Workload Sources are intended to be combined with Namespace Sources for flexible, fine-grained control. For example,
you may choose to create a Namespace Source to enable instrumentation for namespace `foo`, but want to exclude deployment
`bar`. In this case, you can create a Workload Source for deployment `bar` with `disableInstrumentation: true`:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-namespace
  namespace: foo
spec:
  workload:
    kind: Namespace
    name: foo
    namespace: foo
---
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: exclude-bar
  namespace: foo
spec:
  disableInstrumentation: true
  workload:
    kind: Deployment
    name: bar
    namespace: foo
```

The opposite is also possible: you could disable instrumentation for namespace `foo` with a Namespace Source but
choose to explicitly enable deployment `bar` via a Workload Source that sets `disableInstrumentation: false` (the default).

This is because Workload Sources take priority over Namespace Sources, and any settings in a Workload
Source will override those in a parent Namespace Source.

## Source Workflows

Sources are intended to control and configure instrumentation as objects that are completely decoupled from their
target workloads. When working with Sources, they should be treated as such.

The only requirements for a Source to identify a workload are:

* The Source must set `spec.workload.name`, `spec.workload.namespace`, and `spec.workload.kind` to match the workload (or namespace).
* The Source must exist in the same namespace as the target workload (or namespace).

Besides these requirements, you are encouraged to manage your Sources as their own objects.

For example, when creating Sources from YAML manifests, consider storing them together in their own file
such as `sources.yaml`, separate from your workload manifests. This separation of concerns allows you
to declaratively control your application's observability without tying it to your application manifests.

You should also define RBAC policies that restrict access to only what is needed to work with Source objects.
In other words, your observability team may not require the ability to view deployments in a namespace to manage
the instrumentation for those deployments.

For example, the following Role grants full editor permissions for Sources:

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: source-editor
rules:
- apiGroups: ["odigos.io"]
  resources: ["sources"]
  verbs: ["*"]
```

## Source Labels

Odigos uses mutating webhooks to automatically copy the values of `spec.workload` to matching labels. It is
encouraged to use these labels to manage Sources in a cluster:

* `odigos.io/workload-name`
* `odigos.io/workload-namespace`
* `odigos.io/workload-kind`

The value of these labels will be validated to always match the corresponding fields in `spec.workload`. They can be used to group and list Sources, for example:

```shell theme={null}
kubectl get sources -l odigos.io/workload-kind=Deployment
```

This will list all `Sources` for `Deployments`.

## Automatically created Sources

In some scenarios, Odigos may create Sources for you automatically.

* When instrumenting workloads in the UI, Odigos will create a Workload Source for each workload.
* When choosing **Include future workloads** in the UI, Odigos will create a Namespace Source for the namespace.
* When creating a Source grouping in the UI, Odigos will create a Workload Source for each Workload in the group.

In all of these cases, if a Source already exists, Odigos will either patch the existing Source or
return an error if it can't.


# Troubleshooting
Source: https://docs.odigos.io/enterprise/pipeline/sources/troubleshooting

If you selected a workload to auto-instrument and you don't see any data, use the checklist below to troubleshoot the issue.

> [!TIP]
>
> It is recommended to use the [`odigos describe`](../../cli/odigos_describe) command to get detailed information in one command.

## Check is Source Created?

A `Source` object is required for Odigos to instrument the workload or namespace, such as:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example
  namespace: foo
spec:
  workload:
    name: myapp
    namespace: foo
    kind: StatefulSet
```

If you expect to see data from a workload but don't, check if the `Source` object exists:

```shell get all in namespace theme={null}
kubectl get sources -n <namespace>
```

```shell get by deployment name theme={null}
kubectl get sources -n <namespace> -l odigos.io/workload-name=<deployment-name>
```

```shell get by deployment kind theme={null}
kubectl get sources -n <namespace> -l odigos.io/workload-kind=<deployment-kind>
```

## Check is Language Detected?

After a `Source` object is created, Odigos will attempt to detect the programming language of the containers in the workload pods, and write the detected language to an `InstrumentationConfig` object in the same namespace.

The resulting language is used to determine which instrumentation to apply.

If a `Source` object exists, verify that the `InstrumentationConfig` also exists:

```shell get all in namespace theme={null}
kubectl get instrumentationconfigs -n <namespace>
```

```shell get by crd name theme={null}
kubectl get instrumentationconfigs -n <namespace> <crd-name> -o yaml
```

If an `InstrumentationConfig` object exists, verify that the `status.runtimeDetailsByContainer` array contains the expected languages, for example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationConfig
metadata:
  name: example
  namespace: foo
spec:
  serviceName: example
status:
  runtimeDetailsByContainer:
  - containerName: example
    language: go
    runtimeVersion: 1.21.2
```

## Check is Device Instrumented?

Once an `InstrumentationConfig` object is created, and the supported language is detected, Odigos will wait until a [destination](../../backends-overview) is added, and OpenTelemetry Collectors are deployed to the cluster (which will receive the telemetry data).

To inject the instrumentation agents into the pods, Odigos will patch the workload template containers with a resource based on the detected language.

If an `InstrumentationConfig` object exists, and the languages are detected, verify that the `Deployment` object contains the expected instrumentation devices:

```shell get all in namespace theme={null}
kubectl get deployments -n <namespace>
```

```shell get by deployment name theme={null}
kubectl get deployments -n <namespace> <deployment-name> -o yaml
```

It should look something like this:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: foo
spec:
  template:
    spec:
      containers:
      - resources:
          limits:
            instrumentation.odigos.io/go-ebpf-community: "1"
```

The `instrumentation.odigos.io/go-ebpf-community` resource-limit instructs Odigos on how to mount the relevant OpenTelemetry agents and environment variables into the container, and which processes should be instrumented with eBPF.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Configuration
Source: https://docs.odigos.io/enterprise/pipeline/sources/update



Beyond just enabling or disabling instrumentation, Source objects can be used to configure instrumentation settings as well.

All Source settings can be configured with the Odigos CLI (see [CLI documentation on updating Sources](../../cli/odigos_sources_update)) or Kubernetes-native tools like `kubectl`.

## service.name

The `service.name` attribute is a fundamental resource attribute in [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/resource/#service).\

It is required, and is reported in all telemetry data.\

It is used to identify the service that emitted a span, metric or log record.

Odigos will automatically populate this attribute based on the name of the workload. If there are multiple containers in the workload, the `service.name` will be set to the container name.

If you want to override the default value, you can also set the `service.name` attribute by adding the `otelServiceName` attribute to an exiting Source, or by creating a `Source` with that attribute:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
  otelServiceName: my-custom-service-name
```

Namespace Sources do not support the `otelServiceName` attribute. If a namespace is instrumented, and a Source for a specific workload is not present, a specific source for that workload will need to be created including the desired `otelServiceName`.

### From Odigos UI

The easiest way to update a source from Odigos is to use the Odigos UI.\

After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI:

```shell theme={null}
odigos ui
```

And then, in the sources page, select the source you want to edit and fill in the value for the `service.name` field.
To undo the override, simply erase the value from the `service.name` field.

### Odigos CLI

To set the `service.name` using the Odigos CLI, use the `odigos sources update` command ([see documentation](../../cli/odigos_sources_update)).

If you know the name of the Source object to update, pass that to `odigos sources update` along with the `--otel-service` flag:

```shell theme={null}
odigos sources update <mysource> --otel-service=my-service-name -n <namespace>
```

If you don't know the name of the Source object to update, you can provide flags to `odigos sources update` to identify the Source by workload:

```shell theme={null}
odigos sources update --workload-kind=<Deployment/StatefulSet/DaemonSet> \
--workload-name=<my-workload> \
--workload-namespace=<namespace> \
--otel-service=my-service-name
```

### Kubernetes Manifest

If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can can set the `service.name` attribute by updating the corresponding Source.

For example, to set the `service.name` attribute in telemetry for an existing source:

To get the relevant Source for your workload you can either use:

```shell theme={null}
kubectl get sources -n <source namespace> \
-l odigos.io/workload-kind=<Deployment/DaemonSet/StatefulSet/Namespace> \
-l odigos.io/workload-name=<Workload name> \
-l odigos.io/workload-namespace=<source namespace>
```

or list the sources in the relevant namespace:

```shell theme={null}
kubectl get sources -n <namespace>
```

After we have the desired Source we can patch it:

```shell theme={null}
kubectl patch source/<source-name> -n <namespace> --type=merge -p '{"spec":{"otelServiceName":"<serviceName>"}}'
```

If you don't know the name of a Source object (such as when a Source
is automatically created by instrumentation done via the UI), you can
combine the use of Source labels with the `kubectl patch` command. For
example:

```shell theme={null}
kubectl patch source \
$(kubectl get source -l 'odigos.io/workload-name=<workloadName>,odigos.io/workload-kind=<Deployment/StatefulSet/DaemonSet>' --no-headers -o custom-columns=":metadata.name") \
--type=merge \
-p '{"spec":{"otelServiceName":"<serviceName>"}}'
```


# Building a pipeline
Source: https://docs.odigos.io/enterprise/quickstart/building-a-pipeline



## Choosing target applications

You should now see the following page:

![Select target applications](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_choose_apps.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=f6fd9fc8e033b3a0507112d0e2490708)

Select all the applications in the `default` namespace and click `Next`.

> [!NOTE]
>
> Selecting **Apply for future apps** will automatically generate distributed traces for any application that will be deployed on that namespace in the future.

## Adding Jaeger as a destination

In the next step, you will be asked to select a destination for your traces.
Scroll down to the self hosted destinations and click on **Jaeger**.

![Select Jaeger](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_choose_dest.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=4cd44e8b085d4b669293f075b593d0d0)

#### Connection details

You will now be asked to provide the connection details for your Jaeger instance:

* **Destination name**: A name for your destination
* **Host**: The hostname of your Jaeger instance.

Enter any name you want for the destination, and enter `jaeger.tracing:4317` as the host.

![Jaeger connection details](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_jaeger_connection.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=2a0e1189e01181665972c87f2b1c6a1d)

**That's it!** You can now click `Done` and finish the wizard.

Odigos will now instrument your selected applications and deploy the nessesary OpenTelemetry collectors.
The following page will show an overview of your observability pipeline:

![Observability pipeline](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_overview.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=c1db71afde509a147b602457ef55a23c)


# Generating Data
Source: https://docs.odigos.io/enterprise/quickstart/generation-data



## Wait for collectors to be ready

Before generating data, make sure that the collectors and applications are ready. To check this run the following command:

```bash theme={null}
kubectl get pods -n default
```

```bash theme={null}
kubectl get pods -n odigos-system
```

![Wait for collectors to be ready](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/collectors_ready.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=4724ddc8c75204c4640bd3652d433256)

## Access frontend application and generate data

In order to access the frontend application, run the following command:

```bash theme={null}
kubectl port-forward svc/frontend 8080:8080
```

Then, open the following URL in your browser: [http://localhost:8080](http://localhost:8080) and purchase some items.

![Frontend application](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/demo_frontend_app.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=70e9781773d14615c3c929ac108b2200)


# Installation
Source: https://docs.odigos.io/enterprise/quickstart/installation



## Installing Odigos CLI

Odigos CLI is the easiest way to install Odigos on your Kubernetes cluster. It
is a single binary that can be installed on any Linux, MacOS or Windows machine.

**Homebrew**

If you are on MacOS, you can install Odigos CLI using Homebrew:

```bash theme={null}
brew install odigos-io/odigos-cli/odigos
```

**GitHub Releases**

1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux).

2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system.

3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH".

* For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt.

* For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`.

4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH.

**Windows**

1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows).

2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files.

3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how:

* **Option 1**:
  Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder.

* **Option 2**:
  Add the folder containing the extracted binary to your PATH manually:
  1. Right-click the **Start** button and select **System**.
  2. Click **Advanced system settings** on the left.
  3. In the **System Properties** window, click **Environment Variables** at the bottom.
  4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**.
  5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file.
  6. Click **OK** to close all windows.

4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located.

## Installing Odigos on Kubernetes

To install Odigos on your Kubernetes cluster, run the following command:

```bash theme={null}
odigos install
```

## Accessing Odigos UI

After Odigos is installed, open the Odigos UI by running the following
command:

```bash theme={null}
odigos ui
```

By default, Odigos UI is available at [http://localhost:3000](http://localhost:3000).


# Introduction
Source: https://docs.odigos.io/enterprise/quickstart/introduction



In this tutorial we are going to use Odigos for generating distributed traces for a microservices-based application written in Go, Java, Python, .NET and Node.js.

We are going to deploy the application in a Kubernetes cluster and use [Jaeger](https://www.jaegertracing.io/) as the backend for storing and visualizing the traces.

> [!NOTE]
>
> This guide is designed for:
>
> * IT professionals with experience in system administration or DevOps
> * Software engineers familiar with containerization and Kubernetes concepts
> * Anyone comfortable with command-line interfaces and basic cloud infrastructure
>
> If you're new to these concepts or prefer a more detailed, step-by-step approach, we recommend starting with our [Beginner's Guide to Odigos Installation](../quickstart/installation-guide-for-beginners).

## Preparing the environment

### Creating a Kubernetes cluster

This first step is to create a Kubernetes cluster.
We recommend using [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/docs/) for trying Odigos out in a local environment.

> [!WARNING]
>
> **Mac users**: please avoid using Docker Desktop built-in Kubernetes cluster as it [does not](https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation) support bind propagation.

Create a new local Kubernetes cluster, by running the following command:

**Kind**

```bash theme={null}
kind create cluster
```

**Minikube**

```bash theme={null}
minikube start
```

### Deploying the target application

We are going to install a [demo application](https://github.com/odigos-io/simple-demo) that consists of 6 microservices written in Java, Go, Python, Node.js, .NET and PHP.

![Architecture of demo application](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/demo_architecture.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=079eb8601b77c8fdc5e777a34f686f25)

Deploy the application using the following command:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/odigos-io/simple-demo/main/kubernetes/deployment.yaml
```

### Deploying Jaeger

Jaeger is a popular open source distributed tracing backend. We are going to use it for storing and visualizing the traces generated by Odigos.

Install Jaeger using the following command:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/odigos-io/simple-demo/main/kubernetes/jaeger.yaml
```

### Waiting for the applications to start

Before moving to the next step, make sure that the application pods and Jaeger pods are running, this may take a few minutes.

```bash theme={null}
kubectl wait --for=condition=available --timeout=300s deployment --all --all-namespaces
```

![Deployments Ready](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/pods_ready.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=a417114517cace40ac4344592702d2b3)


# Next Steps
Source: https://docs.odigos.io/enterprise/quickstart/next-steps



You just finished the Odigos quickstart guide! 🎉

## Not just distributed tracing

**Odigos supports metrics and logs too!** You can send metrics and logs from your
application to Odigos, and then visualize them in your favorite backend.

In order to send metrics and logs, just select a destination that supports them.

> **[Many more observability backends!](../backends-overview)**
>
> Odigos supports all the major observability backends, both open-source and commercial.
> Select the relevant backend for your use case below to connect it to Odigos.


# Viewing traces
Source: https://docs.odigos.io/enterprise/quickstart/viewing-traces



Now that we have some traces, let's view them!
Access the Jaeger UI by running the following command:

```bash theme={null}
kubectl port-forward -n tracing svc/jaeger 16686:16686
```

Then, open [http://localhost:16686](http://localhost:16686) in your browser and explore the traces.

![Jaeger UI](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/jaeger_ui.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=623e9ab3fa2cf69effa287c1cdad19da)


# Install Odigos Images from a Custom Docker Registry
Source: https://docs.odigos.io/enterprise/setup/docker-registry



## Tagging and Pushing Images

### Step 1: Set Environment Variables

First, export the `VERSION`, `CUSTOM_DOCKER_REGISTRY`, `PLATFORM`, and `ODIGOS_TOKEN` environment variables in your shell:

```bash theme={null}
export VERSION=<YOUR-DESIRED-VERSION>
export CUSTOM_DOCKER_REGISTRY=<YOUR-CUSTOM-DOCKER-REGISTRY>
export PLATFORM=<YOUR-PLATFORM-ARCHITECTURE>
export ODIGOS_TOKEN=<YOUR-ONPREM-TOKEN>
```

Replace `<YOUR-DESIRED-VERSION>` with the version of the Odigos images you want to use, `<YOUR-CUSTOM-DOCKER-REGISTRY>` with the URL of your custom Docker registry, `<YOUR-PLATFORM-ARCHITECTURE>` with the platform architecture of the environment where you are deploying Odigos (linux/amd64, linux/arm64), and `<YOUR-ONPREM-TOKEN>` with your Odigos Enterprise on-prem token.

***

### Step 2: Authenticate to the Enterprise Registry

Enterprise images are hosted on the private registry `registry.odigos.io`. Authenticate with your on-prem token before pulling. The username is always `odigos`; the password is your on-prem token.

Choose the method that matches how you will pull images:

**Docker**

Log in with Docker so you can `docker pull` images locally or in CI:

```bash theme={null}
echo "$ODIGOS_TOKEN" | docker login registry.odigos.io -u odigos --password-stdin
```

**Podman**

Log in with Podman so you can `podman pull` images locally or in CI:

```bash theme={null}
echo "$ODIGOS_TOKEN" | podman login registry.odigos.io -u odigos --password-stdin
```

**Local auth file**

Manually generate a Docker config JSON pull secret for local Docker or Podman. Use this when `login` is not available, or you need an auth file for CI.

```bash theme={null}
export ODIGOS_DOCKER_AUTH=$(echo -n "odigos:$ODIGOS_TOKEN" | base64)
cat <<EOF > odigos-dockerconfig.json
{
  "auths": {
    "registry.odigos.io": {
      "username": "odigos",
      "password": "$ODIGOS_TOKEN",
      "auth": "$ODIGOS_DOCKER_AUTH"
    }
  }
}
EOF
```

For Docker, write the file to the default config path (this overwrites `~/.docker/config.json` if it already exists):

```bash theme={null}
mkdir -p ~/.docker
cp odigos-dockerconfig.json ~/.docker/config.json
```

To keep your existing Docker config, use a dedicated directory instead:

```bash theme={null}
mkdir -p /tmp/odigos-docker
cp odigos-dockerconfig.json /tmp/odigos-docker/config.json
export DOCKER_CONFIG=/tmp/odigos-docker
```

For Podman, write the file to the default auth path:

```bash theme={null}
mkdir -p ~/.config/containers
cp odigos-dockerconfig.json ~/.config/containers/auth.json
```

Or pass the file per command:

```bash theme={null}
```

**Kubernetes pull secret**

Create a Docker registry pull secret when an in-cluster Job, GitOps tool, or other Kubernetes process needs to pull from `registry.odigos.io`.

```bash theme={null}
kubectl create secret docker-registry odigos-enterprise-registry \
  --namespace odigos-system \
  --docker-server=registry.odigos.io \
  --docker-username=odigos \
  --docker-password="$ODIGOS_TOKEN"
kubectl label secret odigos-enterprise-registry -n odigos-system odigos.io/system-object=true
```

You can use a different secret name if this credential is only for your mirroring pipeline. Use the name `odigos-enterprise-registry` if Odigos itself will pull from `registry.odigos.io`.

Or apply the secret declaratively. To do so, first generate the `auth` value with:

```bash theme={null}
echo -n "odigos:$ODIGOS_TOKEN" | base64
```

Then apply:

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: odigos-enterprise-registry
  namespace: odigos-system
  labels:
    odigos.io/system-object: "true"
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "registry.odigos.io": {
          "username": "odigos",
          "password": "<TOKEN>",
          "auth": "<BASE64_OF_odigos:TOKEN>"
        }
      }
    }
```

> [!NOTE]
>
> The pull, tag, and push examples below use Docker. If you authenticated with Podman, replace `docker` with `podman`.

### Step 3: Pull the Images

Start by pulling the required images from the enterprise registry (`registry.odigos.io`) onto your local machine or CI environment:

```bash theme={null}
docker pull registry.odigos.io/odigos-enterprise-instrumentor:$VERSION
docker pull registry.odigos.io/odigos-ui:$VERSION
docker pull registry.odigos.io/odigos-enterprise-odiglet:$VERSION
docker pull registry.odigos.io/odigos-enterprise-collector:$VERSION
docker pull registry.odigos.io/odigos-victoria-metrics:$VERSION
docker pull registry.odigos.io/odigos-cache:$VERSION
```

If you are using the [k8s-init-container](/enterprise/instrumentations/configuration/mount-method#3-InitContainer) mount method
make sure the odigos-agents image is also pulled, as it's required by the injected InitContainer.
On Kubernetes 1.35+, that init container also needs the registry pull secret in each instrumented namespace. See [Init container image pull secrets](/enterprise/instrumentations/configuration/mount-method#init-container-image-pull-secrets).
**This is not the default configuration**, if you're unsure what this means, you can safely ignore this step and continue with the standard setup.

```bash theme={null}
docker pull registry.odigos.io/odigos-enterprise-agents:$VERSION
```

> [!WARNING]
>
> Odigos component images are published in multi-arch for ARM64 and AMD64 architectures. When pulling these images, be sure to specify the correct architecture *for the environment they will be deployed*.

```bash theme={null}
docker pull --platform $PLATFORM registry.odigos.io/odigos-enterprise-instrumentor:$VERSION
docker pull --platform $PLATFORM registry.odigos.io/odigos-ui:$VERSION
docker pull --platform $PLATFORM registry.odigos.io/odigos-enterprise-odiglet:$VERSION
docker pull --platform $PLATFORM registry.odigos.io/odigos-enterprise-collector:$VERSION
docker pull --platform $PLATFORM registry.odigos.io/odigos-victoria-metrics:$VERSION
docker pull --platform $PLATFORM registry.odigos.io/odigos-cache:$VERSION
```

### Step 4: Tag the Images

Next, Tag each image with your custom Docker registry prefix:

```bash theme={null}
docker tag registry.odigos.io/odigos-enterprise-instrumentor:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-instrumentor:$VERSION
docker tag registry.odigos.io/odigos-ui:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION
docker tag registry.odigos.io/odigos-enterprise-odiglet:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-odiglet:$VERSION
docker tag registry.odigos.io/odigos-enterprise-collector:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-collector:$VERSION
docker tag registry.odigos.io/odigos-victoria-metrics:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION
docker tag registry.odigos.io/odigos-cache:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION
```

> **Note:** Enterprise images are hosted at `registry.odigos.io`. Image names are `odigos-<component>` or `odigos-enterprise-<component>` without an embedded `keyval/` path segment. If you previously mirrored images under an older layout, re-tag them to match this naming when pushing to your custom registry.

> **Note:** Odigos Enterprise now runs a dedicated collector image, `odigos-enterprise-collector`, in place of `odigos-collector`. If you were already mirroring `odigos-collector`, mirror `odigos-enterprise-collector` as well before upgrading, otherwise the gateway and the node collectors will fail to pull.

### Step 5: Push the Images

Now, push the tagged images to your custom Docker registry:

```bash theme={null}
docker push $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-instrumentor:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-odiglet:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-enterprise-collector:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION
```

### Step 6: Configure Access for Private Registries

If your Docker registry is private, configure your Kubernetes cluster to pull images from it. Refer to the official Kubernetes documentation for guidance: [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).

Pass the secret name as `imagePullSecrets` at install time so Odigos components can pull mirrored images. For `k8s-init-container` on Kubernetes 1.35+, also copy that secret into each instrumented namespace. See [Init container image pull secrets](/enterprise/instrumentations/configuration/mount-method#init-container-image-pull-secrets).

> **Note: To ensure your Odigos deployments and daemonsets can pull images from a private registry, patch them to add the image pull secret.**
>
> **Use the following commands:**
>
> ```
> export ODIGOS_NAMESPACE=<NAMESPACE-WHERE-ODIGOS-IS-INSTALLED>
> export IMAGE_PULL_SECRET=<NAME-OF-IMAGE-PULL-SECRET>
> kubectl patch deployment odigos-ui -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-instrumentor -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-gateway -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-victoriametrics -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-cache -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch daemonset odiglet -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch daemonset odigos-data-collection -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> ```

### Step 7: Install Odigos Using Custom Images

Finally, install Odigos using the images from your custom Docker registry:

**Odigos CLI**

```bash theme={null}
odigos install --image-prefix=$CUSTOM_DOCKER_REGISTRY --onprem-token $ODIGOS_TOKEN
```

**Helm Chart**

```bash theme={null}
helm repo update
helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace --set imagePrefix=$CUSTOM_DOCKER_REGISTRY --set onPremToken=$ODIGOS_TOKEN
kubectl label namespace odigos-system odigos.io/system-object="true"
```

> [!TIP]
>
> These steps cover the **Odigos K8s agent** chart (`odigos`). For Odigos Central (`odigos-central`), including the bundled Keycloak image, see [Custom Docker Registry](/central/installation#custom-docker-registry).


# Installation
Source: https://docs.odigos.io/enterprise/setup/installation



> [!TIP]
>
> Installation issues? Need help? Message us on [Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)

Checkout the [installation options](/enterprise/setup/installation-options) to customize your installation.

There are 3 ways to install odigos in your kubernetes cluster:

* [Odigos CLI](#odigos-cli)
* [Helm Chart](#helm-chart)
* [OpenShift Operator](#openshift-operator)

All install methods will deploy and configure resources in the active Kubernetes cluster as per the current context.
Before proceeding with the installation, ensure that you are targeting the correct cluster with `kubectl config current-context`.

> [!NOTE]
>
> Odigos enterprise data collection is done using eBPF, so Odigos requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.

## Odigos CLI

### Install the CLI

**Homebrew**

If you are on MacOS, you can install Odigos CLI using Homebrew:

```bash theme={null}
brew install odigos-io/odigos-cli/odigos
```

**GitHub Releases**

1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You'll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux).

2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you'll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system.

3. Move the File to a Folder in Your `PATH`. You'll need to move this file to a folder where your computer looks for programs, known as your "PATH".

* For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system's PATH. You can find out where by typing echo `%PATH%` in Command Prompt.

* For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`.

4. Verify It's Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you've successfully installed it! If not, double-check that you moved the file to a folder included in your PATH.

**Windows**

1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows).

2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files.

3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you'll need to move the program (binary) into one of these folders. Here's how:

   * **Option 1**:
     Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder.

   * **Option 2**:
     Add the folder containing the extracted binary to your PATH manually:
     1. Right-click the **Start** button and select **System**.
     2. Click **Advanced system settings** on the left.
     3. In the **System Properties** window, click **Environment Variables** at the bottom.
     4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**.
     5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file.
     6. Click **OK** to close all windows.

4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you've successfully installed it. If it doesn't, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located.

### Install Odigos

The enterprise version of Odigos requires a license token. Please contact the Odigos team to inquire about access to the Enterprise version.

```shell theme={null}
odigos install --onprem-token $ODIGOS_TOKEN
```

This will install odigos in the `odigos-system` namespace in your kubernetes cluster.

At this point, odigos is installed in your cluster. You can now add sources and destinations to instrument your cluster with OpenTelemetry.

## Helm Chart

### Add Helm Repository

```shell theme={null}
helm repo add odigos https://odigos-io.github.io/odigos/
```

### Install Odigos

The enterprise version of Odigos requires a license token. Please contact the Odigos team to inquire about access to the Enterprise version.

```shell theme={null}
helm repo update
helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace --set onPremToken=$ODIGOS_TOKEN
```

The license token can also be provided by creating a secret in kubernetes before installing Odigos.

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: odigos-pro
  namespace: odigos-system
  labels:
    odigos.io/system-object: "true"
stringData:
  odigos-onprem-token: <TOKEN>
```

When pulling enterprise images from `registry.odigos.io`, Odigos also needs a Docker registry pull secret named `odigos-enterprise-registry`. The Helm chart can create this automatically when `onPremToken` is set, or when it can read the token from an existing `odigos-pro` secret.

For GitOps tools such as Argo CD that cannot read cluster secrets during `helm template`, manage both secrets externally and set:

```yaml theme={null}
externalOnpremTokenSecret: true
externalOnpremPullSecret: true
```

Those flags only provide the pull secret in the Odigos namespace. With `k8s-init-container` on Kubernetes 1.35+, copy that secret into each instrumented namespace as well. See [Init container image pull secrets](/enterprise/instrumentations/configuration/mount-method#init-container-image-pull-secrets).

Create the registry pull secret before syncing Odigos. The username is always `odigos` and the password is your enterprise token:

```shell theme={null}
kubectl create secret docker-registry odigos-enterprise-registry \
  --namespace odigos-system \
  --docker-server=registry.odigos.io \
  --docker-username=odigos \
  --docker-password="$ODIGOS_TOKEN"
kubectl label secret odigos-enterprise-registry -n odigos-system odigos.io/system-object=true
```

Or apply it declaratively:

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: odigos-enterprise-registry
  namespace: odigos-system
  labels:
    odigos.io/system-object: "true"
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "registry.odigos.io": {
          "username": "odigos",
          "password": "<TOKEN>",
          "auth": "<BASE64_OF_odigos:TOKEN>"
        }
      }
    }
```

Generate the `auth` value with:

```shell theme={null}
echo -n "odigos:$ODIGOS_TOKEN" | base64
```

Once the Odigos pods are online, simply run the following command to launch the Odigos UI.

```shell theme={null}
odigos ui
```

## OpenShift Operator

For Operator installation steps, see the docs on: [OpenShift Installation](/enterprise/setup/installation-options#openshift-installation)


# Installation Options
Source: https://docs.odigos.io/enterprise/setup/installation-options



You can configure odigos installation using the following options:

## Ignore Namespaces

Don't show specific uninteresting namespaces in the Odigos UI.

Default values: `kube-system`, `local-path-storage`, `istio-system`, `linkerd`, `kube-node-lease`

Add additional namespaces to ignore with **cli**:

```shell theme={null}
odigos install --ignore-namespace <namespace1> --ignore-namespace <namespace2>
```

List all namespaces to be ignored with **helm chart**:

```shell theme={null}
helm install odigos odigos/odigos --set ignoreNamespace=<namespace1>,<namespace2>
```

Any Source objects for the namespace and workloads in the namespace will still be honored.

On GKE, Google may warn that a custom admission controller can affect `kube-system`. The Odigos **pod mutating webhook** is cluster-wide so it can inject agents into application namespaces. It uses `failurePolicy: Ignore`, so webhook errors do not block pod creates in `kube-system`. Ignored namespaces (including `kube-system` by default) are not shown for instrumentation and are not instrumented unless you create a Source for them.

## Ignore Containers

This option is useful for excluding specific sidecar containers from being instrumentated by odigos.

Default values: `istio-proxy`

**Odigos CLI**

```shell theme={null}
odigos install --ignore-container <container1> --ignore-container <container2>
```

**Helm Chart**

```shell theme={null}
helm install odigos odigos/odigos --set ignoreContainer=<container1>,<container2>
```

## Custom Container Runtime Socket Path

Odigos uses the container runtime socket to query for application environment variables info, and distinguish between those injected by the runtime (via Dockerfile for example) and those defined in Kubernetes manifests.

To perform this analysis, Odiglet requires access to the container runtime’s Unix socket (e.g., `containerd.sock`). However, in certain environments—such as Rancher—the socket may not be located in a default path.

To address this, Odigos provides a configuration option to mount a custom container runtime socket path into the Odiglet. This path can be specified either during installation or afterwards.

### Usage

**Odigos CLI**

1. During and Post installation using set flag for the install command:

```shell theme={null}
odigos install --set odiglet.customContainerRuntimeSocketPath=<unix socket path>
```

**Helm Chart**

1. During installation:

```shell theme={null}
helm install odigos odigos/odigos --set odiglet.customContainerRuntimeSocketPath=<unix socket path>
```

## OpenShift Installation

Installing on OpenShift requires additional settings that can be
configured based on your installation mode:

**Odigos CLI**

Run the Odigos CLI with the `--openshift` flag:

```shell theme={null}
odigos install --openshift
```

**Helm Chart**

Set `openshift.enabled=true` in `Values.yaml`:

```yaml theme={null}
openshift
  enabled: true
```

**Operator**

In the OpenShift console, navigate to **Operators** > **OperatorHub** and search for **Odigos**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-2.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=0b2238fee5cdadcd3be83db9fd304204)

Click **Install**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-3.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=6c0311adaf5191c43c4054eaddc42d91)

Choose a namespace to install the Operator in (this will be the same namespace that Odigos is installed in) and click **Install**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-4.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=98c89e169348fc88301148d351d4751e)

When installation is complete, click **View Operator**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-5.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=72e90cbaa7bb630f1efcbe1aaf5a7187)

Click **Create Instance** to create an `Odigos` custom resource:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-6.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=c2bec9274033f6e850c336d50651fe41)

If you have an Enterprise Token, fill in the **On Prem Token** field with your Enterprise Token.  Otherwise leave it blank to use Odigos OpenSource.

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-7.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=569391cd0146df727538b74b1b6ba6f7)

Scroll down and click **Create**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-8.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=db6b4869e4a22358525fd4f29fed57a7)

Confirm that the Custom Resource has reached state `OdigosInstalled`:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-9.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=f35484a82b81ace0914ba3536a301aa2)

OpenShift installation does the following:

* Uses [Red Hat certified component container images](https://odigos.io/blog/openshift-certified)
* Configures required SELinux permissions for pods to access auto-instrumentation agents on RHEL nodes
* Configures RBAC permissions for Odigos components required by default in OpenShift clusters

By default, the `--openshift` flag will pull Odigos component images
from `registry.connect.redhat.com/odigos`. These images are based on
[Red Hat's Universal Base Image
9](https://catalog.redhat.com/software/base-images) and denoted with
the `-rhel-certified` suffix. These images are also hosted on the Odigos
DockerHub registry, which you can deploy by setting `--openshift` with
`--image-prefix docker.io/keyval`.

### Exposing the Odigos UI via OpenShift Routes

To access the Odigos UI in OpenShift, use either the standard `odigos
ui` command or create a `Route` that exposes the `ui` service on your
cluster's external address. For example:

```
oc expose service ui -n odigos-system
```

Then get the external address for the Route with `oc get routes ui -n
odigos-system`.

**Note:** If you're deploying Odigos on OpenShift, you may need to set the route's TLS termination to Edge to load it in your browser.
you will need to modify the YAML of the oc route with `spec.tls.termination: edge`

Check the OpenShift documentation or run `oc expose -h` for more
information on configuring OpenShift routes according to your
cluster's requirements.

## GKE Installation

Installing on Google Kubernetes Engine (GKE) requires additional settings
that can be configured based on your cluster type:

**Odigos CLI**

For GKE Standard clusters:

```shell theme={null}
odigos install --set gke.enabled=true
```

For GKE Autopilot clusters:

```shell theme={null}
odigos install --set gke.enabled=true --set gke.autopilot=true
```

**Helm Chart**

For GKE Standard clusters, set `gke.enabled=true` in `values.yaml`:

```yaml theme={null}
gke:
  enabled: true
```

Or pass it as a flag during installation:

```shell theme={null}
helm install odigos odigos/odigos --set gke.enabled=true
```

For GKE Autopilot clusters, also set `gke.autopilot=true`:

```yaml theme={null}
gke:
  enabled: true
  autopilot: true
```

Or pass both flags during installation:

```shell theme={null}
helm install odigos odigos/odigos --set gke.enabled=true --set gke.autopilot=true
```

GKE installation with `gke.enabled=true` does the following:

* Configures a `ResourceQuota` for `system-node-critical` pods, which is required for odiglet on GKE

### GKE Autopilot

GKE Autopilot restricts privileged workloads by default. Odiglet requires
elevated permissions for eBPF-based instrumentation, so it must be allowlisted
before it can run on Autopilot clusters.

Allowlist synchronization requires GKE version `1.32.2-gke.1652000` or later.

When `gke.autopilot=true`, the Helm chart:

* Creates an `AllowlistSynchronizer` that syncs the Odigos odiglet allowlist from the [Google-managed partner repository](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads)
* Skips node label setup in the odiglet init phase, where Autopilot nodes cannot be modified. Autopilot is auto-detected by the `gk3-` node name prefix; set `gke.autopilot=true` to force this behavior

#### Node label limitations

On standard clusters, odiglet sets `odigos.io/odiglet-oss-installed=true` (or
`odigos.io/odiglet-enterprise-installed=true` for enterprise) on each node after
preparing it for instrumentation. Autopilot does not allow modifying node labels,
so odiglet skips this step.

Without the label:

* **Instrumented pod scheduling with hostPath or CSI mount methods** — For
  `k8s-host-path` and `k8s-csi-driver` mount methods, the instrumentor webhook
  adds required node affinity matching the odiglet installed label so pods only
  schedule on nodes where odiglet has prepared `/var/odigos`. Without the label
  on Autopilot nodes, newly instrumented pods may remain Pending and fail to
  schedule. The default `k8s-virtual-device` mount method does not rely on this
  label and is recommended on Autopilot.
* **Karpenter integration** — Odiglet normally removes the `odigos.io/needs-init`
  startup taint when it sets the node label. This step is also skipped on
  Autopilot, so the Karpenter integration workflow is not supported.

#### Manually create the AllowlistSynchronizer

If you do not set `gke.autopilot=true`, create the `AllowlistSynchronizer` yourself
before odiglet pods are scheduled. See the [GKE documentation for running Autopilot
partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads)
for more details.

Apply the following manifest to install all Odigos allowlists from the partner
repository:

```yaml theme={null}
apiVersion: auto.gke.io/v1
kind: AllowlistSynchronizer
metadata:
  name: odigos-allowlist-synchronizer
spec:
  allowlistPaths:
    - "Odigos/Odigos/*"
```

```shell theme={null}
kubectl apply -f allowlist-synchronizer.yaml
```

After the allowlist is installed, verify that it is ready before odiglet pods start:

```shell theme={null}
kubectl wait --for=condition=Ready allowlistsynchronizer/odigos-allowlist-synchronizer --timeout=60s
kubectl get workloadallowlist
```

For more information on privileged workload allowlisting in Autopilot, see the
[GKE documentation on running Autopilot partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads).


# Configure Odigos with Argo CD
Source: https://docs.odigos.io/enterprise/setup/odigos-with-argocd



This guide shows you how to deploy Odigos in your Kubernetes cluster using Argo CD with the official Odigos Helm chart. This allows for GitOps-based management and observability instrumentation across your applications.

## Prerequisites

Before you begin, make sure you have:

* A running Kubernetes cluster
* Argo CD installed and configured in the cluster
* Helm CLI installed (optional but useful for inspection and verification)

## Step 1: Add the Odigos Helm Repository

Add the Odigos Helm repository to your Helm configuration:

```bash theme={null}
helm repo add odigos https://odigos-io.github.io/odigos --force-update
helm repo update
```

## Step 2: Identify the Helm Chart Version

Check available versions of the Odigos chart to determine the desired `targetRevision`:

```bash theme={null}
helm search repo odigos --versions
```

## Step 3: Define the Argo CD Application Manifest

Create an Argo CD `Application` custom resource to deploy Odigos using Helm. Save the following YAML as `odigos-application.yaml`:

```yaml theme={null}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: odigos
  namespace: <ARGOCD_NAMESPACE>  # <- Replace with your Argo CD namespace
  finalizers:
  - resources-finalizer.argocd.argoproj.io/foreground  # <- # Cascade-delete managed resources when the Application is deleted (otherwise only the Application CR is removed).

spec:
  project: default
  source:
    repoURL: https://odigos-io.github.io/odigos
    chart: odigos
    targetRevision:  # REQUIRED: set the version from 'helm search repo odigos --versions'
    helm:
      releaseName: odigos
      values: |
        # OPTIONAL: add your custom Helm values here if needed.
        # Argo CD expects this field to be a YAML-formatted string.
        # To use chart defaults, delete this field entirely or set it to an empty string ("").
  destination:
    server: https://kubernetes.default.svc
    namespace: <ODIGOS_NAMESPACE>  # <- Replace with the desired namespace where you want Odigos to be installed
  # Chart regenerates odigos-deployment-id on every Argo render; ignore it to avoid OutOfSync flaps.
  ignoreDifferences:
    - kind: ConfigMap
      name: odigos-deployment
      jsonPointers:
        - /data/odigos-deployment-id
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      # Without this, sync still overwrites ignored fields from the rendered manifest.
      - RespectIgnoreDifferences=true
```

## Step 4: Apply the Application

Use `kubectl` to create the application in Argo CD:

```bash theme={null}
kubectl apply -f odigos-application.yaml
```

This will instruct Argo CD to deploy Odigos using the Helm chart.

## Step 5: Monitor the Deployment

You can monitor the status of the deployment through the Argo CD UI:

* Navigate to your Argo CD web interface.
* Find the application named `odigos`.
* Verify that it reaches a **Healthy** and **Synced** state.

Once complete, Odigos will be installed in your cluster and ready to instrument workloads.


# Configure Odigos with Karpenter
Source: https://docs.odigos.io/enterprise/setup/odigos-with-karpenter



## What is Karpenter?

[Karpenter](https://karpenter.sh/) is an open-source cluster autoscaler built for Kubernetes. It dynamically launches just the right compute resources to handle your cluster’s workloads and efficiently scales them based on real-time demand.

Unlike traditional autoscalers, Karpenter can provision nodes in seconds and makes scheduling decisions at the pod level, enabling highly dynamic infrastructure scaling.

***

## Mount Method Compatibility

Odigos support for Karpenter depends on the configured [mount method](../instrumentations/configuration/mount-method):

| Mount method | Karpenter support | What you need |
| - | - | - |
| [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer) | Works out of the box | No additional Karpenter configuration |
| [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) | Supported with Odigos Karpenter integration | Enable `karpenter.enabled` and configure NodePool `startupTaints` (see [HostPath and CSI](#hostpath-and-csi-startup-taints)) |
| [`k8s-virtual-device`](../instrumentations/configuration/mount-method#1-VirtualDevice) (default) | Supported only when your Karpenter installation can apply [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) capacity | Enable the NodeOverlay feature (when required) and apply a NodeOverlay CR (see [Virtual Device](#virtual-device-nodeoverlay)) |

Follow the section for your mount method below.

***

## Init Container

If you use `k8s-init-container`, no Karpenter-specific configuration is required. You can stop here.

***

## Virtual Device (NodeOverlay)

With `k8s-virtual-device`, instrumented pods request the extended resource `instrumentation.odigos.io/generic`. That capacity is advertised by the Odigos device plugin on nodes where `odiglet` is already running.

Karpenter simulates new nodes from instance-type information. Because that simulation does not include the Odigos device resource by default, Karpenter can fail provisioning with errors such as:

```text theme={null}
no instance type has enough resources ... instrumentation.odigos.io/generic: "1"
```

### Provider and version caveats

Support depends on your **Karpenter version, cloud provider, and how the controller is packaged**:

* [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is an alpha Karpenter feature (available since Karpenter v1.7.x). Exact enablement steps and API readiness vary by release.
* On many installs (including official AWS Karpenter Helm charts), NodeOverlay is **disabled by default** and must be turned on with a feature gate, for example:

```bash theme={null}
helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace <KARPENTER_NAMESPACE> \
  --reuse-values \
  --set settings.featureGates.nodeOverlay=true
```

Equivalently, set `--feature-gates NodeOverlay=true` / `FEATURE_GATES=...,NodeOverlay=true` on the Karpenter controller.

* Not every Karpenter distribution applies NodeOverlays during provisioning, even if the CRD and feature gate exist. Validate in your environment that creating the overlay below allows Karpenter to launch nodes for pods that request `instrumentation.odigos.io/generic`.

If NodeOverlay is unavailable or ineffective in your Karpenter install, use [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer), or [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) with the [startup taint integration](#hostpath-and-csi-startup-taints).

### Apply a NodeOverlay

After NodeOverlay support is enabled in your Karpenter controller, apply a [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) that adds the Odigos device capacity for the NodePool that should run instrumented workloads.

Replace `default` with your NodePool name if needed:

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: odigos-instrumentation
spec:
  requirements:
    - key: karpenter.sh/nodepool
      operator: In
      values: ["default"]
  capacity:
    instrumentation.odigos.io/generic: "1024"
EOF
```

Confirm the overlay is ready:

```bash theme={null}
kubectl get nodeoverlay odigos-instrumentation
```

Karpenter should then be able to provision nodes for pods that request `instrumentation.odigos.io/generic`. After the node registers, `odiglet` advertises the real device capacity so pods can schedule.

***

## HostPath and CSI (Startup Taints)

For the [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) and [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) mount methods, Odigos adds a **node affinity** rule to instrumented pods so they only run on nodes where `odiglet` is installed.

This works fine in static clusters — but in Karpenter-managed clusters, this causes a problem:

* `odiglet` is installed by a DaemonSet **after** the node is created.
* The label `odigos.io/odiglet-installed=true` is only added to the node at runtime.
* **Karpenter does not know in advance** that this label will appear.
* So when the scheduler tries to place a pod with this affinity, Karpenter concludes: "no nodes exist (or can be created) that match this label" — and **no node is provisioned**.

Once you enable the **Karpenter integration option** (explained below), Odigos will no longer apply node affinity to new instrumented pods.
Instead, it relies on a **startup taint mechanism**:

* Nodes launched by Karpenter should be initialized with a special taint (`odigos.io/needs-init=NoSchedule`).
  \[!NOTE] The exact method for configuring this taint in Karpenter will be explained in the following section.
* The `odiglet` removes this taint after it prepares the node for instrumentation.
* This allows instrumented pods to schedule only **after** the node is ready — achieving the same safety as affinity, without blocking provisioning.

### 1. Configure the Karpenter NodePool CRD

You must configure your Karpenter [NodePool](https://karpenter.sh/docs/concepts/nodepools/) to include a startup taint. This ensures Odigos can prepare each node before instrumented pods are scheduled onto it.

Add the following under `spec.template.spec.startupTaints`:

```yaml theme={null}
spec:
  template:
    spec:
      startupTaints:
      - key: odigos.io/needs-init
        effect: NoSchedule
```

To safely append this taint without overwriting any existing `startupTaints`, use the following command:

```bash theme={null}
export NODEPOOL_NAME=<NODEPOOL_NAME>
kubectl patch nodepool $NODEPOOL_NAME --type merge --patch "$(kubectl get nodepool $NODEPOOL_NAME -o json | jq -c '.spec.template.spec.startupTaints = (.spec.template.spec.startupTaints // []) + [{"key":"odigos.io/needs-init","effect":"NoSchedule"}] | {spec: {template: {spec: .spec.template.spec}}}')"
```

Replace `<NODEPOOL_NAME>` with the actual name of your Karpenter NodePool.

### 2. Add the Startup Taint to Existing Nodes

To manually add the startup taint to all current nodes:

```bash theme={null}
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} kubectl taint node {} odigos.io/needs-init=:NoSchedule --overwrite
```

This ensures existing nodes behave consistently until they're replaced by Karpenter-managed ones.

### 3. Enable the Karpenter Integration in Odigos

Once the tainting setup is complete, inform Odigos to disable node affinity and rely on the taint mechanism.

**Odigos CLI**

Run the following using the Odigos CLI:

```bash theme={null}
odigos set config karpenter-enabled=true
```

**Helm Chart**

If installing or upgrading using helm:

```bash theme={null}
helm upgrade --install odigos odigos/odigos \
  --namespace odigos-system \
  --set karpenter.enabled=true
```

### 4. Restart the Odiglet DaemonSet

To ensure that taint-removal logic is triggered on all nodes, perform a rolling restart of the `odiglet` DaemonSet:

```bash theme={null}
kubectl rollout restart daemonset/odiglet -n odigos-system
```


# System Requirements
Source: https://docs.odigos.io/enterprise/setup/system-requirements

To ensure optimal performance and compatibility when deploying Odigos, your environment must meet the following system requirements:

## Kubernetes Version

Odigos requires Kubernetes version 1.21.12 or higher.

> [!NOTE]
>
> Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.
>
> The required Linux capabilities depend on your kernel version:
>
> * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`.
> * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`.
>
> We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`.

**What to expect if some nodes don't meet the Kernel requirement:**

* eBPF Agents: If a node's kernel version is below 5.4.0, eBPF-based data collection will not be applied to workloads on that node. An error will appear in the UI for any source running on unsupported nodes, indicating that instrumentation was not possible.
* Non-eBPF Agents: Non-eBPF-based agents are not affected by kernel version requirements and will run and instrument workloads on any Linux version.

This allows users to maintain observability on nodes that meet the requirements without needing to upgrade all cluster nodes immediately.

## Kubernetes Cluster Types

* [Kind](https://kind.sigs.k8s.io)
* [Minikube](https://minikube.sigs.k8s.io/docs/start)
* [K3s](https://k3s.io)
* [Red Hat OpenShift](https://catalog.redhat.com/software/container-stacks/detail/675201b3ade18c062e2efc0b) (see [how to install on OpenShift](../setup/installation-options#openshift-installation))
* [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)
* [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) (see [how to install on GKE](/enterprise/setup/installation-options#gke-installation))
* [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service)

## Resource Requirements

To achieve its functionality Odigos uses a set of Deployments and DaemonSets.

### Cluster

The following components run as Deployment and request the following resources **(per pod)**:

| Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit |
| - | - | - | - | - |
| odigos-autoscaler | 64Mi | 512Mi | 10m | 500m |
| odigos-instrumentor (2 replicas) | 64Mi | 512Mi | 10m | 500m |
| odigos-ui | 64Mi | 512Mi | 10m | 500m |

In addition, Odigos deploys OpenTelemetry Collector Deployment,
with auto scaling (multiple replicas) and configurable resources requests and limits.
The default values (unless overridden) are:

| Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit |
| - | - | - | - | - |
| odigos-gateway | 600Mi | 600Mi | 1000m | 1000m |

Read more [here](/enterprise/pipeline/configuration).

### Per Node

A DaemonSet running on every node in the cluster, consuming resources as outlined below:

| Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit |
| - | - | - | - | - |
| odiglet | 500Mi | 500Mi | 500m | 500m |
| data-collection | 250Mi | 500Mi | 250m | 500m |
| deviceplugin | 200Mi | 300Mi | 40m | 100m |

You can override these values by providing a custom configuration (read more [here](/enterprise/pipeline/configuration)).

## Network Requirements

> [!WARNING]
>
> Ensure that any network-layer components are properly configured to allow the required connectivity between Odigos components. Improper configuration may impact the correct operation of Odigos.

To ensure proper operation of Odigos components and instrumented applications, the following network connectivity must be allowed inside the Kubernetes cluster:

### Core Network Connectivity

| Source | Destination Service | Ports | Purpose |
| - | - | - | - |
| Instrumented Application Pods | `odiglet-local` | `4320` | Retrieve instrumentation configuration from Odiglet |
| Instrumented Application Pods | `odigos-data-collection-local-traffic` | `4317` (gRPC), `4318` (HTTP) | Export application telemetry data (OTLP protocol) |
| Odiglet Pods | `odigos-data-collection-local-traffic` | `4318` | Export eBPF-collected telemetry data |

### Network Layer Considerations

In some environments, the following network components may affect traffic between these services:

* Service meshes (e.g. Istio, Linkerd, Cilium, Consul)
* HTTP proxies or egress gateways
* Transparent proxies (corporate or cluster-level)
* Network security appliances (WAFs, L7 firewalls, DPI systems)
* Kubernetes NetworkPolicies


# Uninstall
Source: https://docs.odigos.io/enterprise/setup/uninstall



## With CLI

```shell theme={null}
odigos uninstall
```

## Uninstall Existing Odigos Installation with Helm

If you installed Odigos using Helm, you can uninstall it by running the following commands:

```shell theme={null}
helm delete odigos -n odigos-system
kubectl delete ns odigos-system
```


# Upgrade
Source: https://docs.odigos.io/enterprise/setup/upgrade



## Finding the Current Version

```shell theme={null}
odigos version
```

Will print the version of the CLI tool and the version of odigos that is currently installed in the active cluster.

## Upgrading Existing Odigos Installation with CLI

First, upgrade the CLI tool to latest version:

**Homebrew**

If you are on MacOS, you can install Odigos CLI using Homebrew:

```bash theme={null}
brew install odigos-io/odigos-cli/odigos
```

**GitHub Releases**

1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux).

2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system.

3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH".

* For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt.

* For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`.

4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH.

**Windows**

1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows).

2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files.

3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how:

   * **Option 1**:
     Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder.

   * **Option 2**:
     Add the folder containing the extracted binary to your PATH manually:
     1. Right-click the **Start** button and select **System**.
     2. Click **Advanced system settings** on the left.
     3. In the **System Properties** window, click **Environment Variables** at the bottom.
     4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**.
     5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file.
     6. Click **OK** to close all windows.

4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located.

And upgrade Odigos installation in the active kubernetes cluster with CLI:

```shell theme={null}
odigos upgrade
```

## Upgrade Existing Odigos Installation with Helm

**Upgrade to Latest Version**

```shell theme={null}
helm repo update
helm upgrade odigos odigos/odigos --namespace odigos-system
```

**Upgrade to Specific Version**

```shell theme={null}
helm repo update
helm search repo odigos --versions ## list all available versions
helm upgrade --install odigos odigos/odigos \
--namespace odigos-test \
--create-namespace \
--version <VERSION>
```


# Verifying Odigos Images
Source: https://docs.odigos.io/enterprise/setup/verifying-images



All Odigos container images are signed at build time with [Sigstore cosign](https://docs.sigstore.dev/)
(keyless), and each carries a CycloneDX SBOM attestation. Signatures are stored in the
registry next to the image and recorded in the public
[Rekor transparency log](https://search.sigstore.dev/), so you can verify that any image
you pull was published by the Odigos release pipeline — no keys involved.

## Verify an image

Verification needs [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) v3+.
Always verify by digest:

> [!NOTE]
>
> Verify against the canonical registry, `us-central1-docker.pkg.dev/odigos-cloud/components`.
> `registry.odigos.io` is a pull alias: images resolve through it, but signature
> discovery for retroactively signed versions does not currently surface there.

```bash theme={null}
IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-odiglet
DIGEST=$(docker buildx imagetools inspect "${IMG}:<version>" --format '{{.Manifest.Digest}}')

cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \
  "${IMG}@${DIGEST}"
```

The identity accepts two workflows on purpose: new releases are signed by the
release pipeline, while versions published before signing went live were signed
retroactively by the audited backfill workflow. A single-identity pattern would
fail on one era or the other.

To verify and inspect the attached SBOM (same identity rules):

```bash theme={null}
cosign verify-attestation --type cyclonedx \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \
  "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components'
```

The SBOM attestation is attached to the **multi-arch index digest** — always pass
the digest from `imagetools inspect` as above, not a per-platform child digest
(children carry signatures but no attestation).

> **Full worked example: verifying odigos-instrumentor v1.35.0**
>
> Run as-is — no placeholders to fill in:
>
> ```bash theme={null}
> IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor
> TAG=v1.35.0
> DIGEST=$(docker buildx imagetools inspect "${IMG}:${TAG}" --format '{{.Manifest.Digest}}')
> IDENTITY='^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$'
>
> cosign verify \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp "$IDENTITY" \
>   "${IMG}@${DIGEST}"
>
> cosign verify-attestation --type cyclonedx \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp "$IDENTITY" \
>   "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components | length'
> ```
>
> Expected output (trimmed):
>
> ```text theme={null}
> Verification for us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor@sha256:f46fa614… --
> The following checks were performed on each of these signatures:
>   - The cosign claims were validated
>   - Existence of the claims in the transparency log was verified offline
>   - The code-signing certificate was verified using trusted certificate authority certificates
> [{"critical":{"identity":{"docker-reference":"…/odigos-instrumentor"},
>   "image":{"docker-manifest-digest":"sha256:f46fa614…"},"type":"cosign container image signature"}, …}]
>
> 1022
> ```
>
> The 1022 components are the SBOM contents — inspect them with
> `jq '.predicate.components'` instead of `length`, e.g. the first entries here are
> `base-files 13.8+deb13u6` and `github.com/argoproj/argo-rollouts v1.9.1`. Because
> v1.35.0 predates release-time signing, the certificate for this version names the
> audited backfill workflow (`ci-core/.github/workflows/backfill-sign.yml`) — that is
> the expected identity for retroactively signed history, and the reason the identity
> regexp accepts both workflows.

## Signing identities

Each artifact is signed by its own release workflow — never verify with a broad
pattern like `.*odigos-io.*`. The canonical list of identities and the version each
artifact is signed from lives in
[`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md).

| Artifact | Release identity (regexp) |
| - | - |
| Component images (`odigos-*`) | `^https://github\.com/odigos-io/odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*$` |
| CLI image + `odigos-victoria-metrics` + `odigos-cache` | `^https://github\.com/odigos-io/odigos/\.github/workflows/release\.yml@refs/heads/main$` |
| Enterprise images (`odigos-enterprise-*`) | `^https://github\.com/odigos-io/odigos-enterprise/\.github/workflows/release-images\.yml@refs/(tags/.*\|heads/.*)$` |
| `odigos-cli-offsets` | `^https://github\.com/odigos-io/enterprise-go-instrumentation/\.github/workflows/offsets-push\.yml@refs/heads/.*$` |
| `odigos-vmagent-instrumentations` | `^https://github\.com/odigos-io/vm-agent/\.github/workflows/publish-agents\.yml@refs/(heads/.*\|tags/agents/.*)$` |

To also cover retroactively signed history, combine the release identity with the
backfill identity in one pattern, as the commands above do:
`^https://github\.com/odigos-io/(<release workflow path>@<ref>|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$`

Two more certificate details are expected and legitimate:

* **Backfill**: historical releases were signed retroactively by
  `odigos-io/ci-core/.github/workflows/backfill-sign.yml` — the certificate honestly
  records the backfill identity and date, and it is auditable in Rekor.
* **Stable `x.y.0` tags** verify against the release-candidate tag identity: the stable
  tag is a retag of the exact RC digest, so the certificate names the RC that built it.

## Attestable images

Every image below is signed and carries a CycloneDX SBOM attestation, from the
version listed in [`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md):

* **Odigos platform**: `odigos-instrumentor`,
  `odigos-collector`, `odigos-odiglet`, `odigos-ui`, `odigos-operator`, `odigos-agents`,
  `odigos-cli`, `odigos-victoria-metrics`, `odigos-cache`
* **Odigos Enterprise**: `odigos-enterprise-odiglet`, `odigos-enterprise-instrumentor`,
  `odigos-enterprise-ui`, `odigos-enterprise-collector`, `odigos-enterprise-central-proxy`,
  `odigos-enterprise-central-backend`, `odigos-enterprise-central-ui`, `odigos-enterprise-agents`,
  `odigos-enterprise-connector-runtime`, `odigos-enterprise-connector-aws`,
  `odigos-enterprise-connector-azure`, `odigos-enterprise-connector-gcp` — `odiglet`
  and `agents` also publish `-extended` variants (separate digests, same identity)
* **Supporting images**: `odigos-cli-offsets`, and
  `odigos-vmagent-instrumentations` (signed for publishes after 2026-08-12)

## CLI binary verification

Every [GitHub release](https://github.com/odigos-io/odigos/releases) ships a
`checksums.txt` covering the CLI archives, keyless-signed by the release pipeline.
The Sigstore bundle (`checksums.txt.sigstore.json`) is published as a release asset
next to it. Verify the checksum file, then check your downloaded archive against it:

```bash theme={null}
VERSION=<version>   # the release you downloaded, e.g. v1.40.0
BASE=https://github.com/odigos-io/odigos/releases/download/${VERSION}

curl -sfLO "${BASE}/checksums.txt"
curl -sfLO "${BASE}/checksums.txt.sigstore.json"

cosign verify-blob \
  --bundle checksums.txt.sigstore.json \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/odigos-io/odigos/\.github/workflows/release\.yml@refs/heads/main$' \
  checksums.txt

curl -sfLO "${BASE}/cli_${VERSION#v}_linux_amd64.tar.gz"   # your platform's archive
sha256sum -c checksums.txt --ignore-missing
```

`--ignore-missing` limits the check to the archives you actually downloaded
(on macOS use `shasum -a 256 -c checksums.txt --ignore-missing`).

## Air-gapped verification

The Rekor proof travels inside the signature bundle, so verification works offline:
supply `--trusted-root` with a Sigstore trusted-root file instead of relying on
`--offline` (removed in cosign 3.x).

## Linux package signing keys

Every `.rpm` and `.deb` we publish is signed with the current key below. The keyring at
`https://raw.githubusercontent.com/odigos-io/odigos/main/docs/keys/odigos-package-signing-keys.asc` always contains every key that signed a published package: point `gpgkey=` at it once
and never change it, rotation only appends.

| Fingerprint | Issued | Status | Notes |
| - | - | - | - |
| `6E80516AF71835E5FD12E3D8272251E89CC74CFF` | 2026-10-18 | current | initial package signing key (RUN-1042) |


# Telemetry Types
Source: https://docs.odigos.io/enterprise/telemetry-types



Odigos supports the producing and correlation of the following telemetry types:

* Traces
* Metrics
* Logs

For a telemetry type to be collected, a [destination](./backends-overview) that accepts it must be
configured. In addition, users can enable or disable the collection of a
telemetry type on a per-application basis. If none of the applications are
configured to collect a telemetry type, the telemetry type will not be
collected.

## Traces

Distributed traces collected by Odigos will automatically include spans for
popular open source projects such as HTTP clients and servers, gRPC clients and
servers, Database clients, and many more.

In addition, users can enrich their distributed traces with manually created
spans by using the relevant
[OpenTelemetry APIs](https://opentelemetry.io/docs/instrumentation/).

The collection of traces is achieved by combining two open source technologies:

* OpenTelemetry for languages with JIT compilation such as Python, Java,.NET and
  Javascript.
* eBPF for compiled languages such as Go.

Being based on popular open-source standards allows Odigos to automatically
support a huge number of libraries and frameworks. Below is a list of supported
libraries for every language:

> **[Go](https://github.com/odigos-io/opentelemetry-go-instrumentation/tree/master/pkg/instrumentors/bpf)**
>
> opentelemetry-go-instrumentation/ pkg/ instrumentors/ bpf/

> **[Javascript](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node)**
>
> opentelemetry-js-contrib/ plugins/ node/

> **[Python](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation)**
>
> opentelemetry-python-contrib/ instrumentation/

> **[Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation)**
>
> opentelemetry-java-instrumentation/ instrumentation/

> **[PHP](https://github.com/open-telemetry/opentelemetry-php-contrib/tree/main/src/Instrumentation)**
>
> opentelemetry-php-contrib/ src/ Instrumentation/

> **[Ruby](https://github.com/open-telemetry/opentelemetry-ruby-contrib/tree/main/instrumentation)**
>
> opentelemetry-ruby-contrib/ instrumentation/

> **[.NET](https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/tree/main/docs/config.md#instrumented-traces-libraries-and-frameworks)**
>
> opentelemetry-dotnet-instrumentation/ docs/ config.md

## Metrics

There are three kinds of metrics that Odigos supports:

* Metrics related to the running of the application (number of HTTP requests,
  latency, DB connections, etc.)
* Metrics related to the language runtime (GC, threads, heap, etc.)
* Metrics related to the host environment (CPU, memory, disk, etc.)

### Applicative Metrics

Application-related metrics are derived from the distributed traces described
above. Many observability vendors automatically compute application metrics
according to the distributed traces. For example, the number of requests served
by the application is computed by counting the number of spans with the
`http.server` label. In case a destination that does not automatically compute
application metrics (like Prometheus + Tempo) is configured, Odigos will compute
them on its own by invoking the
[spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md)
connector.

### Runtime Metrics

Those metrics describe the behavior of the language runtime, such as garbage
collection, threads, heap, etc. Runtime metrics are collected by the same
mechanism that collects distributed traces.

### Host Metrics

Host-related metrics are collected via the
[host metrics receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver)

## Logs

Currently, Odigos will ship logs written by the application to the stdout or
stderr of the process. Correlation to other telemetry types will be done
according to timestamp and resource.

We are in the process of shipping logs using the same mechanism that collects
distributed traces, this would allow users to correlate logs to distributed
traces and metrics in a more accurate way.


# Usage report
Source: https://docs.odigos.io/enterprise/usage-reports



Odigos sends non-sensitive, non-personally identifiable information to our system.
This information helps Odigos maintainers to understand how the open-source community runs Odigos and
improve the project.

### Disabling the report

#### Using helm chart

When installing Odigos using helm chart, you can disable the report by setting `telemetry.enabled` to `false`.

#### Using CLI

When installing Odigos using CLI, you can disable the report by setting `--telemetry` to `false`.

### What is being reported

Odigos collects the following anonymous information:

* Are traces/metrics/logs enabled
* Selected destinations type (e.g. Jaeger, Prometheus, etc.)
* Number of instrumented applications by programming language
* Number of unrecognized applications
* Number of nodes


# Adding New Observability Destination
Source: https://docs.odigos.io/oss/adding-new-dest

There are tens if not hundreds of different observability destinations. Odigos's goal is to provide a seamless and easy way to ship observability data to any one of them.

In this guide, you will learn how to contribute a new destination to Odigos. We will create a new dummy destination called `mydest`.

Creating a new destination involves these steps:

1. Extending the UI for the new destination
2. Adding the collector configuration for the new destination
3. Generating the documentation for the new destination

### User Interface

For our new destination to be visible in the UI, we need to define the destination metadata and fields in YAML format:

**1. Add Logo**

Go to [`destinations/logos/`](https://github.com/odigos-io/odigos/tree/main/destinations/logos) directory and add your logo.

* Must be in SVG format
* Example: `mydest.svg`

```svg mydest.svg theme={null}
<svg fill="#F5F5F5" width="16" height="12" viewBox="0 0 16 12" xmlns="http://www.w3.org/2000/svg">
  <title>My Destination</title>
  <g>
    <path d="M5.17215 9.01884L2.18377 7.69813C1.48159 7.38732 1.0448 6.73677 1.0448 5.99948C1.0448 5.2622 1.48056 4.61165 2.18377 4.30083L6.36172 2.45349L6.45156 2.41425V0H1.66127C1.0417 0 0.538818 0.4781 0.538818 1.06772V10.9312C0.538818 11.5209 1.0417 11.999 1.66127 11.999H5.83303H6.45259V9.5682L5.17319 9.01781H5.17215V9.01884Z"/>
    <path d="M14.3397 0H9.5484V2.4411L10.8041 2.98115L10.801 2.9698L13.8162 4.3029C14.5184 4.61372 14.9552 5.26426 14.9552 6.00155C14.9552 6.73883 14.5194 7.38938 13.8162 7.7002L10.1649 9.31417L9.54736 9.57646V12H14.3387C14.9583 12 15.4611 11.5219 15.4611 10.9323V1.06772C15.4611 0.4781 14.9583 0 14.3387 0H14.3397Z"/>
    <path d="M8.00048 8.47972C9.36972 8.47972 10.4808 7.36966 10.4808 5.99938C10.4808 4.6291 9.37076 3.51904 8.00048 3.51904C6.6302 3.51904 5.52014 4.6291 5.52014 5.99938C5.52014 7.36966 6.6302 8.47972 8.00048 8.47972Z"/>
  </g>
</svg>
```

**2. Add Destination Metadata**

Go to [`destinations/data/`](https://github.com/odigos-io/odigos/tree/main/destinations/data) directory and create a new YAML file.

* Example: `mydest.yaml`

> **Allowed properties**
>
> | Key | Example | Required | Description | Docs |
> | - | - | :-: | - | :-: |
> | metadata.type | `mydest` | ✅ | unique identifier for the destination | ✅ |
> | metadata.displayName | `My Destination` | ✅ | display name of the destination | ✅ |
> | metadata.category | `managed` | ✅ | one-of: `managed` `self hosted` | ✅ |
> | spec.image | `mydest.svg` | ✅ | logo of the destination | ✅ |
> | spec.signals.traces.supported | `true` | ✅ | is tracing supported? | ✅ |
> | spec.signals.metrics.supported | `true` | ✅ | is metrics supported? | ✅ |
> | spec.signals.logs.supported | `false` | ✅ | is logging supported? | ✅ |
> | spec.note.type | `Note` | ❌ | type of [callout box](https://mintlify.com/docs/content/components/callouts) to display one-of: `Note` `Info` `Warning` `Tip` `Check` | ✅ |
> | spec.note.content | `We handle the endpoint internally, so you don't need to provide it.` | ❌ | content of the callout box | ✅ |
> | spec.fields | - | ✅ | list of fields to be configured, [see specification here](#allowed-properties-for-spec-fields) | ✅ |

> **Allowed properties for spec.fields\[]**
>
> | Key | Example | Required | Description | Docs |
> | - | - | :-: | - | :-: |
> | name | `MYDEST_API_KEY` | ✅ | key name of the field | ✅ |
> | displayName | `API Key` | ✅ | display name of the field | ✅ |
> | secret | `true` | ❌ | is the field a secret? (e.g. password, token, etc.) | ❌ |
> | initialValue | - | ❌ | default value for the field | ✅ |
> | componentType | `input` | ✅ | one-of: `input` `textarea` `dropdown` `checkbox` `multiInput` `keyValuePairs` | ✅ |
> | componentProps.type | `password` | ❌ | the type of value (e.g. string, number, password, etc.) | ✅ |
> | componentProps.required | `true` | ❌ | is the field required? | ✅ |
> | componentProps.values | - | ❌ | list of options for dropdown | ❌ |
> | componentProps.placeholder | `glc_eyJvIj...` | ❌ | placeholder text for input | ✅ |
> | componentProps.tooltip | `Obtained from your "My Dest" account` | ❌ | tooltip text for input | ✅ |
> | renderCondition | `["MYDEST_URL", "!=", ""]` | ❌ | hide/show the field in the UI (form only) one-of: `[boolean]` `[string, operator, any]` where `string` is the key name of the dependency field, and `operator` is one-of: `!=` `==` `>=` `<=` `>` `<` | ❌ |
> | hideFromReadData | `["MYDEST_URL", "==", ""]` | ❌ | hide/show the field in the UI (read only) one-of: `[boolean]` `[string, operator, any]` where `string` is the key name of the dependency field, and `operator` is one-of: `!=` `==` `>=` `<=` `>` `<` | ❌ |
> | customReadDataLabels | - | ❌ | custom labels for the UI (read only) | ❌ |

```yaml mydest.yaml theme={null}
apiVersion: internal.odigos.io/v1beta1
kind: Destination
metadata:
  type: mydest
  displayName: My Destination
  category: managed
spec:
  image: mydest.svg
  signals:
    traces:
      supported: true
    metrics:
      supported: true
    logs:
      supported: false
  fields:
    - name: MYDEST_URL
      displayName: URL
      componentType: input
      componentProps:
        required: true
    - name: MYDEST_REGION
      displayName: Region
      componentType: input
      componentProps:
        required: true
    - name: MYDEST_API_KEY
      displayName: API Key
      componentType: input
      componentProps:
        type: password
        required: true
      secret: true
```

### Collector Configuration

Now that our new vendor can be persisted/loaded in the Kubernetes data store, we need to implement the collector configuration.

**1. Add Destination Type**

Go to [`common/dests.go`](https://github.com/odigos-io/odigos/tree/main/common/dests.go) and add your new destination to the `DestinationType` enum.

* Make sure the value is the same as the `type` property in `mydest.yaml`

**2. Add Collector Config**

Go to [`common/config`](https://github.com/odigos-io/odigos/tree/main/common/config) directory and create a new GO file.

* Example: `mydest.go`

> **Explain Config**
>
> * **DestType** - Returns the enum value of the destination added to [`common/dests.go`](https://github.com/odigos-io/odigos/tree/main/common/dests.go) in step 1.
> * **ModifyConfig** - Is called with the `dest` object which holds the data received from the UI and the `currentConfig` object. The `currentConfig` object contains the current configuration of the gateway collector. Modify this object to include the [OpenTelemetry configuration](https://opentelemetry.io/docs/collector/configuration/) needed by your destination. You can assume a basic configuration is already provided in the `currentConfig` object, for details see `getBasicConfig` method in [`common/config/root.go`](https://github.com/odigos-io/odigos/tree/main/common/config/root.go) file.
>   * Make sure to give any exporter or pipeline a unique name in order to avoid conflicts, use the conventions:
>     * `otlp/<dest-name>-<dest.GetID()>` for OTel gRPC exporters
>     * `otlphttp/<dest-name>-<dest.GetID()>` for OTel HTTP exporters
>     * `traces/<dest-name>-<dest.GetID()>` for traces pipelines
>     * `metrics/<dest-name>-<dest.GetID()>` for metrics pipelines
>     * `logs/<dest-name>-<dest.GetID()>` for logs pipelines
>   * You can use the utility methods `isTracingEnabled`, `isMetricsEnabled` and `isLoggingEnabled` to determine which signals are selected by the user for the destination, and configure the collector accordingly.

```go mydest.go theme={null}
package config

import (
  "fmt"

  "github.com/odigos-io/odigos/common"
)

const (
  regionKey = "MYDEST_REGION"
  urlKey    = "MYDEST_URL"
  apiKeyKey = "MYDEST_API_KEY"
)

type MyDest struct{}

func (m *MyDest) DestType() common.DestinationType {
  // DestinationType defined in common/dests.go
  return common.MyDestDestinationType
}

func (m *MyDest) ModifyConfig(dest ExporterConfigurer, currentConfig *Config) ([]string, error) {
  config := dest.GetConfig()
  // To make sure that the exporter and pipeline names are unique, we'll need to define a unique ID
  uniqueUri := "mydest-" + dest.GetID()

  region, exists := config[regionKey]
  if !exists {
    region = "us"
    return errorMissingKey(regionKey)
  }

  endpoint, exists := config[urlKey]
  if !exists {
    endpoint = fmt.Sprintf("https://%s.mydest.com:4317", region)
  }

  // Modify the exporter here
  exporterName := "otlp/" + uniqueUri
  exporterConfig := GenericMap{
    "endpoint": endpoint,
    "headers": GenericMap{
      // The API Key is a secret, so we can't expose it in the config
      "x-mydest-header-apikey": "${MYDEST_API_KEY}",
    },
  }

  currentConfig.Exporters[exporterName] = exporterConfig

  // Modify the pipelines here
  var pipelineNames []string

  if isTracingEnabled(dest) {
    pipeName := "traces/" + uniqueUri
    currentConfig.Service.Pipelines[pipeName] = Pipeline{
      Exporters: []string{exporterName},
    }
    pipelineNames = append(pipelineNames, pipeName)
  }

  if isMetricsEnabled(dest) {
    pipeName := "metrics/" + uniqueUri
    currentConfig.Service.Pipelines[pipeName] = Pipeline{
      Exporters: []string{exporterName},
    }
    pipelineNames = append(pipelineNames, pipeName)
  }

  if isLoggingEnabled(dest) {
    pipeName := "logs/" + uniqueUri
    currentConfig.Service.Pipelines[pipeName] = Pipeline{
      Exporters: []string{exporterName},
    }
    pipelineNames = append(pipelineNames, pipeName)
  }

  return pipelineNames, nil
}
```

**3. Add Available Config**

Go to [`common/config/root.go`](https://github.com/odigos-io/odigos/tree/main/common/config/root.go) and add the new destination to the `availableConfigers` list.

* Example: `&MyDest{}`

```go common/config/root.go theme={null}
var availableConfigers = []Configer{
  &MyDest{},
  /* List of existing destinations...  */
}
```

### Kubernetes-Specific Collector Configuration

If your destination configures Kubernetes settings, for example you have a config field that accepts a value that should be mounted
in the Collector Pod, create a `K8sConfiger` object in [`instrumentor/k8sconfig`](https://github.com/odigos-io/odigos/tree/main/instrumentor/k8sconfig).

This interface provides an additional function, `ModifyGatewayCollectorDeployment`, which accepts a Destination and Collector Deployment.
Your implementation can use this function to modify the Deployment as necessary based on the Destination config.

Note: It is the responsibility of your implementation to check for the validity of this config, such as checking for conflicts or values
that already exist.

The `K8sConfiger` interface also requires a `DestType()` function, similar to common Destinations. This function returns the type
of common Destination that this Kubernetes implementation relies on. This is used by the instrumentor to conditionally apply the `K8sConfiger`
only for the matching Destination.

When it is implemented, add your `K8sConfiger` implementation to the `availableConfigers` list in
[`instrumentor/k8sconfig/root.go`](https://github.com/odigos-io/odigos/tree/main/instrumentor/k8sconfig/root.go)

### Generating Documentation

**1. Install Python**

Make sure you have [Python installed](https://www.python.org/downloads/) on your system.

**2. Generate Documentation**

Go to [`docs`](https://github.com/odigos-io/odigos/tree/main/docs) directory, and run:

```bash theme={null}
python sync-dest-doc.py
# OR
python3 sync-dest-doc.py
```

**3. Edit Documentation**

The documentation has been generated in the [`docs/backends`](https://github.com/odigos-io/odigos/tree/main/docs/backends) directory. Feel free to add any additional content to the generated file.

* Edit only between the `!! START CUSTOM EDIT !!` and `!! END CUSTOM EDIT !!` comments
* Example: `mydest.mdx`

```mdx mydest.mdx theme={null}
{/*
    !! Do not remove this comment, this acts as a key indicator in `docs/sync-dest-doc.py` !!
    !! START CUSTOM EDIT !!
*/}

**Creating Account**<br />
Go to the **[🔗 website](https://odigos.io) > Account** and click **Sign Up**

**Obtaining Access Token**<br />
Go to **⚙️ > Access Tokens** and click **Create New**

{/*
    !! Do not remove this comment, this acts as a key indicator in `docs/sync-dest-doc.py` !!
    !! END CUSTOM EDIT !!
*/}
```

**4. Update README.md**

Go to [`README.md`](https://github.com/odigos-io/odigos/tree/main/README.md) and add your new destination to the list of destinations.

#### That's it! Now you can use your new destination in the UI and send data to it.

**Please submit a PR to the [odigos git repository](https://github.com/odigos-io/odigos).**


# actions.odigos.io/v1alpha1
Source: https://docs.odigos.io/oss/api-reference/actions.odigos.io.v1alpha1



## Resource Types

* [AddClusterInfo](#AddClusterInfo)
* [DeleteAttribute](#DeleteAttribute)
* [K8sAttributesResolver](#K8sAttributesResolver)
* [RenameAttribute](#RenameAttribute)

## `AddClusterInfo`&#x20;

AddClusterInfo is the Schema for the addclusterinfo odigos action API
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`AddClusterInfo`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`AddClusterInfoSpec`](#AddClusterInfoSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`AddClusterInfoStatus`](#AddClusterInfoStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttribute`&#x20;

DeleteAttribute is the Schema for the DeleteAttribute odigos action API
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`DeleteAttribute`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`DeleteAttributeSpec`](#DeleteAttributeSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`DeleteAttributeStatus`](#DeleteAttributeStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributesResolver`&#x20;

K8sAttributesResolver allows adding an action to collect k8s attributes.
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`K8sAttributesResolver`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`K8sAttributesSpec`](#K8sAttributesSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`K8sAttributesStatus`](#K8sAttributesStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttribute`&#x20;

RenameAttribute is the Schema for the RenameAttribute odigos action API
DEPRECATED: Use odigosv1.Action instead

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`RenameAttribute`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`RenameAttributeSpec`](#RenameAttributeSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`RenameAttributeStatus`](#RenameAttributeStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `AddClusterInfoConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `clusterAttributes` \[Required]
      </td>

      <td>
        [`[]OtelAttributeWithValue`](#OtelAttributeWithValue)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `overwriteExistingValues` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `AddClusterInfoSpec`&#x20;

**Appears in:**

* [AddClusterInfo](#AddClusterInfo)

AddClusterInfoSpec defines the desired state of AddClusterInfo action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `clusterAttributes` \[Required]
      </td>

      <td>
        [`[]OtelAttributeWithValue`](#OtelAttributeWithValue)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `overwriteExistingValues` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `AddClusterInfoStatus`&#x20;

**Appears in:**

* [AddClusterInfo](#AddClusterInfo)

AddClusterInfoStatus defines the observed state of AddClusterInfo action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a addclusterinfo's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttributeConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `attributeNamesToDelete` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttributeSpec`&#x20;

**Appears in:**

* [DeleteAttribute](#DeleteAttribute)

DeleteAttributeSpec defines the desired state of DeleteAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `attributeNamesToDelete` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DeleteAttributeStatus`&#x20;

**Appears in:**

* [DeleteAttribute](#DeleteAttribute)

DeleteAttributeStatus defines the observed state of DeleteAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a DeleteAttribute's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `K8sAnnotationAttribute`&#x20;

**Appears in:**

* [K8sAttributesConfig](#K8sAttributesConfig)
* [K8sAttributesSpec](#K8sAttributesSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `annotationKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The annotation name to be extracted.
        e.g. "kubectl.kubernetes.io/restartedAt"
      </td>
    </tr>

    <tr>
      <td>
        `attributeKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The attribute key to be used for the resource attribute created from the annotation.
        e.g. "kubectl.kubernetes.restartedAt"
      </td>
    </tr>

    <tr>
      <td>
        `from` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The source of the annotation.
        e.g. "pod" or "namespace"
        Deprecated: Use FromSources instead for specifying multiple sources with precedence.
      </td>
    </tr>

    <tr>
      <td>
        `fromSources` \[Required]
      </td>

      <td>
        [`[]K8sAttributeSource`](#K8sAttributeSource)
      </td>

      <td>
        The sources from which to extract the annotation, in order of precedence (most specific first).
        When multiple sources are specified, the most specific source (e.g., pod) takes precedence
        over less specific sources (e.g., namespace).
        If an annotation exists in multiple sources, the value from the most specific source will be used.
        Supported sources: "pod", "namespace", "node"
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributeSource`&#x20;

(Alias of `string`)

**Appears in:**

* [K8sAnnotationAttribute](#K8sAnnotationAttribute)
* [K8sLabelAttribute](#K8sLabelAttribute)

## `K8sAttributesConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `collectContainerAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following container related attributes:
        k8s.container.name
        container.id
        container.image.name
        container.image.tag
      </td>
    </tr>

    <tr>
      <td>
        `collectReplicaSetAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        collect replicaset related attributes (when relevant, e.g. for deployments):
        k8s.replicaset.name
        if CollectWorkloadUID is set, also collect:
        k8s.replicaset.uid
        DEPRECATED: ReplicaSet attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectWorkloadUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following workload UID attributes:
        k8s.deployment.uid
        k8s.daemonset.uid
        k8s.statefulset.uid
        DEPRECATED: Workload UID attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectClusterUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the k8s.cluster.uid attribute, which is set to the uid of the namespace "kube-system"
      </td>
    </tr>

    <tr>
      <td>
        `labelsAttributes`
      </td>

      <td>
        [`[]K8sLabelAttribute`](#K8sLabelAttribute)
      </td>

      <td>
        list of labels to be extracted from the pod, and the attribute key to be used for the resource attribute created from each label.
      </td>
    </tr>

    <tr>
      <td>
        `annotationsAttributes`
      </td>

      <td>
        [`[]K8sAnnotationAttribute`](#K8sAnnotationAttribute)
      </td>

      <td>
        list of annotations to be extracted from the pod, and the attribute key to be used for the resource attribute created from each annotation.
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributesSpec`&#x20;

**Appears in:**

* [K8sAttributesResolver](#K8sAttributesResolver)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `collectContainerAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following container related attributes:
        k8s.container.name
        container.id
        container.image.name
        container.image.tag
      </td>
    </tr>

    <tr>
      <td>
        `collectReplicaSetAttributes` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        collect replicaset related attributes (when relevant, e.g. for deployments):
        k8s.replicaset.name
        if CollectWorkloadUID is set, also collect:
        k8s.replicaset.uid
        DEPRECATED: ReplicaSet attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectWorkloadUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the following workload UID attributes:
        k8s.deployment.uid
        k8s.daemonset.uid
        k8s.statefulset.uid
        DEPRECATED: Workload UID attributes are now collected by default during instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `collectClusterUID` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Collect the k8s.cluster.uid attribute, which is set to the uid of the namespace "kube-system"
      </td>
    </tr>

    <tr>
      <td>
        `labelsAttributes`
      </td>

      <td>
        [`[]K8sLabelAttribute`](#K8sLabelAttribute)
      </td>

      <td>
        list of labels to be extracted from the pod, and the attribute key to be used for the resource attribute created from each label.
      </td>
    </tr>

    <tr>
      <td>
        `annotationsAttributes`
      </td>

      <td>
        [`[]K8sAnnotationAttribute`](#K8sAnnotationAttribute)
      </td>

      <td>
        list of annotations to be extracted from the pod, and the attribute key to be used for the resource attribute created from each annotation.
      </td>
    </tr>
  </tbody>
</table>

## `K8sAttributesStatus`&#x20;

**Appears in:**

* [K8sAttributesResolver](#K8sAttributesResolver)

K8sAttributesStatus defines the observed state of K8sAttributes action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a k8sattributes' current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `K8sLabelAttribute`&#x20;

**Appears in:**

* [K8sAttributesConfig](#K8sAttributesConfig)
* [K8sAttributesSpec](#K8sAttributesSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `labelKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The label name to be extracted.
        e.g. "app.kubernetes.io/name"
      </td>
    </tr>

    <tr>
      <td>
        `attributeKey` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The attribute key to be used for the resource attribute created from the label.
        e.g. "app.kubernetes.name"
      </td>
    </tr>

    <tr>
      <td>
        `from` \[Required]
      </td>

      <td>
        [`K8sAttributeSource`](#K8sAttributeSource)
      </td>

      <td>
        The source of the label.
        e.g. "pod" or "namespace"
        Deprecated: Use FromSources instead for specifying multiple sources with precedence.
      </td>
    </tr>

    <tr>
      <td>
        `fromSources` \[Required]
      </td>

      <td>
        [`[]K8sAttributeSource`](#K8sAttributeSource)
      </td>

      <td>
        The sources from which to extract the label, in order of precedence (most specific first).
        When multiple sources are specified, the most specific source (e.g., pod) takes precedence
        over less specific sources (e.g., namespace).
        If a label exists in multiple sources, the value from the most specific source will be used.
        Supported sources: "pod", "namespace", "node"
      </td>
    </tr>
  </tbody>
</table>

## `OtelAttributeWithValue`&#x20;

**Appears in:**

* [AddClusterInfoConfig](#AddClusterInfoConfig)
* [AddClusterInfoSpec](#AddClusterInfoSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `attributeName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        the name of the attribute to insert
      </td>
    </tr>

    <tr>
      <td>
        `attributeStringValue` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        if the value is a string, this field should be used.
        empty string is a valid value
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttributeConfig`&#x20;

**Appears in:**

* [ActionSpec](#odigos-io-v1alpha1-ActionSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `renames` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttributeSpec`&#x20;

**Appears in:**

* [RenameAttribute](#RenameAttribute)

RenameAttributeSpec defines the desired state of RenameAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `renames` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `RenameAttributeStatus`&#x20;

**Appears in:**

* [RenameAttribute](#RenameAttribute)

RenameAttributeStatus defines the observed state of RenameAttribute action

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a RenameAttribute's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>


# odigos.io/v1alpha1
Source: https://docs.odigos.io/oss/api-reference/odigos.io.v1alpha1



Package v1alpha1 is the v1alpha1 version of the API.

## Resource Types

* [Action](#odigos-io-v1alpha1-Action)
* [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)
* [Destination](#odigos-io-v1alpha1-Destination)
* [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)
* [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)
* [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)
* [Processor](#odigos-io-v1alpha1-Processor)
* [Recommendation](#odigos-io-v1alpha1-Recommendation)
* [Sampling](#odigos-io-v1alpha1-Sampling)
* [Source](#odigos-io-v1alpha1-Source)

## `Action`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Action`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`ActionSpec`](#odigos-io-v1alpha1-ActionSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`ActionStatus`](#odigos-io-v1alpha1-ActionStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroup`&#x20;

CollectorsGroup is the Schema for the collectors API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`CollectorsGroup`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`CollectorsGroupSpec`](#odigos-io-v1alpha1-CollectorsGroupSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`CollectorsGroupStatus`](#odigos-io-v1alpha1-CollectorsGroupStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Destination`&#x20;

Destination is the Schema for the destinations API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Destination`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`DestinationSpec`](#odigos-io-v1alpha1-DestinationSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`DestinationStatus`](#odigos-io-v1alpha1-DestinationStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationConfig`&#x20;

InstrumentationConfig is the Schema for the instrumentationconfig API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`InstrumentationConfig`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`InstrumentationConfigSpec`](#odigos-io-v1alpha1-InstrumentationConfigSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`InstrumentationConfigStatus`](#odigos-io-v1alpha1-InstrumentationConfigStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationInstance`&#x20;

InstrumentationInstance is the Schema for the InstrumentationInstances API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`InstrumentationInstance`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`InstrumentationInstanceSpec`](#odigos-io-v1alpha1-InstrumentationInstanceSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`InstrumentationInstanceStatus`](#odigos-io-v1alpha1-InstrumentationInstanceStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationRule`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`InstrumentationRule`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`InstrumentationRuleSpec`](#odigos-io-v1alpha1-InstrumentationRuleSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`InstrumentationRuleStatus`](#odigos-io-v1alpha1-InstrumentationRuleStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Processor`&#x20;

Processor is the Schema for an Opentelemetry Collector Processor that is added to Odigos pipeline

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Processor`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`ProcessorSpec`](#odigos-io-v1alpha1-ProcessorSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`ProcessorStatus`](#odigos-io-v1alpha1-ProcessorStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Recommendation`&#x20;

Recommendation is the Schema for enabling a catalog recommendation in the cluster.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Recommendation`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`RecommendationSpec`](#odigos-io-v1alpha1-RecommendationSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`RecommendationStatus`](#odigos-io-v1alpha1-RecommendationStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Sampling`&#x20;

Sampling is the Schema for the sampling rules API.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Sampling`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`SamplingSpec`](#odigos-io-v1alpha1-SamplingSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`SamplingStatus`](#odigos-io-v1alpha1-SamplingStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `Source`&#x20;

Source configures an application for auto-instrumentation.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr><td>`apiVersion`</td><td>string</td><td>`odigos.io/v1alpha1`</td></tr>
    <tr><td>`kind`</td><td>string</td><td>`Source`</td></tr>

    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`SourceSpec`](#odigos-io-v1alpha1-SourceSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`SourceStatus`](#odigos-io-v1alpha1-SourceStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `ActionSpec`&#x20;

**Appears in:**

* [Action](#odigos-io-v1alpha1-Action)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `actionName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        Allows you to attach a meaningful name to the action for convenience. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        A free-form text field that allows you to attach notes regarding the action for convenience. For example: why it was added. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        A boolean field allowing to temporarily disable the action, but keep it around for future use
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        Which signals should this action operate on.
      </td>
    </tr>

    <tr>
      <td>
        `addClusterInfo` \[Required]
      </td>

      <td>
        `actions/v1alpha1.AddClusterInfoConfig`
      </td>

      <td>
        AddClusterInfo is the config for the AddClusterInfo Action.
      </td>
    </tr>

    <tr>
      <td>
        `deleteAttribute` \[Required]
      </td>

      <td>
        `actions/v1alpha1.DeleteAttributeConfig`
      </td>

      <td>
        DeleteAttribute is the config for the DeleteAttribute Action.
      </td>
    </tr>

    <tr>
      <td>
        `renameAttribute` \[Required]
      </td>

      <td>
        `actions/v1alpha1.RenameAttributeConfig`
      </td>

      <td>
        RenameAttribute is the config for the RenameAttribute Action.
      </td>
    </tr>

    <tr>
      <td>
        `piiMasking` \[Required]
      </td>

      <td>
        `actions.PiiMaskingConfig`
      </td>

      <td>
        PiiMasking is the config for the PiiMasking Action.
      </td>
    </tr>

    <tr>
      <td>
        `k8sAttributes` \[Required]
      </td>

      <td>
        `actions/v1alpha1.K8sAttributesConfig`
      </td>

      <td>
        K8sAttributes is the config for the K8sAttributes Action.
      </td>
    </tr>

    <tr>
      <td>
        `urlTemplatization` \[Required]
      </td>

      <td>
        `actions.URLTemplatizationConfig`
      </td>

      <td>
        URLTemplatization is the config for the URLTemplatization Action.
      </td>
    </tr>

    <tr>
      <td>
        `spanRenamer` \[Required]
      </td>

      <td>
        `actions.SpanRenamerConfig`
      </td>

      <td>
        SpanRenamer is the config for the SpanRenamer Action.
      </td>
    </tr>

    <tr>
      <td>
        `extractAttribute` \[Required]
      </td>

      <td>
        `actions.ExtractAttributeConfig`
      </td>

      <td>
        ExtractAttribute is the config for the ExtractAttribute Action.
      </td>
    </tr>

    <tr>
      <td>
        `dbQueryTemplatization` \[Required]
      </td>

      <td>
        `actions.DbQueryTemplatizationConfig`
      </td>

      <td>
        DbQueryTemplatization is the config for the DbQueryTemplatization Action.
      </td>
    </tr>

    <tr>
      <td>
        `inferDbAttributes` \[Required]
      </td>

      <td>
        `actions.InferDbAttributesConfig`
      </td>

      <td>
        InferDbAttributes is the config for the InferDbAttributes Action.
      </td>
    </tr>
  </tbody>
</table>

## `ActionStatus`&#x20;

**Appears in:**

* [Action](#odigos-io-v1alpha1-Action)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a action's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `AgentEnabledReason`&#x20;

(Alias of `string`)

**Appears in:**

* [ContainerAgentConfig](#odigos-io-v1alpha1-ContainerAgentConfig)

## `AgentsTelemetrySettings`&#x20;

**Appears in:**

* [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

## `Attribute`&#x20;

**Appears in:**

* [InstrumentationInstanceStatus](#odigos-io-v1alpha1-InstrumentationInstanceStatus)
* [InstrumentationLibraryStatus](#odigos-io-v1alpha1-InstrumentationLibraryStatus)

Attribute is a key-value pair that describes a component or instrumentation

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `key` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `value` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupMetricsCollectionSettings`&#x20;

**Appears in:**

* [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `spanMetrics` \[Required]
      </td>

      <td>
        [`common.MetricsSourceSpanMetricsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MetricsSourceSpanMetricsConfiguration)
      </td>

      <td>
        if not nil for node collector, it means span to metrics is enabled,
        and the node collector should set it up in the pipeline.
        span to metrics is the ability to calculate metrics like http requests/errors/duration etc
        from the individual spans recorded for relevant operation.
      </td>
    </tr>

    <tr>
      <td>
        `hostMetrics` \[Required]
      </td>

      <td>
        [`common.MetricsSourceHostMetricsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MetricsSourceHostMetricsConfiguration)
      </td>

      <td>
        if not nil for node collector, it means host metrics is enabled,
        and the opentelemetry collector "hostmetrics" receiver should be included in the pipeline.
        host metrics are metrics that are collected from the host node,
        such as cpu, memory, disk, network, etc.
      </td>
    </tr>

    <tr>
      <td>
        `kubeletStats` \[Required]
      </td>

      <td>
        [`common.MetricsSourceKubeletStatsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MetricsSourceKubeletStatsConfiguration)
      </td>

      <td>
        if not nil for node collector, it means kubelet stats is enabled,
        and the opentelemetry collector "kubeletstats" receiver should be included in the pipeline.
        kubelet stats are metrics that are collected from the kubelet point of view,
        such as cpu, memory, disk, network, per pod, node and more.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraph` \[Required]
      </td>

      <td>
        [`ServiceGraphSettings`](#odigos-io-v1alpha1-ServiceGraphSettings)
      </td>

      <td>
        if not nil for cluster collector, it means service graph is enabled,
        and metrics for the "connectivity" between services should be calculated
        to be exported to metrics destinations.
      </td>
    </tr>

    <tr>
      <td>
        `odigosOwnMetrics` \[Required]
      </td>

      <td>
        [`OdigosOwnMetricsSettings`](#odigos-io-v1alpha1-OdigosOwnMetricsSettings)
      </td>

      <td>
        if not nil for node collector, it means that some metric destinations are
        intresseted in collecting metrics about: odigos, the collected data, and the pipeline itself.
        this allows for users to monitor and operate odigos within their existing system,
        create dashboards, alerting, and more.
      </td>
    </tr>

    <tr>
      <td>
        `agentsTelemetry` \[Required]
      </td>

      <td>
        [`AgentsTelemetrySettings`](#odigos-io-v1alpha1-AgentsTelemetrySettings)
      </td>

      <td>
        this part controls the metrics which are received from agents in the otlp receiver.
        it is generally enabled when we want to record metrics, and listed here for completeness.
        any "otlp receiver" specific settings can go here
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupResourcesSettings`&#x20;

**Appears in:**

* [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)

The raw values to control the collectors group resources and behavior.
any defaulting, validations and calculations should be done in the controllers
that create this CR.
Values will be used as is without any further processing.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `minReplicas` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        Minumum + Maximum number of replicas for the collector - these relevant only for gateway.
      </td>
    </tr>

    <tr>
      <td>
        `maxReplicas` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `memoryRequestMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        MemoryRequestMiB is the memory resource request to be used on the pod template.
        it will be embedded in the as a resource request of the form `memory: <value>Mi`
      </td>
    </tr>

    <tr>
      <td>
        `memoryLimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        This option sets the limit on the memory usage of the collector.
        since the memory limiter mechanism is heuristic, and operates on fixed intervals,
        while it cannot fully prevent OOMs, it can help in reducing the chances of OOMs in edge cases.
        the settings should prevent the collector from exceeding the memory request,
        so one can set this to the same value as the memory request or higher to allow for some buffer for bursts.
      </td>
    </tr>

    <tr>
      <td>
        `cpuRequestMillicores` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        CPU resource request to be used on the pod template.
        it will be embedded in the as a resource request of the form `cpu: <value>m`
      </td>
    </tr>

    <tr>
      <td>
        `cpuLimitMillicores` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        CPU resource limit to be used on the pod template.
        it will be embedded in the as a resource limit of the form `cpu: <value>m`
      </td>
    </tr>

    <tr>
      <td>
        `memoryLimiterLimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        this parameter sets the "limit\_mib" parameter in the memory limiter configuration for the collector.
        it is the hard limit after which a force garbage collection will be performed.
        this value will end up comparing against the go runtime reported heap Alloc value.
        According to the memory limiter docs:

        > Note that typically the total memory usage of process will be about 50MiB higher than this value
        > a test from nov 2024 showed that fresh odigos collector with no traffic takes 38MiB,
        > thus the 50MiB is a good value to start with.
      </td>
    </tr>

    <tr>
      <td>
        `memoryLimiterSpikeLimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        this parameter sets the "spike\_limit\_mib" parameter in the memory limiter configuration for the collector memory limiter.
        note that this is not the processor soft limit itself, but the diff in Mib between the hard limit and the soft limit.
        according to the memory limiter docs, it is recommended to set this to 20% of the hard limit.
        changing this value allows trade-offs between memory usage and resiliency to bursts.
      </td>
    </tr>

    <tr>
      <td>
        `gomemlimitMiB` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        the GOMEMLIMIT environment variable value for the collector pod.
        this is when go runtime will start garbage collection.
        it is recommended to be set to 80% of the hard limit of the memory limiter.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupRole`&#x20;

(Alias of `string`)

**Appears in:**

* [CollectorsGroupSpec](#odigos-io-v1alpha1-CollectorsGroupSpec)
* [ProcessorSpec](#odigos-io-v1alpha1-ProcessorSpec)

## `CollectorsGroupSpec`&#x20;

**Appears in:**

* [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)

CollectorsGroupSpec defines the desired state of Collector

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `role` \[Required]
      </td>

      <td>
        [`CollectorsGroupRole`](#odigos-io-v1alpha1-CollectorsGroupRole)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `collectorOwnMetricsPort` \[Required]
      </td>

      <td>
        `int32`
      </td>

      <td>
        The port to use for exposing the collector's own metrics as a prometheus endpoint.
        This can be used to resolve conflicting ports when a collector is using the host network.
      </td>
    </tr>

    <tr>
      <td>
        `resourcesSettings` \[Required]
      </td>

      <td>
        [`CollectorsGroupResourcesSettings`](#odigos-io-v1alpha1-CollectorsGroupResourcesSettings)
      </td>

      <td>
        Resources \[memory/cpu] settings for the collectors group.
        these settings are used to protect the collectors instances from:

        <ul>
          <li>running out of memory and being killed by the k8s OOM killer</li>
          <li>consuming all available memory on the node which can lead to node instability</li>
          <li>pushing back pressure to the instrumented applications</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphDisabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        ServiceGraphEnabled is a feature that allows you to visualize the service graph of your application.
        It is enabled by default and can be disabled by setting the enabled flag to false.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphExtraDimensions` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        ServiceGraphExtraDimensions are additional span attribute names to include as dimensions
        in the service graph metrics, on top of the default service.name dimension.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphVirtualNodePeerAttributes` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        ServiceGraphVirtualNodePeerAttributes is an ordered list of span attributes used to identify
        uninstrumented (virtual) nodes in the service graph.
        The connector picks the first attribute that has a value on the span.
        When nil/empty, the connector uses its built-in defaults: \[peer.service, db.name, db.system].
      </td>
    </tr>

    <tr>
      <td>
        `enableDataCompression` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Deprecated - use OtlpExporterConfiguration instead.
        EnableDataCompression is a feature that allows you to enable data compression before sending data to the Gateway collector.
        It is disabled by default and can be enabled by setting the enabled flag to true.
      </td>
    </tr>

    <tr>
      <td>
        `otlpExporterConfiguration` \[Required]
      </td>

      <td>
        [`common.OtlpExporterConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#OtlpExporterConfiguration)
      </td>

      <td>
        OtlpExporterConfiguration is the configuration for the OTLP exporter from node collector to cluster gateway collector.
      </td>
    </tr>

    <tr>
      <td>
        `resourceDetectors` \[Required]
      </td>

      <td>
        [`common.ResourceDetectorsConfiguration`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ResourceDetectorsConfiguration)
      </td>

      <td>
        ResourceDetectors controls which OpenTelemetry resource detectors are enabled
        on the node collector's resourcedetection processor.
      </td>
    </tr>

    <tr>
      <td>
        `clusterMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        ClusterMetricsEnabled is a feature that allows you to enable the cluster metrics.
        It is disabled by default and can be enabled by setting the enabled flag to true.
      </td>
    </tr>

    <tr>
      <td>
        `httpsProxyAddress` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        for destinations that uses https for exporting data, this value can be used to set the address for an https proxy.
        when unset or empty, no proxy will be used.
      </td>
    </tr>

    <tr>
      <td>
        `metrics` \[Required]
      </td>

      <td>
        [`CollectorsGroupMetricsCollectionSettings`](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)
      </td>

      <td>
        configuration for metrics handling in this collectors group
        if metric collection is disabled, this will be nil
        the content is populated only if relevant to this collectors group
        it's being calculated based on active destinations and their settings,
        and global settings provided in the odigos configuration or instrumentation rules.
        it allows for the collector group reconciler to be simplified,
        and for visibility into the aggregated settings being used to derive configurations deployments and rollouts.
      </td>
    </tr>

    <tr>
      <td>
        `nodeSelector` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        Node selector for the collectors group deployment.
        Use this to force the gateway to run only on nodes with specific labels.
        This is a hard requirement: the pod will be scheduled ONLY on nodes that match all labels.
        If no matching nodes exist, the pod will remain Pending.
      </td>
    </tr>

    <tr>
      <td>
        `deploymentName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        Deployment name for the collectors group deployment.
        Only relevant for cluster gateway collector.
      </td>
    </tr>

    <tr>
      <td>
        `tailSampling` \[Required]
      </td>

      <td>
        `common/api/sampling.TailSamplingConfiguration`
      </td>

      <td>
        Sampling holds the sampling configuration derived from the OdigosConfiguration.
        Currently this is only relevant for the cluster gateway collector.
      </td>
    </tr>

    <tr>
      <td>
        `samplingDryRun` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Set to true to enable dry run mode for sampling.
        When enabled, odigos will invoke the sampling logic, but will not drop any traces.
        This is useful while evaluating sampling rules to check for effectiveness and correctness
        before committing to any changes that might lose data.
      </td>
    </tr>

    <tr>
      <td>
        `spanSamplingAttributes` \[Required]
      </td>

      <td>
        `common/api/sampling.SpanSamplingAttributesConfiguration`
      </td>

      <td>
        Controls whether spans are enhanced with sampling attributes (e.g. category and decisions).
        Capturing these attributes gives visibility into sampling decision-making and effective
        sampling percentages when viewing traces or querying the database with tools.
      </td>
    </tr>
  </tbody>
</table>

## `CollectorsGroupStatus`&#x20;

**Appears in:**

* [CollectorsGroup](#odigos-io-v1alpha1-CollectorsGroup)

CollectorsGroupStatus defines the observed state of Collector

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `ready` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `receiverSignals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        Receiver Signals are the signals (trace, metrics, logs) that the collector has setup
        an otlp receiver for, thus it can accept data from an upstream component.
        this is used to determine if a workload should export each signal or not.
        this list is calculated based on the odigos destinations that were configured
      </td>
    </tr>

    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a collectorsroup's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `ContainerAgentConfig`&#x20;

**Appears in:**

* [InstrumentationConfigSpec](#odigos-io-v1alpha1-InstrumentationConfigSpec)

ContainerAgentConfig is a configuration for a specific container in a workload.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the container to which this configuration applies.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        boolean flag to indicate if the agent should be enabled for this container.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnabledReason` \[Required]
      </td>

      <td>
        [`AgentEnabledReason`](#odigos-io-v1alpha1-AgentEnabledReason)
      </td>

      <td>
        An enum reason for the agent injection decision.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnabledMessage` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        free text message to provide more information about the instrumentation decision.
        can be left empty if reason is self-explanatory.
      </td>
    </tr>

    <tr>
      <td>
        `podManifestInjectionOptional` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        set to true if the agent used in this container can be injected without pod manifest changes.
      </td>
    </tr>

    <tr>
      <td>
        `otelDistroName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the otel distribution to use for this container.
        if the name is empty, this container should not be instrumented.
      </td>
    </tr>

    <tr>
      <td>
        `distroParams` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        Additional parameters to the distro that controls how it's being applied.
        Keys are parameter names (like "libc") and values are the value to use for that parameter (glibc / musl)
      </td>
    </tr>

    <tr>
      <td>
        `envInjectionMethod` \[Required]
      </td>

      <td>
        [`common.EnvInjectionDecision`](https://pkg.go.dev/github.com/odigos-io/odigos/common#EnvInjectionDecision)
      </td>

      <td>
        What method to use for injecting the agent environment variables (just those covered by the loader (PYTHONPATH, JAVA\_TOOLS\_OPTIONS, NODE\_OPTIONS))
        Can be either "loader" or "pod-manifest".
        The injection should still check the actual values in the container manifest before injecting.
        Nil means that this container should not have env injection (agent should not be injected, or distro does not specify "loader" injection envs).
      </td>
    </tr>

    <tr>
      <td>
        `traces` \[Required]
      </td>

      <td>
        `common/api/agentsignalconfig.AgentTracesConfig`
      </td>

      <td>
        Each enabled signal must be set with a non-nil value (even if the config content is empty).
        nil means that the signal is disabled and should not be instrumented/collected by the agent.
      </td>
    </tr>

    <tr>
      <td>
        `metrics` \[Required]
      </td>

      <td>
        `common/api/agentsignalconfig.AgentMetricsConfig`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `logs` \[Required]
      </td>

      <td>
        `common/api/agentsignalconfig.AgentLogsConfig`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `agentDiagnostics` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.AgentDiagnostics`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#AgentDiagnostics)
      </td>

      <td>
        Configure the log level for the agent itselg
      </td>
    </tr>
  </tbody>
</table>

## `ContainerOverride`&#x20;

**Appears in:**

* [InstrumentationConfigSpec](#odigos-io-v1alpha1-InstrumentationConfigSpec)
* [SourceSpec](#odigos-io-v1alpha1-SourceSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the container to override.
      </td>
    </tr>

    <tr>
      <td>
        `runtimeInfo` \[Required]
      </td>

      <td>
        [`RuntimeDetailsByContainer`](#odigos-io-v1alpha1-RuntimeDetailsByContainer)
      </td>

      <td>
        RuntimeInfo to use for agent enabling.
        If set for a container, the automatic detection will not be used for this container,
        and the distro to use will be calculated based on this value.
      </td>
    </tr>

    <tr>
      <td>
        `otelDistroName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        select one specific otel distro to use for this container.
        the selected distro must match the programming language and any constraints of the distro.
        if the programmin language is incorrect, the runtime info can be overridden to match
      </td>
    </tr>

    <tr>
      <td>
        `allowConcurrentAgents` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Odigos by default will attempt to detect if another agent is already running in the container,
        and if so, will avoid adding odigos agent to the container.
        This option allows to override this setting for a specific container in a source.
        Can be used for selective setting of which containers should run odigos concurrently with other agents.
        empty means take odigos global config for this setting.
        when not empty, the value will override the global config for this container
      </td>
    </tr>
  </tbody>
</table>

## `CostReductionRule`&#x20;

**Appears in:**

* [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user provided name, for easier identification and reference.
        use short and descriptive name, like "health check", "always keep /transaction in payment service", etc.
        odigos does not use or assume any meaning from this field,
        but it is written as metric attribute, and stored as span attribute on participating spans.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if set to true, the rule will be disabled,
        e.g. will not be taken into account for any sampling decisions.
        disabled rules still participate in metrics calculations,
        allowing enhanced tools and data for troubleshooting and sampling maintenance.
      </td>
    </tr>

    <tr>
      <td>
        `sourceScopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        limit this rule to specific sources (by name, namespace, language, etc.)
        an empty list will match any source.
        if multiple items are set, the operation match if any one matches
        this relates to the "ResourceAttributes" part of a span.
      </td>
    </tr>

    <tr>
      <td>
        `operation` \[Required]
      </td>

      <td>
        `common/api/sampling.TailSamplingOperationMatcher`
      </td>

      <td>
        limit this rule to specific operations.
        for example: specific endpoint or kafka topic.
        this field is optional, and if not set, the rule will be applied to all operations.
      </td>
    </tr>

    <tr>
      <td>
        `percentageAtMost` \[Required]
      </td>

      <td>
        `float64`
      </td>

      <td>
        sampling percentage for cost reduction.
        this field is required.
        the final sampling percentage for traces that match this rule
        will be the highest possible, but at most this value.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        optional free-form text field that allows you to attach notes
        for future context and maintenance.
        users can write why this rule was added, observations, document considerations, etc.
      </td>
    </tr>
  </tbody>
</table>

## `DestinationMetricsSettings`&#x20;

**Appears in:**

* [DestinationSpec](#odigos-io-v1alpha1-DestinationSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `spanMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        following settings enables/disables specific metric sources
        true - collect and send this metric source
        false - do not collect and send this metric source
        nil - use the default setting (from destination manifest, or cluster global setting)
      </td>
    </tr>

    <tr>
      <td>
        `hostMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `kubeletStatsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `serviceGraphEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `odigosOwnMetricsEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `agentsTelemetryEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `DestinationSpec`&#x20;

**Appears in:**

* [Destination](#odigos-io-v1alpha1-Destination)

DestinationSpec defines the desired state of Destination

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        [`common.DestinationType`](https://pkg.go.dev/github.com/odigos-io/odigos/common#DestinationType)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `destinationName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `data` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `secretRef` \[Required]
      </td>

      <td>
        [`core/v1.LocalObjectReference`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#localobjectreference-v1-core)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `metricsSettings`
      </td>

      <td>
        [`DestinationMetricsSettings`](#odigos-io-v1alpha1-DestinationMetricsSettings)
      </td>

      <td>
        MetricsSettings defines the metrics settings for this destination.
        ignored if destination does not support metrics or metrics are not enabled in signals
      </td>
    </tr>

    <tr>
      <td>
        `sourceSelector`
      </td>

      <td>
        [`SourceSelector`](#odigos-io-v1alpha1-SourceSelector)
      </td>

      <td>
        SourceSelector defines which sources can send data to this destination.
        If not specified, defaults to "all".
      </td>
    </tr>
  </tbody>
</table>

## `DestinationStatus`&#x20;

**Appears in:**

* [Destination](#odigos-io-v1alpha1-Destination)

DestinationStatus defines the observed state of Destination

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a destination's current state.
      </td>
    </tr>
  </tbody>
</table>

## `EnvVar`&#x20;

**Appears in:**

* [RuntimeDetailsByContainer](#odigos-io-v1alpha1-RuntimeDetailsByContainer)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `value` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `HighlyRelevantOperation`&#x20;

**Appears in:**

* [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

define operations (spans) with high observability value.
if found anywhere in the trace, the entire trace will be kept
regaradless of any cost reduction rules.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user provided name, for easier identification and reference.
        use short and descriptive name, like "health check", "always keep /transaction in payment service", etc.
        odigos does not use or assume any meaning from this field,
        but it is written as metric attribute, and stored as span attribute on participating spans.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if set to true, the rule will be disabled,
        e.g. will not be taken into account for any sampling decisions.
        disabled rules still participate in metrics calculations,
        allowing enhanced tools and data for troubleshooting and sampling maintenance.
      </td>
    </tr>

    <tr>
      <td>
        `sourceScopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        limit the operation to specific sources.
        an empty list will match any source.
        if multiple items are set, the operation match if any one matches
        this relates to the "ResourceAttributes" part of a span.
      </td>
    </tr>

    <tr>
      <td>
        `error` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if "Error" is set to true, only spans with SpanStatus set to "Error" are considered
      </td>
    </tr>

    <tr>
      <td>
        `durationAtLeastMs` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        if Duration is set, only operations with duration in milli seconds larger then this value are considered
      </td>
    </tr>

    <tr>
      <td>
        `operation` \[Required]
      </td>

      <td>
        `common/api/sampling.TailSamplingOperationMatcher`
      </td>

      <td>
        optionally, limit this rule to specific operations.
        for example: specific endpoint or kafka topic.
        this field is optional, and if not set, the rule will be applied to all operations.
      </td>
    </tr>

    <tr>
      <td>
        `percentageAtLeast` \[Required]
      </td>

      <td>
        `float64`
      </td>

      <td>
        traces that contains this operation will be sampled by at least this percentage.
        if unset, 100% of such the traces will be sampled.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        optional free-form text field that allows you to attach notes
        for future context and maintenance.
        users can write why this rule was added, observations, document considerations, etc.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationConfigSpec`&#x20;

**Appears in:**

* [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)

Config for the OpenTelemeetry SDKs that should be applied to a workload.
The workload is identified by the owner reference

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `serviceName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        the service.name property is used to populate the `service.name` resource attribute in the telemetry generated by this workload
      </td>
    </tr>

    <tr>
      <td>
        `agentInjectionEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        determines if odigos should inject agents to pods of this workload.
      </td>
    </tr>

    <tr>
      <td>
        `podManifestInjectionOptional` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        true if all containers with agent injection enabled in this source
        can be enabled without pod manifest changes.
        if false, at least one container requires pod manifest changes to enable agent injection.
      </td>
    </tr>

    <tr>
      <td>
        `containers` \[Required]
      </td>

      <td>
        [`[]ContainerAgentConfig`](#odigos-io-v1alpha1-ContainerAgentConfig)
      </td>

      <td>
        configuration for each instrumented container in the workload
      </td>
    </tr>

    <tr>
      <td>
        `workloadCollectorConfig` \[Required]
      </td>

      <td>
        `[]common/api.ContainerCollectorConfig`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `containersOverrides` \[Required]
      </td>

      <td>
        [`[]ContainerOverride`](#odigos-io-v1alpha1-ContainerOverride)
      </td>

      <td>
        will always list all containers of this workload by name,
        and override data in case it is configured on the source.
        this peoperty can be used to know all container names in the workload, even if other controllers did not yet run.
      </td>
    </tr>

    <tr>
      <td>
        `containerOverridesHash` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        An hash of the containers overrides, used to determine if the overrides have changed for event filtering.
        this is updated only when the overrides are changed, and not when some other change in the source occurs.
      </td>
    </tr>

    <tr>
      <td>
        `agentsMetaHash` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        this hash is used to determine the deployment of the agents.
        e.g. when the distro for container changes, or it's compatibility version,
        or something else that requires rollout, the hash change will indicate that.
        if the hash is empty, it means that no agent should be enabled in any pod container.
      </td>
    </tr>

    <tr>
      <td>
        `agentsMetaHashChangedTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        The last time at which the agents meta hash value was changed.
        Pods created before this time may not be in alignment with the AgentsMetaHash.
        e.g. can lack the odigos label, or have a different value.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationConfigStatus`&#x20;

**Appears in:**

* [InstrumentationConfig](#odigos-io-v1alpha1-InstrumentationConfig)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `runtimeDetailsByContainer` \[Required]
      </td>

      <td>
        [`[]RuntimeDetailsByContainer`](#odigos-io-v1alpha1-RuntimeDetailsByContainer)
      </td>

      <td>
        Capture Runtime Details for the workloads that this CR applies to.
      </td>
    </tr>

    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a InstrumentationConfig's current state.
      </td>
    </tr>

    <tr>
      <td>
        `workloadRolloutHash` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        This hash is recorded only after the rollout took place.
        it allows us to determine if the workload needs to be rollout based on previous rollout and the current config.
        if this field is different than the spec.AgentsDeploymentHash it means rollout is needed or not yet updated.
      </td>
    </tr>

    <tr>
      <td>
        `rollbackOccurred` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Check if rollback happened to an application
      </td>
    </tr>

    <tr>
      <td>
        `instrumentationTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        This time recorded only after the rollout took place.
        This allows us to determine whether a crashing application should be rolled back or not
      </td>
    </tr>

    <tr>
      <td>
        `podsManifestInjectionStatus` \[Required]
      </td>

      <td>
        [`PodsManifestInjectionStatus`](#odigos-io-v1alpha1-PodsManifestInjectionStatus)
      </td>

      <td>
        Represents the status of odigos MANIFEST injection to existing pods template.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationInstanceSpec`&#x20;

**Appears in:**

* [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        stores the name of the container in the pod where the SDK is running.
        The pod details can be found as the owner reference on the CR.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationInstanceStatus`&#x20;

**Appears in:**

* [InstrumentationInstance](#odigos-io-v1alpha1-InstrumentationInstance)

InstrumentationInstanceStatus defines the observed state of InstrumentationInstance
If the instrumentation is not active, this CR should be deleted

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `identifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that identify the SDK and are reported as resource attributes in the generated telemetry.
        One can identify if an arbitrary telemetry is generated by this SDK by checking those resource attributes.
      </td>
    </tr>

    <tr>
      <td>
        `nonIdentifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that are not reported as resource attributes but useful to describe characteristics of the SDK.
      </td>
    </tr>

    <tr>
      <td>
        `healthy` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Healthy true means that the odigos agent has started the SDK, and there are no errors. User can expect telemetry to be generated.
        Healthy false means that the agent has stopped and telemetry data is not expected to be generated.
        Healthy nil means that the agent did not report any health status yet (prefer to always report health status).
      </td>
    </tr>

    <tr>
      <td>
        `message` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        message is a human readable message indicating details about the SDK general health.
        can be omitted if healthy is true
      </td>
    </tr>

    <tr>
      <td>
        `reason` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        reason contains a programmatic identifier indicating the reason for the component status.
        Producers of specific condition types may define expected values and meanings for this field,
        and whether the values are considered a guaranteed API.
      </td>
    </tr>

    <tr>
      <td>
        `lastStatusTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `components` \[Required]
      </td>

      <td>
        [`[]InstrumentationLibraryStatus`](#odigos-io-v1alpha1-InstrumentationLibraryStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryConfigTraces`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `enabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Whether the instrumentation library is enabled to record traces.
        When false, it is expected that the instrumentation library does not produce any spans regardless of any other configuration.
        When true, the instrumentation library should produce spans according to the other configuration options.
        If not specified, the default value for this signal should be used (whether to enable libraries by default or not).
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryId`&#x20;

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `libraryName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        The name of the instrumentation library

        <ul>
          <li>Node.js: The name of the npm package: `@opentelemetry/instrumentation-<name>`</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        `spanKind` \[Required]
      </td>

      <td>
        [`common.SpanKind`](https://pkg.go.dev/github.com/odigos-io/odigos/common#SpanKind)
      </td>

      <td>
        SpanKind is only supported by Golang and will be ignored for any other SDK language.
        In Go, SpanKind is used because the same instrumentation library can be utilized for different span kinds (e.g., client/server).
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryStatus`&#x20;

**Appears in:**

* [InstrumentationInstanceStatus](#odigos-io-v1alpha1-InstrumentationInstanceStatus)

InstrumentationLibraryStatus defines the observed state of an InstrumentationLibrary.
if a library is not active/disable, it should not be included in the status

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        for example ("net/http", "@opentelemetry/instrumentation-redis")
      </td>
    </tr>

    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        [`InstrumentationLibraryType`](#odigos-io-v1alpha1-InstrumentationLibraryType)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `identifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that identify the component.
        The combination of (Name, Type, IdentifyingAttributes) must be unique.
      </td>
    </tr>

    <tr>
      <td>
        `nonIdentifyingAttributes` \[Required]
      </td>

      <td>
        [`[]Attribute`](#odigos-io-v1alpha1-Attribute)
      </td>

      <td>
        Attributes that do not necessarily identify the component but help describe
        its characteristics.
      </td>
    </tr>

    <tr>
      <td>
        `healthy` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `message` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        message is a human readable message indicating details about the component health.
        can be omitted if healthy is true
      </td>
    </tr>

    <tr>
      <td>
        `reason` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        reason contains a programmatic identifier indicating the reason for the SDK status.
        Producers of specific condition types may define expected values and meanings for this field,
        and whether the values are considered a guaranteed API.
      </td>
    </tr>

    <tr>
      <td>
        `lastStatusTime` \[Required]
      </td>

      <td>
        [`meta/v1.Time`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#time-v1-meta)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationLibraryType`&#x20;

(Alias of `string`)

**Appears in:**

* [InstrumentationLibraryStatus](#odigos-io-v1alpha1-InstrumentationLibraryStatus)

## `InstrumentationRuleSpec`&#x20;

**Appears in:**

* [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `ruleName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        Allows you to attach a meaningful name to the rule for convenience. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        A free-form text field that allows you to attach notes regarding the rule for convenience. For example: why it was added. Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        A boolean field allowing to temporarily disable the rule, but keep it around for future use
      </td>
    </tr>

    <tr>
      <td>
        `scopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        SourcesScopes lists SourcesScope entries to which the rule should be applied. If unset or empty, the rule applies to all workloads.
      </td>
    </tr>

    <tr>
      <td>
        `instrumentationLibraries` \[Required]
      </td>

      <td>
        [`[]github.com/odigos-io/odigos/api/odigos/v1alpha1.InstrumentationLibraryGlobalId`](#odigos-io-v1alpha1-\[]github-com-odigos-io-odigos-api-odigos-v1alpha1-InstrumentationLibraryGlobalId)
      </td>

      <td>
        For fine grained control, the user can specify the instrumentation library to use.
        One can specify same rule for multiple languages and libraries at the same time.
        If nil, all instrumentation libraries will be used.
        If empty, no instrumentation libraries will be used.
      </td>
    </tr>

    <tr>
      <td>
        `payloadCollection` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.PayloadCollection`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#PayloadCollection)
      </td>

      <td>
        Allows to configure payload collection aspects for different types of payloads.
      </td>
    </tr>

    <tr>
      <td>
        `otelDistros` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.OtelDistros`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#OtelDistros)
      </td>

      <td>
        Set the otel distros to use instead of the defaults.
      </td>
    </tr>

    <tr>
      <td>
        `codeAttributes` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.CodeAttributes`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#CodeAttributes)
      </td>

      <td>
        Configure which code attributes should be recorded as span attributes.
      </td>
    </tr>

    <tr>
      <td>
        `headersCollection` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.HttpHeadersCollection`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#HttpHeadersCollection)
      </td>

      <td>
        Allows to configure the collection of http headers for different types of payloads.
      </td>
    </tr>

    <tr>
      <td>
        `traceConfig` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.TraceConfig`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#TraceConfig)
      </td>

      <td>
        Configure the tracing configuration for the library.
      </td>
    </tr>

    <tr>
      <td>
        `customInstrumentations` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.CustomInstrumentationsWithMetadata`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#CustomInstrumentationsWithMetadata)
      </td>

      <td>
        Add custom instrumentation probes
      </td>
    </tr>

    <tr>
      <td>
        `headSamplingFallbackFraction` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.HeadSamplingFallbackFraction`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#HeadSamplingFallbackFraction)
      </td>

      <td>
        Configure general fraction of traces to record if none of the rules evaluate to true.
        this can help in reducing very noisy and heavy traffic services.
        note that traces will be dropped regardless of thier attributes/errors/importance.
        @deprecated: use odigos config to set this value instead.
      </td>
    </tr>

    <tr>
      <td>
        `ebpfLogCapture` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.EbpfLogCapture`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#EbpfLogCapture)
      </td>

      <td>
        Configure eBPF-based log capture. When enabled, the node collector logs pipeline
        will use only the eBPF receiver instead of the filelog receiver, and odiglet will
        register instrumented processes for eBPF log capture.
      </td>
    </tr>

    <tr>
      <td>
        `networkMetrics` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.NetworkMetricsConfig`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#NetworkMetricsConfig)
      </td>

      <td>
        Configure network flow and TCP stats metrics for scoped workloads.
      </td>
    </tr>

    <tr>
      <td>
        `traceVerbosity` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.TraceVerbosity`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#TraceVerbosity)
      </td>

      <td>
        Configure the verbosity of the traces for the library.
      </td>
    </tr>

    <tr>
      <td>
        `agentDiagnostics` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.AgentDiagnostics`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#AgentDiagnostics)
      </td>

      <td>
        Configure the agent own logging configuration.
      </td>
    </tr>

    <tr>
      <td>
        `kernelTraces` \[Required]
      </td>

      <td>
        [`common/api/instrumentationrules.KernelTraces`](https://pkg.go.dev/github.com/odigos-io/odigos/common/api/instrumentationrules#KernelTraces)
      </td>

      <td>
        Adds spans for the operating system work of the scoped workloads, from any language, like the
        files they open and the processes they start. A span is a child of the span in progress when the
        work happens. File and process spans start a trace when there is none.
      </td>
    </tr>
  </tbody>
</table>

## `InstrumentationRuleStatus`&#x20;

**Appears in:**

* [InstrumentationRule](#odigos-io-v1alpha1-InstrumentationRule)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a instrumentationrule's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `NoisyOperation`&#x20;

**Appears in:**

* [SamplingSpec](#odigos-io-v1alpha1-SamplingSpec)

endpoints (or other operations) which are considered "noise", and provide no or very little observability value.
these traces should not be collected at all, or dropped aggresevly.
motivation is data sentization and performance improvment (even if cost is not a factor)examples:

<ul>
  <li>health-checks (readiness and liveness probes)</li>
  <li>metrics scrape endpoints (prometheus /metrics endpoint)</li>
  <li>other agents calling home (outgoing http requests to collector.my.vendor.com)</li>
</ul>

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user provided name, for easier identification and reference.
        use short and descriptive name, like "health check", "always keep /transaction in payment service", etc.
        odigos does not use or assume any meaning from this field,
        but it is written as metric attribute, and stored as span attribute on participating spans.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if set to true, the rule will be disabled,
        e.g. will not be taken into account for any sampling decisions.
        disabled rules still participate in metrics calculations,
        allowing enhanced tools and data for troubleshooting and sampling maintenance.
      </td>
    </tr>

    <tr>
      <td>
        `sourceScopes` \[Required]
      </td>

      <td>
        [`k8sconsts.SourcesScopes`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#SourcesScopes)
      </td>

      <td>
        limit this rule to specific sources (by name, namespace, language, etc.)
        for example: if "other agent" rule for noisty operation is relevant only in java,
        limit this rule by setting source scope to java and prevent other languages from being affected
        if the list is empty - all sources are matched.
      </td>
    </tr>

    <tr>
      <td>
        `operation` \[Required]
      </td>

      <td>
        `common/api/sampling.HeadSamplingOperationMatcher`
      </td>

      <td>
        limit this rule to specific operations.
        for example: specific http server endpoint (GET "/healthz" as an example).
        this field is optional, and if not set, the rule will be applied to all operations.
      </td>
    </tr>

    <tr>
      <td>
        `percentageAtMost` \[Required]
      </td>

      <td>
        `float64`
      </td>

      <td>
        sampling percentage for noisy operations.
        if unset, 0% of such the traces will be collected.
        this percentage has "at most" semantics - the final sampling percentage for traces that match this rule
        will be the highest possible, but at most this value.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        optional free-form text field that allows you to attach notes
        for future context and maintenance.
        users can write why this rule was added, observations, document considerations, etc.
      </td>
    </tr>
  </tbody>
</table>

## `OdigosOwnMetricsSettings`&#x20;

**Appears in:**

* [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

configuration for collecting and exporting odigos own metrics.
e.g. metrics about odigos components, and not about the user's application.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `sendToMetricsDestinations` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if true, odigos will send all the metrics it collects about itself to the metrics pipeline,
        which will make them available to the metrics destinations.
        users can troubleshoot odigos itself by monitoring these metrics in their existing systems,
        and create their own dashboards, alerting, and more.
      </td>
    </tr>

    <tr>
      <td>
        `sendToOdigosMetricsStore` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if true, odigos will send all the metrics it collects about itself to the odigos metrics store,
        which is available to odigos UI.
        it can help in presenting a consistent view of odigos itself, without relying on user system and integrations.
      </td>
    </tr>

    <tr>
      <td>
        `interval` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        time interval for flusing metrics (format: 15s, 1m etc). defaults: 10s
      </td>
    </tr>
  </tbody>
</table>

## `OtherAgent`&#x20;

**Appears in:**

* [RuntimeDetailsByContainer](#odigos-io-v1alpha1-RuntimeDetailsByContainer)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `PodsManifestInjectionStatus`&#x20;

**Appears in:**

* [InstrumentationConfigStatus](#odigos-io-v1alpha1-InstrumentationConfigStatus)

represents the status of odigos MANIFEST injection to existing pods template.
each pod can be in one of 3 states: injected and up-to-date, injected and out-of-date, or not injected.
actual agent in pod container can be injected or not regardless of the pod manifest injection status.
status only shows if pods exists from each category and not count, to avoid frequent (noisy/expensive) updates in steady state

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `hasInjectedUpToDatePods` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if the source has any pods which are using the latest agent deployment setup.
      </td>
    </tr>

    <tr>
      <td>
        `hasInjectedOutOfDatePods` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if the source has any pods which are using an outdated agent deployment setup.
      </td>
    </tr>

    <tr>
      <td>
        `hasUninjectedPods` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        if the source has any pods which do not have the odigos agent injected.
      </td>
    </tr>
  </tbody>
</table>

## `ProcessorSpec`&#x20;

**Appears in:**

* [Processor](#odigos-io-v1alpha1-Processor)

ProcessorSpec defines the an OpenTelemetry Collector processor in odigos telemetry pipeline

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        type of the processor (batch, attributes, etc).
        this field is only the type, not it's instance name in the collector configuration yaml
      </td>
    </tr>

    <tr>
      <td>
        `processorName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        this name is solely for the user convenience, to attach a meaningful name to the processor.
        odigos must not assume any semantics from this name.
        odigos cannot assume this name is unique, not empty, exclude spaces or dots, limited in length, etc.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        user can attach notes to the processor, to document its purpose, usage, etc.
      </td>
    </tr>

    <tr>
      <td>
        `disabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        disable is a flag to enable or disable the processor.
        if the processor is disabled, it will not be included in the collector configuration yaml.
        this allows the user to keep the processor configuration in the CR, but disable it temporarily.
      </td>
    </tr>

    <tr>
      <td>
        `signals` \[Required]
      </td>

      <td>
        [`[]common.ObservabilitySignal`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ObservabilitySignal)
      </td>

      <td>
        signals can be used to control which observability signals are processed by the processor.
      </td>
    </tr>

    <tr>
      <td>
        `collectorRoles` \[Required]
      </td>

      <td>
        [`[]CollectorsGroupRole`](#odigos-io-v1alpha1-CollectorsGroupRole)
      </td>

      <td>
        control which collector roles in odigos pipeline this processor is attached to.
      </td>
    </tr>

    <tr>
      <td>
        `orderHint` \[Required]
      </td>

      <td>
        `int`
      </td>

      <td>
        control the order of processors.
        a processor with lower order hint value will be placed before other processors with higher value.
        if 2 processors have the same value, the order is arbitrary.
        if the value is missing (or 0) the processor can be placed anywhere in the pipeline
      </td>
    </tr>

    <tr>
      <td>
        `processorConfig` \[Required]
      </td>

      <td>
        [`k8s.io/apimachinery/pkg/runtime.RawExtension`](https://pkg.go.dev/k8s.io/apimachinery/pkg/runtime/#RawExtension)
      </td>

      <td>
        this it the configuration of the opentelemetry collector processor component with the type specified in 'type'.
      </td>
    </tr>
  </tbody>
</table>

## `ProcessorStatus`&#x20;

**Appears in:**

* [Processor](#odigos-io-v1alpha1-Processor)

ProcessorStatus defines the observed state of the processor

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a processor's current state.
      </td>
    </tr>
  </tbody>
</table>

## `RecommendationSpec`&#x20;

**Appears in:**

* [Recommendation](#odigos-io-v1alpha1-Recommendation)

RecommendationSpec defines the desired state of Recommendation

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `type` \[Required]
      </td>

      <td>
        [`common.RecommendationType`](https://pkg.go.dev/github.com/odigos-io/odigos/common#RecommendationType)
      </td>

      <td>
        Type identifies which recommendation from the catalog this resource enables.
      </td>
    </tr>

    <tr>
      <td>
        `applied`
      </td>

      <td>
        `bool`
      </td>

      <td>
        Applied indicates whether this recommendation is currently applied
        (via the recommendation flow or configured organically).
      </td>
    </tr>

    <tr>
      <td>
        `conditionsMet`
      </td>

      <td>
        `bool`
      </td>

      <td>
        ConditionsMet indicates whether the catalog prerequisites for this recommendation
        currently hold (for example GoEnterpriseSources for AutoGoOffsetUpdater).
      </td>
    </tr>
  </tbody>
</table>

## `RecommendationStatus`&#x20;

**Appears in:**

* [Recommendation](#odigos-io-v1alpha1-Recommendation)

RecommendationStatus defines the observed state of Recommendation

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a recommendation's current state.
      </td>
    </tr>
  </tbody>
</table>

## `RuntimeDetailsByContainer`&#x20;

**Appears in:**

* [ContainerOverride](#odigos-io-v1alpha1-ContainerOverride)
* [InstrumentationConfigStatus](#odigos-io-v1alpha1-InstrumentationConfigStatus)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `containerName` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `language` \[Required]
      </td>

      <td>
        [`common.ProgrammingLanguage`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ProgrammingLanguage)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `runtimeVersion` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `envVars` \[Required]
      </td>

      <td>
        [`[]EnvVar`](#odigos-io-v1alpha1-EnvVar)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `otherAgents` \[Required]
      </td>

      <td>
        [`[]OtherAgent`](#odigos-io-v1alpha1-OtherAgent)
      </td>

      <td>
        OtherAgents lists other instrumentation agents detected in the container
        (a process can carry more than one); empty when none.
      </td>
    </tr>

    <tr>
      <td>
        `libCType` \[Required]
      </td>

      <td>
        [`common.LibCType`](https://pkg.go.dev/github.com/odigos-io/odigos/common#LibCType)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `secureExecutionMode` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        Indicates whether the target process is running is secure-execution mode.
        nil means we were unable to determine the secure-execution mode.
      </td>
    </tr>

    <tr>
      <td>
        `criErrorMessage` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        CriErrorMessage is set if the value in EnvFromContainerRuntime was not computed correctly and cannot be used safely.
        Sometimes, even if CRI check failed, it is possible to tell that relevant env vars are not coming from container runtime.
        Thus, this field is set only when there is:

        <ul>
          <li>Actual CRI check failed</li>
          <li>The observed environment variables might come from container runtime</li>
        </ul>
      </td>
    </tr>

    <tr>
      <td>
        `envFromContainerRuntime` \[Required]
      </td>

      <td>
        [`[]EnvVar`](#odigos-io-v1alpha1-EnvVar)
      </td>

      <td>
        Holds the environment variables retrieved from the container runtime.
      </td>
    </tr>
  </tbody>
</table>

## `SamplingSpec`&#x20;

**Appears in:**

* [Sampling](#odigos-io-v1alpha1-Sampling)

define sampling rules.
the rules can be defined as one or multiple objects in kubernetes,
and are all joined together to form the global sampling rules.
odigos users can group rules based on whatever criteria that makes sense for them,
for example - by team, by client, by usecase, admin-policy, etc.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `name` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        give these sampling rules a name for display, easier identification and reference.
      </td>
    </tr>

    <tr>
      <td>
        `notes` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        a free-form text field that allows you to attach notes regardinag the rule for convenience.
        Odigos does not use or assume any meaning from this field.
      </td>
    </tr>

    <tr>
      <td>
        `noisyOperations` \[Required]
      </td>

      <td>
        [`[]NoisyOperation`](#odigos-io-v1alpha1-NoisyOperation)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `highlyRelevantOperations` \[Required]
      </td>

      <td>
        [`[]HighlyRelevantOperation`](#odigos-io-v1alpha1-HighlyRelevantOperation)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `costReductionRules` \[Required]
      </td>

      <td>
        [`[]CostReductionRule`](#odigos-io-v1alpha1-CostReductionRule)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `SamplingStatus`&#x20;

**Appears in:**

* [Sampling](#odigos-io-v1alpha1-Sampling)

SamplingStatus defines the observed state of Sampling.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a Sampling's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>

## `ServiceGraphSettings`&#x20;

**Appears in:**

* [CollectorsGroupMetricsCollectionSettings](#odigos-io-v1alpha1-CollectorsGroupMetricsCollectionSettings)

## `SourceSelector`&#x20;

**Appears in:**

* [DestinationSpec](#odigos-io-v1alpha1-DestinationSpec)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `namespaces`
      </td>

      <td>
        `[]string`
      </td>

      <td>
        If a namespace is specified, all workloads (sources) within that namespace are allowed to send data.
        Example:
        namespaces: \["default", "production"]
        This means the destination will receive data from all sources in "default" and "production" namespaces.
      </td>
    </tr>

    <tr>
      <td>
        `dataStreams`
      </td>

      <td>
        `[]string`
      </td>

      <td>
        Workloads (sources) are assigned to Datastreams via labels (odigos.io/data-stream: true), allowing a more flexible selection mechanism.
        Example:
        dataStreams: \["backend", "monitoring"]
        This means the destination will receive data only from sources labeled with "backend" or "monitoring".
      </td>
    </tr>
  </tbody>
</table>

## `SourceSpec`&#x20;

**Appears in:**

* [Source](#odigos-io-v1alpha1-Source)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `workload` \[Required]
      </td>

      <td>
        [`k8sconsts.PodWorkload`](https://pkg.go.dev/github.com/odigos-io/odigos/api/k8sconsts#PodWorkload)
      </td>

      <td>
        Workload represents the workload or namespace to be instrumented.
        This field is required upon creation and cannot be modified.
      </td>
    </tr>

    <tr>
      <td>
        `disableInstrumentation` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        DisableInstrumentation excludes this workload from auto-instrumentation.
      </td>
    </tr>

    <tr>
      <td>
        `otelServiceName`
      </td>

      <td>
        `string`
      </td>

      <td>
        OtelServiceName determines the "service.name" resource attribute which will be reported by the instrumentations of this source.
        If not set, the workload name will be used.
        It is not valid for namespace sources.
      </td>
    </tr>

    <tr>
      <td>
        `containerOverrides`
      </td>

      <td>
        [`[]ContainerOverride`](#odigos-io-v1alpha1-ContainerOverride)
      </td>

      <td>
        Specify specific override values for containers in a workload source.
        Not valid for namespace sources.
        Can be used to set the runtime info in case the automatic detection fails or produce wrong results.
        Containers are identified by their names.
        All containers not listed will retain their default behavior.
      </td>
    </tr>

    <tr>
      <td>
        `matchWorkloadNameAsRegex`
      </td>

      <td>
        `bool`
      </td>

      <td>
        MatchWorkloadNameAsRegex indicates that the workload name should be matched using regex.
        When true, spec.workload.name is treated as a regular expression pattern.
        This allows matching multiple workloads with a single Source CRD.
        Not valid for namespace sources.
      </td>
    </tr>
  </tbody>
</table>

## `SourceStatus`&#x20;

**Appears in:**

* [Source](#odigos-io-v1alpha1-Source)

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Represents the observations of a source's current state.
        Known .status.conditions.type are: "Available", "Progressing"
      </td>
    </tr>
  </tbody>
</table>


# operator.odigos.io/v1alpha1
Source: https://docs.odigos.io/oss/api-reference/operator.odigos.io.v1alpha1



## Resource Types

## `Odigos`&#x20;

Odigos is the Schema for the odigos API

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `metadata` \[Required]
      </td>

      <td>
        [`meta/v1.ObjectMeta`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)
      </td>

      <td>
        No description provided.
        Refer to the Kubernetes API documentation for the fields of the `metadata` field.
      </td>
    </tr>

    <tr>
      <td>
        `spec` \[Required]
      </td>

      <td>
        [`OdigosSpec`](#OdigosSpec)
      </td>

      <td>
        No description provided.
      </td>
    </tr>

    <tr>
      <td>
        `status` \[Required]
      </td>

      <td>
        [`OdigosStatus`](#OdigosStatus)
      </td>

      <td>
        No description provided.
      </td>
    </tr>
  </tbody>
</table>

## `OdigosSpec`&#x20;

**Appears in:**

* [Odigos](#Odigos)

OdigosSpec defines the desired state of Odigos

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `onPremToken` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        (Optional) OnPremToken is an enterprise token for Odigos Enterprise.
      </td>
    </tr>

    <tr>
      <td>
        `uiMode` \[Required]
      </td>

      <td>
        [`common.UiMode`](https://pkg.go.dev/github.com/odigos-io/odigos/common#UiMode)
      </td>

      <td>
        (Optional) UIMode sets the UI mode to either "normal" or "readonly".
        In "normal" mode the UI is fully interactive, allowing users to view and edit
        Odigos configuration and settings. In "readonly" mode, the UI can only be
        used to view current Odigos configuration and is not interactive.
        Default=default
      </td>
    </tr>

    <tr>
      <td>
        `telemetryEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) TelemetryEnabled records general telemetry regarding Odigos usage.
        Default=false
      </td>
    </tr>

    <tr>
      <td>
        `ignoredNamespaces` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        (Optional) IgnoredNamespaces is a list of namespaces to not show in the Odigos UI.
      </td>
    </tr>

    <tr>
      <td>
        `ignoredContainers` \[Required]
      </td>

      <td>
        `[]string`
      </td>

      <td>
        (Optional) IgnoredContainers is a list of container names to exclude from instrumentation (useful for ignoring sidecars).
      </td>
    </tr>

    <tr>
      <td>
        `profiles` \[Required]
      </td>

      <td>
        [`[]common.ProfileName`](https://pkg.go.dev/github.com/odigos-io/odigos/common#ProfileName)
      </td>

      <td>
        (Optional) Profiles is a list of preset profiles with a specific configuration.
      </td>
    </tr>

    <tr>
      <td>
        `agentEnvVarsInjectionMethod` \[Required]
      </td>

      <td>
        [`common.EnvInjectionMethod`](https://pkg.go.dev/github.com/odigos-io/odigos/common#EnvInjectionMethod)
      </td>

      <td>
        (Optional) AgentEnvVarsInjectionMethod is the method to inject the Odigos agent env vars into the pod.
        Default=pod-manifest
      </td>
    </tr>

    <tr>
      <td>
        `nodeSelector` \[Required]
      </td>

      <td>
        `map[string]string`
      </td>

      <td>
        (Optional) NodeSelector is a map of key-value Kubernetes NodeSelector labels to apply to all Odigos components.
        Note that Odigos will only be able to instrument applications on the same node.
      </td>
    </tr>

    <tr>
      <td>
        `skipWebhookIssuerCreation` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) SkipWebhookIssuerCreation skips creating the Issuer and Certificate for the Instrumentor pod webhook if cert-manager is installed.
        Default=false
      </td>
    </tr>

    <tr>
      <td>
        `podSecurityPolicy` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) PodSecurityPolicy allows Odigos pods to use a privileged pod security policy.
        Default=false
      </td>
    </tr>

    <tr>
      <td>
        `imagePrefix` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        (Optional) ImagePrefix is a prefix for all container images.
        This should only be used if you are pulling Odigos images from the non-default registry.
        Default (community): docker.io/keyval
        Default (enterprise): registry.odigos.io
        Default (OpenShift): registry.connect.redhat.com/odigos
      </td>
    </tr>

    <tr>
      <td>
        `logLevel` \[Required]
      </td>

      <td>
        `string`
      </td>

      <td>
        (Optional) LogLevel sets the default log level for all Odigos components (e.g. instrumentor, odiglet, collector).
        Accepted values: debug, info, warn, error. Default=info when unset.
      </td>
    </tr>

    <tr>
      <td>
        `mountMethod` \[Required]
      </td>

      <td>
        [`common.MountMethod`](https://pkg.go.dev/github.com/odigos-io/odigos/common#MountMethod)
      </td>

      <td>
        (Optional) MountMethod optionally defines the mechanism for mounting Odigos files into instrumented pods.
        One of "k8s-virtual-device" (default) or "k8s-host-path".
      </td>
    </tr>

    <tr>
      <td>
        `openshiftEnabled` \[Required]
      </td>

      <td>
        `bool`
      </td>

      <td>
        (Optional) OpenShiftEnabled configures selinux on OpenShift nodes.
        DEPRECATED: OpenShift clusters are auto-detected and this API field will be removed in a future release.
      </td>
    </tr>
  </tbody>
</table>

## `OdigosStatus`&#x20;

**Appears in:**

* [Odigos](#Odigos)

OdigosStatus defines the observed state of Odigos

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td>
        `conditions` \[Required]
      </td>

      <td>
        [`[]meta/v1.Condition`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#condition-v1-meta)
      </td>

      <td>
        Conditions store the status conditions of the Odigos instances
      </td>
    </tr>
  </tbody>
</table>


# Overview
Source: https://docs.odigos.io/oss/backends-overview

Odigos makes it simple to add and configure destinations, allowing you to select the specific signals (`traces`,`metrics`,`logs`) that you want to send to each destination.

> [!TIP]
>
> Can't find your backend in Odigos? Please tell us! We are constantly adding new integrations.\\
>
> You can also follow our quick [add new destination](/adding-new-dest) guide and submit a PR.

| | | | Traces | Metrics | Logs | Profiles |
| - | - | - | :-: | :-: | :-: | :-: |
| ![alibabacloud](https://d15jtxgb40qetw.cloudfront.net/alibabacloud.svg) | [Alibaba Cloud](/backends/alibabacloud) | Managed | ✅ | | | |
| ![appdynamics](https://d15jtxgb40qetw.cloudfront.net/appdynamics.svg) | [AppDynamics](/backends/appdynamics) | Managed | ✅ | ✅ | ✅ | |
| ![cloudwatch](https://d15jtxgb40qetw.cloudfront.net/awscloudwatch.svg) | [AWS CloudWatch](/backends/awscloudwatch) | Managed | | ✅ | ✅ | |
| ![s3](https://d15jtxgb40qetw.cloudfront.net/awss3.svg) | [AWS S3](/backends/awss3) | Managed | ✅ | ✅ | ✅ | |
| ![xray](https://d15jtxgb40qetw.cloudfront.net/awsxray.svg) | [AWS X-Ray](/backends/awsxray) | Managed | ✅ | | | |
| ![axiom](https://d15jtxgb40qetw.cloudfront.net/axiom.svg) | [Axiom](/backends/axiom) | Managed | ✅ | | ✅ | |
| ![azureblob](https://d15jtxgb40qetw.cloudfront.net/blobstorage.svg) | [Azure Blob Storage](/backends/azureblob) | Managed | ✅ | | ✅ | |
| ![azuremonitor](https://d15jtxgb40qetw.cloudfront.net/azuremonitor.svg) | [Azure Monitor](/backends/azuremonitor) | Managed | ✅ | ✅ | ✅ | |
| ![betterstack](https://d15jtxgb40qetw.cloudfront.net/betterstack.svg) | [Better Stack](/backends/betterstack) | Managed | | ✅ | ✅ | |
| ![bonree](https://d15jtxgb40qetw.cloudfront.net/bonree.svg) | [Bonree ONE](/backends/bonree) | Managed | ✅ | ✅ | | |
| ![causely](https://d15jtxgb40qetw.cloudfront.net/causely.svg) | [Causely](/backends/causely) | Managed | ✅ | ✅ | | |
| ![checkly](https://d15jtxgb40qetw.cloudfront.net/checkly.svg) | [Checkly](/backends/checkly) | Managed | ✅ | | | |
| ![chronosphere](https://d15jtxgb40qetw.cloudfront.net/chronosphere.svg) | [Chronosphere](/backends/chronosphere) | Managed | ✅ | ✅ | | |
| ![clickhouse](https://d15jtxgb40qetw.cloudfront.net/clickhouse.svg) | [Clickhouse](/backends/clickhouse) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![coralogix](https://d15jtxgb40qetw.cloudfront.net/coralogix.svg) | [Coralogix](/backends/coralogix) | Managed | ✅ | ✅ | ✅ | |
| ![dash0](https://d15jtxgb40qetw.cloudfront.net/dash0.svg) | [Dash0](/backends/dash0) | Managed | ✅ | ✅ | ✅ | |
| ![datadog](https://d15jtxgb40qetw.cloudfront.net/datadog.svg) | [Datadog](/backends/datadog) | Managed | ✅ | ✅ | ✅ | |
| ![dynamic](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg) | [Dynamic Destination](/backends/dynamic) | Self-Hosted | ✅ | ✅ | ✅ | ✅ |
| ![dynatrace](https://d15jtxgb40qetw.cloudfront.net/dynatrace.svg) | [Dynatrace](/backends/dynatrace) | Managed | ✅ | ✅ | ✅ | |
| ![elasticapm](https://d15jtxgb40qetw.cloudfront.net/elasticapm.svg) | [Elastic APM](/backends/elasticapm) | Managed | ✅ | ✅ | ✅ | |
| ![elasticsearch](https://d15jtxgb40qetw.cloudfront.net/elasticsearch.svg) | [Elasticsearch](/backends/elasticsearch) | Self-Hosted | ✅ | | ✅ | |
| ![qryn](https://d15jtxgb40qetw.cloudfront.net/gigapipe.svg) | [Gigapipe](/backends/gigapipe) | Managed | ✅ | ✅ | ✅ | |
| ![googlecloud](https://d15jtxgb40qetw.cloudfront.net/gcp.svg) | [Google Cloud Monitoring](/backends/googlecloudmonitoring) | Managed | ✅ | | ✅ | |
| ![googlecloudotlp](https://d15jtxgb40qetw.cloudfront.net/gcp.svg) | [Google Cloud (OTLP)](/backends/googlecloudotlp) | Managed | ✅ | | | |
| ![grafanacloudloki](https://d15jtxgb40qetw.cloudfront.net/grafana.svg) | [Grafana Cloud Loki](/backends/grafanacloudloki) | Managed | | | ✅ | |
| ![grafanacloudprometheus](https://d15jtxgb40qetw.cloudfront.net/grafana.svg) | [Grafana Cloud Prometheus](/backends/grafanacloudprometheus) | Managed | | ✅ | | |
| ![grafanacloudtempo](https://d15jtxgb40qetw.cloudfront.net/grafana.svg) | [Grafana Cloud Tempo](/backends/grafanacloudtempo) | Managed | ✅ | | | |
| ![greptime](https://d15jtxgb40qetw.cloudfront.net/greptime.svg) | [GreptimeDB](/backends/greptime) | Managed | | ✅ | | |
| ![groundcover](https://d15jtxgb40qetw.cloudfront.net/groundcover.svg) | [Groundcover inCloud](/backends/groundcover) | Managed | ✅ | ✅ | ✅ | |
| ![honeycomb](https://d15jtxgb40qetw.cloudfront.net/honeycomb.svg) | [Honeycomb](/backends/honeycomb) | Managed | ✅ | ✅ | ✅ | |
| ![hyperdx](https://d15jtxgb40qetw.cloudfront.net/hyperdx.svg) | [HyperDX](/backends/hyperdx) | Managed | ✅ | ✅ | ✅ | |
| ![instana](https://d15jtxgb40qetw.cloudfront.net/instana.svg) | [IBM Instana](/backends/instana) | Managed | ✅ | ✅ | ✅ | |
| ![jaeger](https://d15jtxgb40qetw.cloudfront.net/jaeger.svg) | [Jaeger](/backends/jaeger) | Self-Hosted | ✅ | | | |
| ![kafka](https://d15jtxgb40qetw.cloudfront.net/kafka.svg) | [Kafka](/backends/kafka) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![kloudmate](https://d15jtxgb40qetw.cloudfront.net/kloudmate.svg) | [KloudMate](/backends/kloudmate) | Managed | ✅ | ✅ | ✅ | |
| ![last9](https://d15jtxgb40qetw.cloudfront.net/last9.svg) | [Last9](/backends/last9) | Managed | ✅ | ✅ | ✅ | |
| ![lightstep](https://d15jtxgb40qetw.cloudfront.net/lightstep.svg) | [Lightstep](/backends/lightstep) | Managed | ✅ | | | |
| ![logzio](https://d15jtxgb40qetw.cloudfront.net/logzio.svg) | [Logz.io](/backends/logzio) | Managed | ✅ | ✅ | ✅ | |
| ![loki](https://d15jtxgb40qetw.cloudfront.net/loki.svg) | [Loki](/backends/loki) | Self-Hosted | | | ✅ | |
| ![lumigo](https://d15jtxgb40qetw.cloudfront.net/lumigo.svg) | [Lumigo](/backends/lumigo) | Managed | ✅ | ✅ | ✅ | |
| ![middleware](https://d15jtxgb40qetw.cloudfront.net/middleware.svg) | [Middleware](/backends/middleware) | Managed | ✅ | ✅ | ✅ | |
| ![newrelic](https://d15jtxgb40qetw.cloudfront.net/newrelic.svg) | [New Relic](/backends/newrelic) | Managed | ✅ | ✅ | ✅ | |
| ![observe](https://d15jtxgb40qetw.cloudfront.net/observe.svg) | [Observe](/backends/observe) | Managed | ✅ | ✅ | ✅ | |
| ![oneuptime](https://d15jtxgb40qetw.cloudfront.net/oneuptime.svg) | [OneUptime](/backends/oneuptime) | Managed | ✅ | ✅ | ✅ | |
| ![openobserve](https://d15jtxgb40qetw.cloudfront.net/openobserve.svg) | [OpenObserve](/backends/openobserve) | Managed | ✅ | | ✅ | |
| ![oracle](https://d15jtxgb40qetw.cloudfront.net/oracle.svg) | [Oracle Cloud](/backends/oracle) | Managed | ✅ | ✅ | | |
| ![otlp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg) | [OTLP gRPC](/backends/otlp) | Self-Hosted | ✅ | ✅ | ✅ | ✅ |
| ![otlphttp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg) | [OTLP http](/backends/otlphttp) | Self-Hosted | ✅ | ✅ | ✅ | ✅ |
| ![prometheus](https://d15jtxgb40qetw.cloudfront.net/prometheus.svg) | [Prometheus](/backends/prometheus) | Self-Hosted | | ✅ | | |
| ![pyroscope](https://d15jtxgb40qetw.cloudfront.net/pyroscope.svg) | [Pyroscope](/backends/pyroscope) | Self-Hosted | | | | ✅ |
| ![qryn-oss](https://d15jtxgb40qetw.cloudfront.net/qryn.svg) | [qryn](/backends/qryn) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![quickwit](https://d15jtxgb40qetw.cloudfront.net/quickwit.svg) | [Quickwit](/backends/quickwit) | Self-Hosted | ✅ | | ✅ | |
| ![seq](https://d15jtxgb40qetw.cloudfront.net/seq.svg) | [Seq](/backends/seq) | Self-Hosted | ✅ | | ✅ | |
| ![signalfx](https://d15jtxgb40qetw.cloudfront.net/signalfx.svg) | [SignalFx](/backends/signalfx) | Managed | ✅ | ✅ | | |
| ![signoz](https://d15jtxgb40qetw.cloudfront.net/signoz.svg) | [SigNoz](/backends/signoz) | Self-Hosted | ✅ | ✅ | ✅ | |
| ![splunk](https://d15jtxgb40qetw.cloudfront.net/splunk.svg) | [Splunk (Legacy) (Deprecated)](/backends/splunk) | Managed | ✅ | | | |
| ![splunkotlp](https://d15jtxgb40qetw.cloudfront.net/splunk.svg) | [Splunk (OTLP)](/backends/splunkotlp) | Managed | ✅ | | | |
| ![sumologic](https://d15jtxgb40qetw.cloudfront.net/sumologic.svg) | [Sumo Logic](/backends/sumologic) | Managed | ✅ | ✅ | ✅ | |
| ![telemetryhub](https://d15jtxgb40qetw.cloudfront.net/telemetryhub.svg) | [TelemetryHub](/backends/telemetryhub) | Managed | ✅ | ✅ | ✅ | |
| ![tempo](https://d15jtxgb40qetw.cloudfront.net/tempo.svg) | [Tempo](/backends/tempo) | Self-Hosted | ✅ | | | |
| ![tingyun](https://d15jtxgb40qetw.cloudfront.net/tingyun.svg) | [Tingyun 基调听云](/backends/tingyun) | Managed | ✅ | ✅ | | |
| ![traceloop](https://d15jtxgb40qetw.cloudfront.net/traceloop.svg) | [Traceloop](/backends/traceloop) | Managed | ✅ | ✅ | | |
| ![uptrace](https://d15jtxgb40qetw.cloudfront.net/uptrace.svg) | [Uptrace](/backends/uptrace) | Managed | ✅ | ✅ | ✅ | |
| ![victoriametricscloud](https://d15jtxgb40qetw.cloudfront.net/victoriametrics.svg) | [VictoriaMetrics Cloud](/backends/victoriametricscloud) | Managed | | ✅ | | |


# Alibaba Cloud
Source: https://docs.odigos.io/oss/backends/alibabacloud

Configuring the Alibaba Cloud backend (Managed)

### Getting Started

![alibabacloud](https://d15jtxgb40qetw.cloudfront.net/alibabacloud.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **ALIBABA\_ENDPOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is required. Do not add the prefix `http://`.
  * This field is required
* **ALIBABA\_TOKEN** `string` : Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Alibaba Cloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `alibabacloud.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: alibabacloud-example
  namespace: odigos-system
spec:
  data:
    ALIBABA_ENDPOINT: <OTLP gRPC Endpoint>
  destinationName: alibabacloud
  secretRef:
    name: alibabacloud-secret
  signals:
  - TRACES
  type: alibabacloud

---

apiVersion: v1
data:
  ALIBABA_TOKEN: <Base64 Token>
kind: Secret
metadata:
  name: alibabacloud-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f alibabacloud.yaml
```


# AppDynamics
Source: https://docs.odigos.io/oss/backends/appdynamics

Configuring the AppDynamics backend (Managed)

### Getting Started

![appdynamics](https://d15jtxgb40qetw.cloudfront.net/appdynamics.svg)

1. [Register](https://accounts.appdynamics.com/trial)/[Login](https://login.appdynamics.com/sso/authenticate) to AppDynamics.
2. Navigate to the OpenTelemetry Configuration page
3. Click on `Processors`, you'll find 3 attributes, locate `appdynamics.controller.account` and copy its value, this is your `Account Name`.
4. Click on `Exporters`, copy the `Endpoint`, and generate an `API Key`.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **APPDYNAMICS\_APPLICATION\_NAME** `string` : Application Name. Will define a namespace in AppDynamics
  * This field is optional
* **APPDYNAMICS\_ACCOUNT\_NAME** `string` : Account Name.
  * This field is required
* **APPDYNAMICS\_ENDPOINT\_URL** `string` : Endpoint URL.
  * This field is required
* **APPDYNAMICS\_API\_KEY** `string` : API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AppDynamics` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `appdynamics.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: appdynamics-example
  namespace: odigos-system
spec:
  data:
    APPDYNAMICS_ACCOUNT_NAME: <Account Name>
    APPDYNAMICS_ENDPOINT_URL: <Endpoint URL>
    # Note: The commented fields below are optional.
    # APPDYNAMICS_APPLICATION_NAME: <Application Name>
  destinationName: appdynamics
  secretRef:
    name: appdynamics-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: appdynamics

---

apiVersion: v1
data:
  APPDYNAMICS_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: appdynamics-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f appdynamics.yaml
```


# AWS CloudWatch
Source: https://docs.odigos.io/oss/backends/awscloudwatch

Configuring the AWS CloudWatch backend (Managed)

### Getting Started

![cloudwatch](https://d15jtxgb40qetw.cloudfront.net/awscloudwatch.svg)

> [!WARNING]
>
> Make sure that you have the necessary `AWS Credentials` and permissions set up.\\
>
> AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PushLogs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:<Region>:<AccountId>:log-group:<LogGroupName>:log-stream:<LogStreamName>"
    },
    {
      "Sid": "CreateLogGroup",
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:<Region>:<AccountId>:log-group:<LogGroupName>"
    },
    {
      "Sid": "DescribeLogGroups",
      "Effect": "Allow",
      "Action": "logs:DescribeLogGroups",
      "Resource": "arn:aws:logs:<Region>:<AccountId>:log-group:<LogGroupName>"
    }
  ]
}
```

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **AWS\_CLOUDWATCH\_LOG\_GROUP\_NAME** `string` : Log Group Name. The group name of the CloudWatch Logs. If it does not exist it will be created automatically.
  * This field is required
* **AWS\_CLOUDWATCH\_LOG\_STREAM\_NAME** `string` : Log Stream Name. The stream name of the CloudWatch Logs. If it does not exist it will be created automatically.
  * This field is required
* **AWS\_CLOUDWATCH\_REGION** `string` : Region. The AWS region where the log stream is in. Region must be specified if it is not already set in the default credential chain.
  * This field is optional
* **AWS\_CLOUDWATCH\_ENDPOINT** `string` : Endpoint. The CloudWatch Logs service endpoint which the requests are forwarded to.
  * This field is optional
* **AWS\_CLOUDWATCH\_LOG\_RETENTION** `string` : Log Retention. LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. The value is in days, 0 = Never Expire. This value can also be changed later-on via the AWS console (UI) or AWS API, it's not a permanent decision once set. Also, the value here might not reflect the actual retention value in the destination if it was modified or some automatic policy is applied.
  * This field is optional and defaults to `0`
* **AWS\_CLOUDWATCH\_TAGS** `{ key: string; value: string; }[]` : Tags. Tags is the option to set tags for the CloudWatch Log Group. If specified, please add at most 50 tags. Keys must be between 1-128 characters and follow the pattern: (alphanumerics, whitespace, and \_.:/=+-!). Values must be between 1-256 characters and follow the pattern: (alphanumerics, whitespace, and \_.:/=+-!).
  * This field is optional
* **AWS\_CLOUDWATCH\_RAW\_LOG** `boolean` : Raw Log. If set to true, only the log message will be exported to CloudWatch Logs. This needs to be set to true for EMF logs.
  * This field is optional and defaults to `False`
* **AWS\_CLOUDWATCH\_METRICS\_NAMESPACE** `string` : Metrics Namespace. Customized CloudWatch metrics namespace. Metrics in different namespaces are isolated from each other. Please note this is not a cluster namespace.
  * This field is optional and defaults to `Odigos`
* **AWS\_CLOUDWATCH\_METRICS\_DIMENSION\_ROLLUP** `string` : Metrics Dimension Rollup. The option for metrics dimension rollup.
  * This field is optional and defaults to `NoDimensionRollup`
* **AWS\_CLOUDWATCH\_METRICS\_DETAILED** `boolean` : Detailed Metrics. Retain detailed datapoint values in exported metrics (e.g instead of exporting a quantile as a statistical value, preserve the quantile's population).
  * This field is optional and defaults to `False`
* **AWS\_CLOUDWATCH\_RETAIN\_INITIAL\_VALUE\_OF\_DELTA\_METRIC** `boolean` : Retain Initial Value of Delta Metric. This option specifies how the first value of a metric is handled. AWS EMF expects metric values to only contain deltas to the previous value. In the default case the first received value is therefor not sent to AWS but only used as a baseline for follow up changes to this metric. This is fine for high throughput metrics with stable labels. In this case it does not matter if the first value of this metric is discarded. However when your metric describes infrequent events or events with high label cardinality, then the exporter in default configuration would still drop the first occurrence of this metric. With this configuration value set to true the first value of all metrics will instead be send to AWS.
  * This field is optional and defaults to `False`

> [!NOTE]
>
> * If the log group and/or log stream are specified in an EMF log, that EMF log will be exported to that log group and/or log stream (i.e. ignores the log group and log stream defined in the configuration).
>   * The log group and log stream will also be created automatically if they do not already exist.
>   * Example of an EMF log with log group and log stream:
>   ```json theme={null}
>   {
>     "_aws": {
>       "Timestamp": 1574109732004,
>       "LogGroupName": "Foo",
>       "LogStreamName": "Bar",
>       "CloudWatchMetrics": [
>         {
>           "Namespace": "MyApp",
>           "Dimensions": [
>             [
>               "Operation"
>             ]
>           ],
>           "Metrics": [
>             {
>               "Name": "ProcessingLatency",
>               "Unit": "Milliseconds",
>               "StorageResolution": 60
>             }
>           ]
>         }
>       ]
>     },
>     "Operation": "Aggregator",
>     "ProcessingLatency": 100
>   }
>   ```
> * Resource ARNs (Amazon Resource Name (ARN) of the AWS resource running the collector) are currently not supported with the CloudWatch Logs Exporter.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AWS CloudWatch` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `cloudwatch.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: cloudwatch-example
  namespace: odigos-system
spec:
  data:
    AWS_CLOUDWATCH_LOG_GROUP_NAME: <Log Group Name>
    AWS_CLOUDWATCH_LOG_STREAM_NAME: <Log Stream Name>
    # Note: The commented fields below are optional.
    # AWS_CLOUDWATCH_REGION: <Region (options: [af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ap-southeast-3, ap-southeast-4, ap-southeast-5, ap-southeast-7, ca-central-1, ca-west-1, eu-central-1, eu-central-2, eu-north-1, eu-south-1, eu-south-2, eu-west-1, eu-west-2, eu-west-3, il-central-1, me-central-1, me-south-1, mx-central-1, sa-east-1, us-east-1, us-east-2, us-gov-east-1, us-gov-west-1, us-west-1, us-west-2])>
    # AWS_CLOUDWATCH_ENDPOINT: <Endpoint (options: [logs.af-south-1.amazonaws.com, logs.af-south-1.api.aws, logs.ap-east-1.amazonaws.com, logs.ap-east-1.api.aws, logs.ap-northeast-1.amazonaws.com, logs.ap-northeast-1.api.aws, logs.ap-northeast-2.amazonaws.com, logs.ap-northeast-2.api.aws, logs.ap-northeast-3.amazonaws.com, logs.ap-northeast-3.api.aws, logs.ap-south-1.amazonaws.com, logs.ap-south-1.api.aws, logs.ap-south-2.amazonaws.com, logs.ap-south-2.api.aws, logs.ap-southeast-1.amazonaws.com, logs.ap-southeast-1.api.aws, logs.ap-southeast-2.amazonaws.com, logs.ap-southeast-2.api.aws, logs.ap-southeast-3.amazonaws.com, logs.ap-southeast-3.api.aws, logs.ap-southeast-4.amazonaws.com, logs.ap-southeast-4.api.aws, logs.ap-southeast-5.amazonaws.com, logs.ap-southeast-5.api.aws, logs.ap-southeast-7.amazonaws.com, logs.ap-southeast-7.api.aws, logs.ca-central-1.amazonaws.com, logs.ca-central-1.api.aws, logs-fips.ca-central-1.amazonaws.com, logs.ca-west-1.amazonaws.com, logs.ca-west-1.api.aws, logs-fips.ca-west-1.amazonaws.com, logs.eu-central-1.amazonaws.com, logs.eu-central-1.api.aws, logs.eu-central-2.amazonaws.com, logs.eu-central-2.api.aws, logs.eu-north-1.amazonaws.com, logs.eu-north-1.api.aws, logs.eu-south-1.amazonaws.com, logs.eu-south-1.api.aws, logs.eu-south-2.amazonaws.com, logs.eu-south-2.api.aws, logs.eu-west-1.amazonaws.com, logs.eu-west-1.api.aws, logs.eu-west-2.amazonaws.com, logs.eu-west-2.api.aws, logs.eu-west-3.amazonaws.com, logs.eu-west-3.api.aws, logs.il-central-1.amazonaws.com, logs.il-central-1.api.aws, logs.me-central-1.amazonaws.com, logs.me-central-1.api.aws, logs.me-south-1.amazonaws.com, logs.me-south-1.api.aws, logs.mx-central-1.amazonaws.com, logs.mx-central-1.api.aws, logs.sa-east-1.amazonaws.com, logs.sa-east-1.api.aws, logs.us-east-1.amazonaws.com, logs-fips.us-east-1.amazonaws.com, logs.us-east-1.api.aws, logs.us-east-2.amazonaws.com, logs.us-east-2.api.aws, logs-fips.us-east-2.amazonaws.com, logs.us-gov-east-1.amazonaws.com, logs.us-gov-west-1.amazonaws.com, logs.us-west-1.amazonaws.com, logs.us-west-1.api.aws, logs-fips.us-west-1.amazonaws.com, logs.us-west-2.amazonaws.com, logs-fips.us-west-2.amazonaws.com, logs.us-west-2.api.aws])>
    # AWS_CLOUDWATCH_LOG_RETENTION: <Log Retention (default: 0) (options: [0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, 3653])>
    # AWS_CLOUDWATCH_TAGS: <Tags>
    # AWS_CLOUDWATCH_RAW_LOG: <Raw Log>
    # AWS_CLOUDWATCH_METRICS_NAMESPACE: <Metrics Namespace (default: Odigos)>
    # AWS_CLOUDWATCH_METRICS_DIMENSION_ROLLUP: <Metrics Dimension Rollup (default: NoDimensionRollup) (options: [NoDimensionRollup, SingleDimensionRollupOnly, ZeroAndSingleDimensionRollup])>
    # AWS_CLOUDWATCH_METRICS_DETAILED: <Detailed Metrics>
    # AWS_CLOUDWATCH_RETAIN_INITIAL_VALUE_OF_DELTA_METRIC: <Retain Initial Value of Delta Metric>
  destinationName: cloudwatch
  signals:
  - METRICS
  - LOGS
  type: cloudwatch
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f cloudwatch.yaml
```


# AWS S3
Source: https://docs.odigos.io/oss/backends/awss3

Configuring the AWS S3 backend (Managed)

### Getting Started

![s3](https://d15jtxgb40qetw.cloudfront.net/awss3.svg)

**Setting up AWS S3 Bucket**

If you haven't already, you need to create an S3 bucket to store your data. You can do it using the [AWS Management Console](https://console.aws.amazon.com/s3/home) or the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html).

For example, to create a bucket named `otel-data` in region `us-east-1` using the AWS CLI, you can run the following command:

```bash theme={null}
aws s3api create-bucket --bucket otel-data --region us-east-1
```

> [!WARNING]
>
> Make sure that you have the necessary `AWS Credentials` and permissions set up.\\
>
> AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<BucketName>", "arn:aws:s3:::<BucketName>/*"]
    }
  ]
}
```

You can do it with the aws cli.

* Replace `otel-data` with the name of your bucket if you used a different name.
* The policy name is `odigos-s3` in this example, you can use it or choose a different name. Take note of the policy ARN, you will need it in the next step.

```bash theme={null}
aws iam create-policy
--policy-name odigos-s3
--policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::otel-data", "arn:aws:s3:::otel-data/*"] } ] }'
```

Or use the aws management console to create the policy.

The last step is to configure this IAM policy to be assumed by the EKS cluster. Use the following cli command as template and adjust it according to your setup:

* make sure to replace the `--attach-policy-arn` value with the ARN of the policy you created in the previous step.
* use your EKS cluster name and region.

```bash theme={null}
eksctl create iamserviceaccount \
--name default \
--namespace odigos-system \
--cluster otel-cluster \
--region=us-east-1 \
--attach-policy-arn arn:aws:iam::123456789012:policy/odigos-s3 \
--override-existing-serviceaccounts \
--approve
```

**Format**

The aws s3 exporter supports two formats: `otlp_json` and `otlp_proto`.

Each batch of logs / metrics / spans will be written as a single s3 object in the specified bucket.
The content of the object will be the serialized data in the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto),
represented as either JSON or Protocol Buffers.

For example, the following JSON string represents a single span in `otlp_json` format, from Odigos demo application:

```json theme={null}
{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"service.name","value":{"stringValue":"inventory"}},{"key":"odigos.device","value":{"stringValue":"python"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"k8s.namespace.name","value":{"stringValue":"default"}},{"key":"k8s.deployment.name","value":{"stringValue":"inventory"}},{"key":"k8s.pod.name","value":{"stringValue":"inventory-7b87f7566c-2d6fq"}},{"key":"container.name","value":{"stringValue":"inventory"}},{"key":"k8s.node.name","value":{"stringValue":"ip-192-168-4-178.ec2.internal"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9ee5a0de5802d3f1e7f93440f892509c","spanId":"88a4bfb5bfcaf297","parentSpanId":"f9d7bb9e9243df06","name":"/buy","kind":2,"startTimeUnixNano":"1707600957298456744","endTimeUnixNano":"1707600958303042894","attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"8080"}},{"key":"http.host","value":{"stringValue":"inventory:8080"}},{"key":"http.target","value":{"stringValue":"/buy?id=10"}},{"key":"net.peer.ip","value":{"stringValue":"192.168.13.194"}},{"key":"http.user_agent","value":{"stringValue":"Java/17.0.9"}},{"key":"net.peer.port","value":{"intValue":"49420"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/buy"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}
```

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **S3\_BUCKET** `string` : Bucket Name. The name of the bucket where the data will be stored
  * This field is required
* **S3\_REGION** `string` : Bucket Region. The AWS region where the bucket is located
  * This field is required
* **S3\_PARTITION** `string` : Time granularity of S3 Bucket (Deprecated, use Partition Format). Deprecated field for time granularity. Use S3\_PARTITION\_FORMAT for custom formats.
  * This field is optional and defaults to `minute`
* **S3\_PARTITION\_FORMAT** `string` : S3 Partition Format. Custom partition format using strftime syntax (e.g., '%Y/%m/%d/%H/%M' for minute-level, '%Y/%m/%d/%H' for hour-level). If not specified, S3\_PARTITION will be used.
  * This field is optional
  * Example: `%Y/%m/%d/%H/%M`
* **S3\_MARSHALER** `string` : Marshaller (Data Format). The format in which the data will be encoded. It can be either `otlp_json` or `otlp_proto`. Default is `otlp_json`
  * This field is required and defaults to `otlp_json`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AWS S3` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `s3.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: s3-example
  namespace: odigos-system
spec:
  data:
    S3_BUCKET: <Bucket Name>
    S3_MARSHALER: '<Marshaller (Data Format) (default: otlp_json) (options: [otlp_json,
      otlp_proto])>'
    S3_REGION: '<Bucket Region (options: [af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2,
      ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ap-southeast-3,
      ap-southeast-4, ap-southeast-5, ap-southeast-7, ca-central-1, ca-west-1, eu-central-1,
      eu-central-2, eu-north-1, eu-south-1, eu-south-2, eu-west-1, eu-west-2, eu-west-3,
      il-central-1, me-central-1, me-south-1, mx-central-1, sa-east-1, us-east-1,
      us-east-2, us-gov-east-1, us-gov-west-1, us-west-1, us-west-2])>'
    # Note: The commented fields below are optional.
    # S3_PARTITION: <Time granularity of S3 Bucket (Deprecated, use Partition Format) (default: minute) (options: [minute, hour])>
    # S3_PARTITION_FORMAT: <S3 Partition Format>
  destinationName: s3
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: s3
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f s3.yaml
```


# AWS X-Ray
Source: https://docs.odigos.io/oss/backends/awsxray

Configuring the AWS X-Ray backend (Managed)

### Getting Started

![xray](https://d15jtxgb40qetw.cloudfront.net/awsxray.svg)

> [!WARNING]
>
> Make sure that you have the necessary `AWS Credentials` and permissions set up.\\
>
> AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "xray:PutTraceSegments",
      "xray:PutTelemetryRecords"
    ],
    "Resource": [
      "*"
    ]
  }
}
```

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **AWS\_XRAY\_REGION** `string` : Region. Send segments to AWS X-Ray service in a specific region.
  * This field is optional
* **AWS\_XRAY\_ENDPOINT** `string` : Endpoint. Optionally override the default X-Ray service endpoint.
  * This field is optional
* **AWS\_XRAY\_PROXY\_ADDRESS** `string` : Proxy Address. Upload segments to AWS X-Ray through a proxy.
  * This field is optional
* **AWS\_XRAY\_RESOURCE\_ARN** `string` : Resource ARN. Amazon Resource Name (ARN) of the AWS resource running the collector.
  * This field is optional
* **AWS\_XRAY\_ROLE\_ARN** `string` : Role ARN. IAM role to upload segments to a different account.
  * This field is optional
* **AWS\_XRAY\_EXTERNAL\_ID** `string` : External ID. Shared identitier used when assuming an IAM role in an external AWS account.
  * This field is optional
* **AWS\_XRAY\_LOG\_GROUPS** `string[]` : Log Groups. List of log group names for CloudWatch.
  * This field is optional
* **AWS\_XRAY\_DISABLE\_SSL** `boolean` : Disable SSL Verification. Enable or disable TLS certificate verification.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_LOCAL\_MODE** `boolean` : Local Mode. Local mode to skip EC2 instance metadata check.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_TELEMETRY\_ENABLED** `boolean` : Telemetry - Enabled. Whether telemetry collection is enabled at all.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_TELEMETRY\_INCLUDE\_METADATA** `boolean` : Telemetry - Include Metadata. Whether to include metadata in the telemetry (InstanceID, Hostname, ResourceARN)
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_TELEMETRY\_HOSTNAME** `string` : Telemetry - Hostname. Sets the Hostname included in the telemetry.
  * This field is optional
* **AWS\_XRAY\_TELEMETRY\_INSTANCE\_ID** `string` : Telemetry - Instance ID. Sets the InstanceID included in the telemetry.
  * This field is optional
* **AWS\_XRAY\_TELEMETRY\_RESOURCE\_ARN** `string` : Telemetry - Resource ARN. Sets the Amazon Resource Name (ARN) included in the telemetry.
  * This field is optional
* **AWS\_XRAY\_TELEMETRY\_CONTRIBUTORS** `string[]` : Telemetry - Contributors. List of X-Ray component IDs contributing to the telemetry (ex. for multiple X-Ray receivers: awsxray/1, awsxray/2)
  * This field is optional
* **AWS\_XRAY\_INDEX\_ALL\_ATTRIBUTES** `boolean` : Index All Attributes. Enable or disable conversion of all OpenTelemetry attributes to X-Ray annotations.
  * This field is optional and defaults to `False`
* **AWS\_XRAY\_INDEXED\_ATTRIBUTES** `string[]` : Indexed Attributes. List of attribute names to be converted to X-Ray annotations.
  * This field is optional

> [!NOTE]
>
> **Traces and logs correlation**
>
> AWS X-Ray can be integrated with CloudWatch Logs to correlate traces with logs.
>
> For this integration to work, the X-Ray segments must have the AWS Property `cloudwatch_logs` set. This property is set using the AWS X-Ray exporter with the following values that are evaluated in this order:
>
> 1. `aws.log.group.arns` resource attribute.
> 2. `aws.log.group.names` resource attribute.
> 3. `aws_log_groups` configuration property.
>
> `aws.log.group.arns` and `aws.log.group.names` are slice resource attributes, alternatively those resource attributes can be set using the [OTEL\_RESOURCE\_ATTRIBUTES environment variable](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md#specifying-resource-information-via-an-environment-variable).
>
> To set multiple log group names / log group arns, you can use `&` to separate them.\\
>
> For example 3 log groups: `log-group1`, `log-group2`, and `log-group3` are set in the following command:
>
> ```bash theme={null}
> export OTEL_RESOURCE_ATTRIBUTES="aws.log.group.names=log-group1&log-group2&log-group3"
> ```

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `AWS X-Ray` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `xray.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: xray-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: xray
  signals:
  - TRACES
  type: xray
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f xray.yaml
```


# Axiom
Source: https://docs.odigos.io/oss/backends/axiom

Configuring the Axiom backend (Managed)

### Getting Started

![axiom](https://d15jtxgb40qetw.cloudfront.net/axiom.svg)

1. [Register](https://app.axiom.co/register)/[Login](https://app.axiom.co/login) to Axiom.
2. Navigate to the [Axiom Datasets](https://app.axiom.co/settings/datasets) page, create a dataset and copy its name.
3. Navigate to the [Axiom API Tokens](https://app.axiom.co/settings/api-tokens) page, and generate a new API Token.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **AXIOM\_DATASET** `string` : Dataset.
  * This field is required
* **AXIOM\_API\_TOKEN** `string` : Axiom API token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Axiom` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `axiom.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: axiom-example
  namespace: odigos-system
spec:
  data:
    AXIOM_DATASET: <Dataset>
  destinationName: axiom
  secretRef:
    name: axiom-secret
  signals:
  - TRACES
  - LOGS
  type: axiom

---

apiVersion: v1
data:
  AXIOM_API_TOKEN: <Base64 Axiom API token>
kind: Secret
metadata:
  name: axiom-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f axiom.yaml
```


# Azure Blob Storage
Source: https://docs.odigos.io/oss/backends/azureblob

Configuring the Azure Blob Storage backend (Managed)

### Getting Started

![azureblob](https://d15jtxgb40qetw.cloudfront.net/blobstorage.svg)

Odigos exports data to Azure Blob Storage in [OTLP format](https://opentelemetry.io/docs/specs/otlp/). Data can be exported either in JSON format or in binary format (protobuf).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **AZURE\_BLOB\_ACCOUNT\_NAME** `string` : Account Name.
  * This field is required
* **AZURE\_BLOB\_CONTAINER\_NAME** `string` : Container Name.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Azure Blob Storage` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `azureblob.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: azureblob-example
  namespace: odigos-system
spec:
  data:
    AZURE_BLOB_ACCOUNT_NAME: <Account Name>
    AZURE_BLOB_CONTAINER_NAME: <Container Name>
  destinationName: azureblob
  signals:
  - TRACES
  - LOGS
  type: azureblob
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f azureblob.yaml
```


# Azure Monitor
Source: https://docs.odigos.io/oss/backends/azuremonitor

Configuring the Azure Monitor backend (Managed)

### Getting Started

![azuremonitor](https://d15jtxgb40qetw.cloudfront.net/azuremonitor.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **AZURE\_MONITOR\_CONNECTION\_STRING** `string` : Connection String. The Connection String is the recommended method for configuring the exporter, aligning with Azure Monitor's best practices. If you don't have a Connection String, you can create one in the Azure Portal (after creating an Application Insights resource).
  * This field is optional
* **AZURE\_MONITOR\_INSTRUMENTATION\_KEY** `string` : Instrumentation Key. Application Insights instrumentation key, which can be found in the Application Insights resource in the Azure Portal. While it is currently supported, its use is discouraged and it is slated for deprecation.
  * This field is optional
* **AZURE\_MONITOR\_ENDPOINT** `string` : Endpoint. The endpoint URL where data will be submitted. While this option remains available, it is important to note that the use of the Connection String is recommended, as it encompasses the endpoint information. The direct configuration of the endpoint is considered to be on a deprecation path.
  * This field is optional

> [!WARNING]
>
> Do not configure the `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable. It will override this configuration!

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Azure Monitor` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `azuremonitor.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: azuremonitor-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: azuremonitor
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: azuremonitor
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f azuremonitor.yaml
```


# Better Stack
Source: https://docs.odigos.io/oss/backends/betterstack

Configuring the Better Stack backend (Managed)

### Getting Started

![betterstack](https://d15jtxgb40qetw.cloudfront.net/betterstack.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **BETTERSTACK\_TOKEN** `string` : Better Stack Source Token.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint for metrics is `https://in-otel.logs.betterstack.com/metrics`
> * The endpoint for logs is `https://in-otel.logs.betterstack.com:443`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Better Stack` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `betterstack.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: betterstack-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: betterstack
  secretRef:
    name: betterstack-secret
  signals:
  - METRICS
  - LOGS
  type: betterstack

---

apiVersion: v1
data:
  BETTERSTACK_TOKEN: <Base64 Better Stack Source Token>
kind: Secret
metadata:
  name: betterstack-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f betterstack.yaml
```


# Bonree ONE
Source: https://docs.odigos.io/oss/backends/bonree

Configuring the Bonree ONE backend (Managed)

### Getting Started

![bonree](https://d15jtxgb40qetw.cloudfront.net/bonree.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **BONREE\_ENDPOINT** `string` : OTLP HTTP Endpoint. APM controller address, obtained from the probe deployment page.
  * This field is required
* **BONREE\_ACCOUNT\_ID** `string` : Account ID. Obtained from the probe deployment page.
  * This field is required
* **BONREE\_ENVIRONMENT\_ID** `string` : Environment ID. The default environment does not need to be configured.
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Bonree ONE` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `bonree.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: bonree-example
  namespace: odigos-system
spec:
  data:
    BONREE_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: bonree
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: bonree-secret
  signals:
  - TRACES
  - METRICS
  type: bonree

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   BONREE_ACCOUNT_ID: <Base64 Account ID>
#   BONREE_ENVIRONMENT_ID: <Base64 Environment ID>
# kind: Secret
# metadata:
#   name: bonree-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f bonree.yaml
```


# Causely
Source: https://docs.odigos.io/oss/backends/causely

Configuring the Causely backend (Managed)

### Getting Started

![causely](https://d15jtxgb40qetw.cloudfront.net/causely.svg)

This destination is for the Causely Mediator Service, so you will need to have a Causely instance running and accessible from the k8s cluster running Odigos.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **CAUSELY\_URL** `string` : Endpoint. The endpoint URL is the combined `<protocol>://<hostname>:<port>` to access your Causely Mediator service. Protocol should be `http`; using `https` or omitting it will automatically be converted to `http`. Hostname should typically follow the format: `mediator.<namespace>`. Namespace is the k8s namespace where the Causely Mediator service is deployed. Port is optional and defaults to the default OTLP gRPC port `4317`
  * This field is required
  * Example: `http://mediator.causely:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Causely` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `causely.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: causely-example
  namespace: odigos-system
spec:
  data:
    CAUSELY_URL: <Endpoint>
  destinationName: causely
  signals:
  - TRACES
  - METRICS
  type: causely
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f causely.yaml
```


# Checkly
Source: https://docs.odigos.io/oss/backends/checkly

Configuring the Checkly backend (Managed)

### Getting Started

![checkly](https://d15jtxgb40qetw.cloudfront.net/checkly.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **CHECKLY\_ENDOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`.
  * This field is required
* **CHECKLY\_API\_KEY** `string` : API Key.
  * This field is required

> [!WARNING]
>
> Odigos will only filter and export traces to Checkly if their `trace_state` includes the tag `checkly="true"`, which is added by a Checkly SDK.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Checkly` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `checkly.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: checkly-example
  namespace: odigos-system
spec:
  data:
    CHECKLY_ENDOINT: <OTLP gRPC Endpoint>
  destinationName: checkly
  secretRef:
    name: checkly-secret
  signals:
  - TRACES
  type: checkly

---

apiVersion: v1
data:
  CHECKLY_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: checkly-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f checkly.yaml
```


# Chronosphere
Source: https://docs.odigos.io/oss/backends/chronosphere

Configuring the Chronosphere backend (Managed)

### Getting Started

![chronosphere](https://d15jtxgb40qetw.cloudfront.net/chronosphere.svg)

Chronosphere is a monitoring and observability platform that provides a scalable, reliable, and cost-effective way to collect, store, and analyze metrics and traces.
To send data from Odigos to Chronosphere, [you need to have an account with Chronosphere](https://chronosphere.io/demo-request/) and configure the Chronosphere destination in Odigos with the following details:

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **CHRONOSPHERE\_DOMAIN** `string` : Chronosphere Company Domain. Company domain in Chronosphere, can be found in the URL of the Chronosphere UI. For example, if your URL is `https://demo-sandbox.chronosphere.io/`, then your company domain is `demo-sandbox`
  * This field is required
* **CHRONOSPHERE\_API\_TOKEN** `string` : API Token. API token generated from your Chronosphere service account in the Chronosphere UI, you should login to your Chronosphere admin account click the `Platform` menu on the left side of the screen, then click `Service Accounts` and create a new service account
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Chronosphere` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `chronosphere.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: chronosphere-example
  namespace: odigos-system
spec:
  data:
    CHRONOSPHERE_DOMAIN: <Chronosphere Company Domain>
  destinationName: chronosphere
  secretRef:
    name: chronosphere-secret
  signals:
  - TRACES
  - METRICS
  type: chronosphere

---

apiVersion: v1
data:
  CHRONOSPHERE_API_TOKEN: <Base64 API Token>
kind: Secret
metadata:
  name: chronosphere-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f chronosphere.yaml
```


# ClickHouse
Source: https://docs.odigos.io/oss/backends/clickhouse

Configuring the ClickHouse backend (Self-Hosted)

### Getting Started

![clickhouse](https://d15jtxgb40qetw.cloudfront.net/clickhouse.svg)

ClickHouse is very popular database for storing telemetry data and running efficient queries on it.

**Use Cases**

* ClickHouse can handle all 3 signals (metrics, logs, traces) in one single database. less overhead to manage multiple databases.
* ClickHouse is a columnar database, which is optimized for time-series immutable data like OpenTelemetry telemetry data.
* ClickHouse is a distributed database, which can scale horizontally and handle large amounts of data with efficient storage and query performance.
* ClickHouse is open-source and has a large community.

**Prerequisites**

* To use ClickHouse as a destination in Odigos, you need to have a ClickHouse deployment running somewhere and accessible from cluster where Odigos is running.
* You should know the service endpoint where ClickHouse listens for incoming client connections.
* If you haven't already, create a database in ClickHouse where you want to store the telemetry data. the default database name is `otel` (configurable). To create it, run the following SQL command: `CREATE DATABASE otel;`
* Understanding of how to maintain, scale, and optimize your self-hosted ClickHouse deployment, as well as how to fine tune setting based on your queries and use case.

**Schema**

When using the ClickHouse destination with Odigos, logs metrics and traces are going to be "INSERT INTO" the ClickHouse database.

It is important to understand what schema is used, e.g. what table names are used, column names, data types, etc.
Then you can run queries on this data, or modify and optimize it for your specific use case.

There are 2 modes of operation for ClickHouse destination in Odigos:

**Create Schema**

In this option, the schema will be automatically created by odigos with reasonable defaults.

The benefit of this option is that you can see the value fast, without needing to apply and manage any schema yourself.
The downside is that the schema may not be optimized for your specific use case, and may make changes more complicated.

To use it:

* Odigos UI - When adding a new ClickHouse destination, select the `Create Scheme` checkbox field.
* Destination K8s Manifest - Set the `CLICKHOUSE_CREATE_SCHEME` setting to value `true`.

The schema which will be used by default can be found [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/a80fdb94cecb1cd3c1fbd7fbe3000396cd588970/exporter/clickhouseexporter/example/default_ddl).

* Indexes - The default schema includes indexes for the data, including trace\_id, resource and scope attributes, span/metric/log attributes etc.
* TTL is set to 180 days. This means data will be kept in the database for this period of time and then deleted.
* Partitioning - The default schema includes partitioning by day, which means data is stored in separate partitions for each day.
* Order By - Optimized for trace queries on service\_name + span\_name + time, for logs on service\_name + time, and for metrics on service\_name + metric\_name + attributes + time.

This option is not recommended for production workloads:

* You may want to adjust the settings to better fit your use case, scale performance requirements, and costs.
* Each new exporter will attempt to create the schema, which is less robust and harder to manage than a pre-created schema.

**Self Managed Schema**

With this option, you are responsible for creating and managing the schema yourself.

To use it:

* Odigos UI - Unselect the `Create Scheme` checkbox field.
* Destination K8s Manifest - Set the `CLICKHOUSE_CREATE_SCHEME` setting to value `false`.

The benefit of this option is that you have full control over the schema, and can optimize it for your specific use case.

How it works?

* Browse to the [default DDL example](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/a80fdb94cecb1cd3c1fbd7fbe3000396cd588970/exporter/clickhouseexporter/example/default_ddl).
* Copy the `sql` files to your local machine.
* Make any changes you need to the schema.
* Run the SQL files in your ClickHouse database to create the schema.

This option is recommended for production workloads:

* You can optimize the schema for your specific use case, scale performance requirements, and costs.
* You can manage the schema in a version control system, and apply changes in a controlled way.
* Applying changes to the schema is more robust and easier to manage than attempting to create it on the fly with each new connection.

Important Settings:

* Indexes - You may want to add or remove indexes based on your queries, to optimize performance and costs.
* TTL - You may want to adjust the TTL based on your retention policy. If you need traces for auditing purposes, you may extend it to 365 days. For high-throughput, low-latency systems, you might want to reduce it to 90 days or even less.
* Partitioning - partitioning by day is a good default, but in high throughput systems you may want to partition by hour or even minute. You may also consider partitioning by service\_name, or other attributes.
* Order By - You may want to adjust the order by clause based on your queries, so that common columns are used first in the query, to optimize performance.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **CLICKHOUSE\_ENDPOINT** `string` : Endpoint. The ClickHouse endpoint is the URL where the ClickHouse server is listening for incoming connections.
  * This field is required
  * Example: `http://host:port`
* **CLICKHOUSE\_USERNAME** `string` : Username. If Clickhouse Authentication is used, provide the username
  * This field is optional
* **CLICKHOUSE\_PASSWORD** `string` : Password. If Clickhouse Authentication is used, provide the password
  * This field is optional
* **CLICKHOUSE\_CREATE\_SCHEME** `boolean` : Create Schema. Should the destination create the schema for you? Set to `false` if you manage your own schema, or `true` to have Odigos create the schema for you
  * This field is required and defaults to `True`
* **CLICKHOUSE\_DATABASE\_NAME** `string` : Database Name. The name of the Clickhouse Database where the telemetry data will be stored. The Database will not be created when not exists, so make sure you have created it before
  * This field is required and defaults to `otel`
* **CLICKHOUSE\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **CLICKHOUSE\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`
* **CLICKHOUSE\_USE\_CUSTOM\_CA** `boolean` : Use Custom CA Certificate. Use a custom CA certificate instead of the system root CA
  * This field is optional and defaults to `False`
* **CLICKHOUSE\_CA\_PEM** `string` : Certificate Authority (CA PEM). CA certificate in PEM format to verify the server
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **CLICKHOUSE\_TRACES\_TABLE** `string` : Traces Table. Name of the ClickHouse Table to use for storing trace spans. This name should be used in span queries
  * This field is required and defaults to `otel_traces`
* **CLICKHOUSE\_LOGS\_TABLE** `string` : Logs Table. Name of the ClickHouse Table to use for storing logs
  * This field is required and defaults to `otel_logs`
* **CLICKHOUSE\_METRICS\_TABLE\_EXP\_HISTOGRAM** `string` : Metrics Table - Exp. Histogram. Name of the table for storing exponential histogram metrics
  * This field is required and defaults to `otel_metrics_exponential_histogram`
* **CLICKHOUSE\_METRICS\_TABLE\_GAUGE** `string` : Metrics Table - Gauge. Name of the table for storing gauge metrics
  * This field is required and defaults to `otel_metrics_gauge`
* **CLICKHOUSE\_METRICS\_TABLE\_HISTOGRAM** `string` : Metrics Table - Histogram. Name of the table for storing histogram metrics
  * This field is required and defaults to `otel_metrics_histogram`
* **CLICKHOUSE\_METRICS\_TABLE\_SUM** `string` : Metrics Table - Sum. Name of the table for storing sum metrics
  * This field is required and defaults to `otel_metrics_sum`
* **CLICKHOUSE\_METRICS\_TABLE\_SUMMARY** `string` : Metrics Table - Summary. Name of the table for storing summary metrics
  * This field is required and defaults to `otel_metrics_summary`
* **HYPERDX\_LOG\_NORMALIZER** `boolean` : Enable Log Normalizer. Enable HyperDX log normalization processor. This parses JSON from log bodies, infers severity levels, and normalizes log attributes for better querying.
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Clickhouse` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `clickhouse.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: clickhouse-example
  namespace: odigos-system
spec:
  data:
    CLICKHOUSE_CREATE_SCHEME: '<Create Schema (default: True)>'
    CLICKHOUSE_DATABASE_NAME: '<Database Name (default: otel)>'
    CLICKHOUSE_ENDPOINT: <Endpoint>
    CLICKHOUSE_LOGS_TABLE: '<Logs Table (default: otel_logs)>'
    CLICKHOUSE_METRICS_TABLE_EXP_HISTOGRAM: '<Metrics Table - Exp. Histogram (default:
      otel_metrics_exponential_histogram)>'
    CLICKHOUSE_METRICS_TABLE_GAUGE: '<Metrics Table - Gauge (default: otel_metrics_gauge)>'
    CLICKHOUSE_METRICS_TABLE_HISTOGRAM: '<Metrics Table - Histogram (default: otel_metrics_histogram)>'
    CLICKHOUSE_METRICS_TABLE_SUM: '<Metrics Table - Sum (default: otel_metrics_sum)>'
    CLICKHOUSE_METRICS_TABLE_SUMMARY: '<Metrics Table - Summary (default: otel_metrics_summary)>'
    CLICKHOUSE_TRACES_TABLE: '<Traces Table (default: otel_traces)>'
    # Note: The commented fields below are optional.
    # CLICKHOUSE_USERNAME: <Username>
    # CLICKHOUSE_TLS_ENABLED: <Enable TLS>
    # CLICKHOUSE_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
    # CLICKHOUSE_USE_CUSTOM_CA: <Use Custom CA Certificate>
    # HYPERDX_LOG_NORMALIZER: <Enable Log Normalizer>
  destinationName: clickhouse
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: clickhouse-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: clickhouse

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   CLICKHOUSE_CA_PEM: <Base64 Certificate Authority (CA PEM)>
#   CLICKHOUSE_PASSWORD: <Base64 Password>
# kind: Secret
# metadata:
#   name: clickhouse-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f clickhouse.yaml
```


# Coralogix
Source: https://docs.odigos.io/oss/backends/coralogix

Configuring the Coralogix backend (Managed)

### Getting Started

![coralogix](https://d15jtxgb40qetw.cloudfront.net/coralogix.svg)

1. Get an [API Key](https://coralogix.com/docs/send-your-data-api-key/).
2. Obtain the [Domain](https://coralogix.com/docs/coralogix-domain/) determined by your Coralogix Account URL.
3. Follow the [Application and Subsystem Names guide](https://coralogix.com/docs/application-and-subsystem-names/) to learn how configure the application and subsystem names for your Coralogix account.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **CORALOGIX\_PRIVATE\_KEY** `string` : Send-Your-Data API Key. The Send-Your-Data API Key is used to authenticate the data sent to Coralogix.
  * This field is required
* **CORALOGIX\_DOMAIN** `string` : Domain. The Coralogix domain to which you want to send the data.
  * This field is required
* **CORALOGIX\_APPLICATION\_NAME** `string` : Application Name. The name of the application that sends the data to Coralogix.
  * This field is required
* **CORALOGIX\_SUBSYSTEM\_NAME** `string` : Subsystem Name. The name of the subsystem that sends the data to Coralogix.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Coralogix` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `coralogix.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: coralogix-example
  namespace: odigos-system
spec:
  data:
    CORALOGIX_APPLICATION_NAME: <Application Name>
    CORALOGIX_DOMAIN: '<Domain (options: [coralogix.com, eu2.coralogix.com, coralogix.us,
      cx498.coralogix.com, coralogix.in, coralogixsg.com])>'
    CORALOGIX_SUBSYSTEM_NAME: <Subsystem Name>
  destinationName: coralogix
  secretRef:
    name: coralogix-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: coralogix

---

apiVersion: v1
data:
  CORALOGIX_PRIVATE_KEY: <Base64 Send-Your-Data API Key>
kind: Secret
metadata:
  name: coralogix-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f coralogix.yaml
```


# Dash0
Source: https://docs.odigos.io/oss/backends/dash0

Configuring the Dash0 backend (Managed)

### Getting Started

![dash0](https://d15jtxgb40qetw.cloudfront.net/dash0.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **DASH0\_ENDPOINT** `string` : Dash0 OTLP gRPC Endpoint. The format is `host:port`. Host is required, located in Dash0 UI (OpenTelemetry Collector). Port is optional and defaults to the default OTLP gRPC port `4317`.
  * This field is required
* **DASH0\_TOKEN** `string` : Dash0 Bearer Token. Located in Dash0 UI (OpenTelemetry Collector).
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Dash0` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `dash0.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: dash0-example
  namespace: odigos-system
spec:
  data:
    DASH0_ENDPOINT: <Dash0 OTLP gRPC Endpoint>
  destinationName: dash0
  secretRef:
    name: dash0-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: dash0

---

apiVersion: v1
data:
  DASH0_TOKEN: <Base64 Dash0 Bearer Token>
kind: Secret
metadata:
  name: dash0-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f dash0.yaml
```


# Datadog
Source: https://docs.odigos.io/oss/backends/datadog

Configuring the Datadog backend (Managed)

### Getting Started

![datadog](https://d15jtxgb40qetw.cloudfront.net/datadog.svg)

On the left sidebar, click on your username and then click on **Organization Settings**

![Organization settings menu](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/datadog1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1fed9ecd072cd3bc44a331e79fef38e0)

Click on **API Keys** and then click on **New Key** and give it a name.

![New API key screen](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/datadog2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=f8d85799fc464f727f77fb89ad458638)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **DATADOG\_API\_KEY** `string` : API Key. The API key for your Datadog account.
  * This field is required
* **DATADOG\_SITE** `string` : Site. The datadog site you are using (the url of the datadog site you are using).
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Datadog` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `datadog.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: datadog-example
  namespace: odigos-system
spec:
  data:
    DATADOG_SITE: '<Site (options: [us3.datadoghq.com, datadoghq.com, us5.datadoghq.com,
      datadoghq.eu, ddog-gov.com, ap1.datadoghq.com])>'
  destinationName: datadog
  secretRef:
    name: datadog-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: datadog

---

apiVersion: v1
data:
  DATADOG_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: datadog-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f datadog.yaml
```


# Dynamic Destination
Source: https://docs.odigos.io/oss/backends/dynamic

Configuring the Dynamic Destination backend (Self-Hosted)

### Getting Started

![dynamic](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg)

Use Dynamic Destination when you need a collector exporter configuration that is not covered by a built-in destination type (for example, OTLP with client mTLS via freeform YAML).

> [!NOTE]
>
> Environment variable placeholders such as `${OTLP_CA_PEM}` in `DYNAMIC_CONFIGURATION_DATA` are expanded by the gateway collector from pod environment variables. To inject those values, create a Secret in the **same namespace** as the Destination (the Odigos system namespace) whose keys match the placeholder names, and set `spec.secretRef.name` on the Destination. Without `secretRef`, the instrumentor will not mount the Secret into the gateway and the placeholders remain unset. Use unique env var names across dynamic destinations — they are mounted into the shared gateway without a destination prefix (built-in destinations are automatically scoped).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ✅ Profiles

* **DYNAMIC\_DESTINATION\_TYPE** `string` : Destination Type. The type of OpenTelemetry Collector Exporter to use for the destination.
  * This field is required
  * Example: `otlp`
* **DYNAMIC\_CONFIGURATION\_DATA** `string` : Config. The raw YAML config for the Collector Exporter for this destination. To inject secrets (e.g. TLS PEMs), use \$ placeholders and set Destination.spec.secretRef to a Secret in the same namespace whose keys match those variable names. Use unique env var names across dynamic destinations — they are mounted into the shared gateway without a destination prefix.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Dynamic Destination` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `dynamic.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: dynamic-example
  namespace: odigos-system
spec:
  data:
    DYNAMIC_CONFIGURATION_DATA: <Config>
    DYNAMIC_DESTINATION_TYPE: <Destination Type>
  destinationName: dynamic
  signals:
  - TRACES
  - METRICS
  - LOGS
  - PROFILES
  type: dynamic
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f dynamic.yaml
```


# Dynatrace
Source: https://docs.odigos.io/oss/backends/dynatrace

Configuring the Dynatrace backend (Managed)

### Getting Started

![dynatrace](https://d15jtxgb40qetw.cloudfront.net/dynatrace.svg)

On the left sidebar click on `Apps`, then under `Manage` select `Access Tokens`:

![Obtaining access token](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/dynatrace1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=9a53c177bf76f5c7bb929c0c0dfdc009)

Make sure to add the required capabilities for the access tokens (e.g `Ingest OpenTelemetry traces` for traces).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **DYNATRACE\_URL** `string` : Tenant URL.
  * This field is required
  * Example: `https://{your-environment-id}.live.dynatrace.com`
* **DYNATRACE\_API\_TOKEN** `string` : API Access Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Dynatrace` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `dynatrace.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: dynatrace-example
  namespace: odigos-system
spec:
  data:
    DYNATRACE_URL: <Tenant URL>
  destinationName: dynatrace
  secretRef:
    name: dynatrace-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: dynatrace

---

apiVersion: v1
data:
  DYNATRACE_API_TOKEN: <Base64 API Access Token>
kind: Secret
metadata:
  name: dynatrace-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f dynatrace.yaml
```


# Elastic APM
Source: https://docs.odigos.io/oss/backends/elasticapm

Configuring the Elastic APM backend (Managed)

### Getting Started

![elasticapm](https://d15jtxgb40qetw.cloudfront.net/elasticapm.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **ELASTIC\_APM\_SERVER\_ENDPOINT** `string` : elastic APM server endpoint.
  * This field is required
* **ELASTIC\_APM\_SECRET\_TOKEN** `string` : elastic APM server token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Elastic APM` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `elasticapm.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: elasticapm-example
  namespace: odigos-system
spec:
  data:
    ELASTIC_APM_SERVER_ENDPOINT: <elastic APM server endpoint>
  destinationName: elasticapm
  secretRef:
    name: elasticapm-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: elasticapm

---

apiVersion: v1
data:
  ELASTIC_APM_SECRET_TOKEN: <Base64 elastic APM server token>
kind: Secret
metadata:
  name: elasticapm-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f elasticapm.yaml
```


# Elasticsearch
Source: https://docs.odigos.io/oss/backends/elasticsearch

Configuring the Elasticsearch backend (Self-Hosted)

### Getting Started

![elasticsearch](https://d15jtxgb40qetw.cloudfront.net/elasticsearch.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **ELASTICSEARCH\_URL** `string` : Elasticsearch URL. Elasticsearch endpoint. Port defaults to `9200` if not specified
  * This field is required
  * Example: `http://host:port`
* **ES\_TRACES\_INDEX** `string` : Traces Index. The name of the index where traces will be stored
  * This field is optional and defaults to `trace_index`
* **ES\_LOGS\_INDEX** `string` : Logs Index. The name of the index where logs will be stored.
  * This field is optional and defaults to `log_index`
* **ELASTICSEARCH\_BASIC\_AUTH\_ENABLED** `boolean` : Enable HTTP Basic Authentication.
  * This field is optional and defaults to `False`
* **ELASTICSEARCH\_USERNAME** `string` : Username. Username used for HTTP Basic Authentication
  * This field is optional
* **ELASTICSEARCH\_PASSWORD** `string` : Password. Password used for HTTP Basic Authentication
  * This field is optional
* **ELASTICSEARCH\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection (Transport Layer Security)
  * This field is optional and defaults to `False`
* **ELASTICSEARCH\_CA\_PEM** `string` : CA Certificate. When using TLS, provide the CA certificate to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Elasticsearch` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `elasticsearch.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: elasticsearch-example
  namespace: odigos-system
spec:
  data:
    ELASTICSEARCH_URL: <Elasticsearch URL>
    # Note: The commented fields below are optional.
    # ES_TRACES_INDEX: <Traces Index (default: trace_index)>
    # ES_LOGS_INDEX: <Logs Index (default: log_index)>
    # ELASTICSEARCH_BASIC_AUTH_ENABLED: <Enable HTTP Basic Authentication>
    # ELASTICSEARCH_USERNAME: <Username>
    # ELASTICSEARCH_TLS_ENABLED: <Enable TLS>
    # ELASTICSEARCH_CA_PEM: <CA Certificate>
  destinationName: elasticsearch
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: elasticsearch-secret
  signals:
  - TRACES
  - LOGS
  type: elasticsearch

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   ELASTICSEARCH_PASSWORD: <Base64 Password>
# kind: Secret
# metadata:
#   name: elasticsearch-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f elasticsearch.yaml
```


# Gigapipe
Source: https://docs.odigos.io/oss/backends/gigapipe

Configuring the Gigapipe backend (Managed)

### Getting Started

![qryn](https://d15jtxgb40qetw.cloudfront.net/gigapipe.svg)

Access your Gigapipe account and generate a **Scoped Token** for Odigos

![Gigapipe Token](https://i.imgur.com/zzmufhk.png)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **QRYN\_API\_SECRET** `string` : API Secret. The API Secret for your Gigapipe Account
  * This field is required
* **QRYN\_API\_KEY** `string` : API Key. The API Token for your Gigapipe Account
  * This field is required
* **QRYN\_URL** `string` : API Url. The API Endpoint for your Gigapipe Account
  * This field is required
  * Example: `https://your-account.gigapipe.com`
* **QRYN\_RESOURCE\_TO\_TELEMETRY\_CONVERSION** `boolean` : Convert container attributes to labels.
  * This field is optional and defaults to `True`
* **QRYN\_ADD\_EXPORTER\_NAME** `boolean` : Add exporter name to labels.
  * This field is optional and defaults to `True`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Gigapipe` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `qryn.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: qryn-example
  namespace: odigos-system
spec:
  data:
    QRYN_API_KEY: <API Key>
    QRYN_URL: <API Url>
    # Note: The commented fields below are optional.
    # QRYN_RESOURCE_TO_TELEMETRY_CONVERSION: <Convert container attributes to labels (default: True)>
    # QRYN_ADD_EXPORTER_NAME: <Add exporter name to labels (default: True)>
  destinationName: qryn
  secretRef:
    name: qryn-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: qryn

---

apiVersion: v1
data:
  QRYN_API_SECRET: <Base64 API Secret>
kind: Secret
metadata:
  name: qryn-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f qryn.yaml
```


# Google Cloud Monitoring
Source: https://docs.odigos.io/oss/backends/googlecloudmonitoring

Configuring the Google Cloud Monitoring backend (Managed)

### Getting Started

![googlecloud](https://d15jtxgb40qetw.cloudfront.net/gcp.svg)

### Google Cloud Trace API

This destination uses the [Google Cloud Trace API](https://cloud.google.com/trace/docs/reference).
It is different from the [Google CLoud Telemetry (OTLP) API destination](https://docs.odigos.io/backends/googlecloudotlp).

On GCP, Odigos currently supports Standard (non-Autopilot) GKE clusters.

### Authenticating with Google Cloud

When running the Google Cloud Exporter on GKE, by default the OpenTelemetry Collector will attempt to use
[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
to authenticate to the Google Cloud Observability backend. This means that for most use cases running on GCP,
there is not authentication or service account token required as your workloads are automatically authenticated.

However, to authenticate with Cloud Observability when running off-GKE (for example, on a local machine for another
cloud provider) you must provide a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console)
for the OpenTelemetry Collector Exporter. In Odigos, this can be provided in the Application Credentials field of the
Google Cloud destination.

When the Application Credentials field is set in the UI, the Odigos UI will save the contents of that field as a Kubernetes Secret
which is mounted as a volume into the Collector Pod. The location of that volume is then set as the `GOOGLE_APPLICATION_CREDENTIALS`
environment variable, instructing the Google Cloud Exporter where to look for its credentials.

When creating the destination as a YAML manifest off-GKE, the SecretRef must point to a Secret containing the Service Account token.
The token must be set as the `GCP_APPLICATION_CREDENTIALS` key in the referenced Secret.

The `GCP_APPLICATION_CREDENTIALS` field does not need to be set on the YAML manifest destination CRD, only in the referenced Secret.
This field is used internally by Odigos when creating GCP destinations from the UI.

Currently, only one Google Cloud destination can be configured with Application Credentials.

See the [Google Cloud OpenTelemetry Collector Exporter docs](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/tree/1ea43a4534606df1d91ee1754e6c5bc31ee8664d/exporter/collector#prerequisite-authenticating)
for more information on authentication.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **GCP\_TIMEOUT** `string` : Timeout. The request timeout in seconds for the Google Cloud Monitoring API. Defaults to 10 seconds. Increasing this value can help with deadline exceeded errors.
  * This field is optional
  * Example: `10s`
* **GCP\_PROJECT\_ID** `string` : Project ID. The project ID for the Google Cloud Monitoring API to send data to. Required if running Odigos outside of Google Cloud. Defaults to the detected project ID when running on GCP.
  * This field is optional
  * Example: `project-id`
* **GCP\_APPLICATION\_CREDENTIALS** `string` : Application Credentials. The JSON GCP Application Credentials file for the GCP project. Either a Workload Identity Federation config file or Service Account Key. Required if running Odigos outside of Google Cloud.
  * This field is optional
  * Example: `Credentials config... (optional)`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Google Cloud Monitoring` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `googlecloud.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: googlecloud-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: googlecloud
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: googlecloud-secret
  signals:
  - TRACES
  - LOGS
  type: googlecloud

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   GCP_APPLICATION_CREDENTIALS: <Base64 Application Credentials>
# kind: Secret
# metadata:
#   name: googlecloud-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f googlecloud.yaml
```


# Google Cloud (OTLP)
Source: https://docs.odigos.io/oss/backends/googlecloudotlp

Configuring the Google Cloud (OTLP) backend (Managed)

### Getting Started

![googlecloudotlp](https://d15jtxgb40qetw.cloudfront.net/gcp.svg)

### Google Cloud Telemetry API

This destination uses the [Google Cloud Telemetry (OTLP) API](https://cloud.google.com/stackdriver/docs/reference/telemetry/overview)
to send data as OTLP to GCP Observability using an OTLP/HTTP exporter. It is different from the
[Google Cloud Monitoring destination](https://docs.odigos.io/backends/googlecloudmonitoring)
which uses the [googlecloud exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/d9769f7c1a0f3bea82c711af551d9b5b44d69e6f/exporter/googlecloudexporter).

Ong GCP, Odigos currently supports Standard (non-Autopilot) GKE clusters.

### Authenticating with Google Cloud

When running the Google Cloud Exporter on GKE, by default the OpenTelemetry Collector will attempt to use
[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials)
to authenticate to the Google Cloud Observability backend. This means that for most use cases running on GCP,
there is not authentication or service account token required as your workloads are automatically authenticated.

However, to authenticate with Cloud Observability when running off-GKE (for example, on a local machine for another
cloud provider) you must provide a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console)
for the OpenTelemetry Collector Exporter. In Odigos, this can be provided in the Application Credentials field of the
Google Cloud destination.

When the Application Credentials field is set in the UI, the Odigos UI will save the contents of that field as a Kubernetes Secret
which is mounted as a volume into the Collector Pod. The location of that volume is then set as the `GOOGLE_APPLICATION_CREDENTIALS`
environment variable, instructing the Google Cloud Exporter where to look for its credentials.

When creating the destination as a YAML manifest off-GKE, the SecretRef must point to a Secret containing the Service Account token.
The token must be set as the `GCP_APPLICATION_CREDENTIALS` key in the referenced Secret.

The `GCP_APPLICATION_CREDENTIALS` field does not need to be set on the YAML manifest destination CRD, only in the referenced Secret.
This field is used internally by Odigos when creating GCP destinations from the UI.

Currently, only one Google Cloud destination can be configured with Application Credentials.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **GCP\_PROJECT\_ID** `string` : Project ID. The project ID for the Google Cloud Monitoring API to send data to. This defaults to the Application Credentials project ID (which is automatically detected when running on GCP). Setting this can override the Application Credentials project ID. Required if running Odigos outside of Google Cloud.
  * This field is optional
  * Example: `project-id`
* **GCP\_APPLICATION\_CREDENTIALS** `string` : Application Credentials. The JSON GCP Application Credentials file for the GCP project. Either a Workload Identity Federation config file or Service Account Key. Required if running Odigos outside of Google Cloud.
  * This field is optional
  * Example: `Credentials config... (optional)`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Google Cloud (OTLP)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `googlecloudotlp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: googlecloudotlp-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: googlecloudotlp
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: googlecloudotlp-secret
  signals:
  - TRACES
  type: googlecloudotlp

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   GCP_APPLICATION_CREDENTIALS: <Base64 Application Credentials>
# kind: Secret
# metadata:
#   name: googlecloudotlp-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f googlecloudotlp.yaml
```


# Google Cloud Storage
Source: https://docs.odigos.io/oss/backends/googlecloudstorage

Configuring Google Cloud Storage

Odigos exports data to Google Cloud Storage in [OTLP format](https://opentelemetry.io/docs/specs/otlp/).
Data can be exported either in JSON format or in binary format (protobuf).

Authentication to Google Cloud Storage is done using [Google Application Default Credentials](https://cloud.google.com/docs/authentication/production).

## Configuring Google Cloud Storage Backend

The only required field is the **bucket name**. The bucket must exists and the odigos gateway pod must have write access to it.


# Grafana Cloud Loki
Source: https://docs.odigos.io/oss/backends/grafanacloudloki

Configuring the Grafana Cloud Loki backend (Managed)

### Getting Started

![grafanacloudloki](https://d15jtxgb40qetw.cloudfront.net/grafana.svg)

> [!NOTE]
>
> These instructions are for the Grafana Cloud **managed** Loki service. If you run a self hosted Loki instance, please follow the instructions on the [Loki](../backends/loki) page.

You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Loki as a **logs** backend.

All the connection details can be found at [grafana.com](https://grafana.com) under **My Account**

![Grafana labs website](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1e83df72b7b02521187f810d913bbd5f)

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Loki" card and click "Send Logs"

![Loki managed service configuration](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1c379a219e7e4e9ad698f8fb2f70d815)

**Loki Labels**

Read about setting grafana loki labels in the [Loki](../backends/loki#loki-labels) page.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **GRAFANA\_CLOUD\_LOKI\_ENDPOINT** `string` : Endpoint. The endpoint of your Grafana Cloud Loki instance
  * This field is required
  * Example: `https://logs-prod-{REGION/SHARD}.grafana.net/otlp`
* **GRAFANA\_CLOUD\_LOKI\_USERNAME** `string` : Username. You can find the loki username in the "Grafana Data Source settings" section as "User" value. The username is a number.
  * This field is required
  * Example: `1234567`
* **GRAFANA\_CLOUD\_LOKI\_PASSWORD** `string` : Password (Api Token). This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `logs:write` scope
  * This field is required
  * Example: `glc_eyJvIj...`
* **GRAFANA\_CLOUD\_LOKI\_LABELS** `string[]` : Labels. Use these OpenTelemetry resource attributes as Loki labels for each log record
  * This field is optional and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]`

> [!NOTE]
>
> The supported Loki version is V3. Please make sure you are using the correct version, and that you have enabled native OTLP support. As part of this version, the path `/loki/api/v1/push` has been deprecated and replaced with `/otlp`.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Grafana Cloud Loki` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `grafanacloudloki.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: grafanacloudloki-example
  namespace: odigos-system
spec:
  data:
    GRAFANA_CLOUD_LOKI_ENDPOINT: <Endpoint>
    GRAFANA_CLOUD_LOKI_USERNAME: <Username>
    # Note: The commented fields below are optional.
    # GRAFANA_CLOUD_LOKI_LABELS: <Labels (default: ["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"])>
  destinationName: grafanacloudloki
  secretRef:
    name: grafanacloudloki-secret
  signals:
  - LOGS
  type: grafanacloudloki

---

apiVersion: v1
data:
  GRAFANA_CLOUD_LOKI_PASSWORD: <Base64 Password (Api Token)>
kind: Secret
metadata:
  name: grafanacloudloki-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f grafanacloudloki.yaml
```


# Grafana Cloud Prometheus
Source: https://docs.odigos.io/oss/backends/grafanacloudprometheus

Configuring the Grafana Cloud Prometheus backend (Managed)

### Getting Started

![grafanacloudprometheus](https://d15jtxgb40qetw.cloudfront.net/grafana.svg)

> [!NOTE]
>
> These instructions are for the Grafana Cloud **managed** Prometheus service. If you run a self hosted Prometheus instance, please follow the instructions on the [Prometheus](../backends/prometheus) page.

You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Prometheus as a **metrics** backend.

All the connection details can be found at [grafana.com](https://grafana.com) under **My Account**

![Grafana labs website](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1e83df72b7b02521187f810d913bbd5f)

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Prometheus" card and click "Send Metrics"

![Prometheus managed service configuration](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud3.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=74e9827c06a656b0895412c1b4d05e00)

**Resource Attributes as Metrics Labels**

To use [resource attributes as labels in your data points](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter#setting-resource-attributes-as-metric-labels), you can set the resource attribute names in the `Resource Attributes as Labels` multi-input config option in the destination page in the Odigos UI.

**Cluster Static Labels**

You can set static labels for each metric data pointsent exported to Grafana Cloud Prometheus from the cluster by setting the `External Labels` configuration with lable keys and values.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **GRAFANA\_CLOUD\_PROMETHEUS\_RW\_ENDPOINT** `string` : Prometheus Remote Write Endpoint. This is the URL of the Prometheus service. From the grafana cloud UI, Prometheus page, make sure you copy the URL from “Remote Write Endpoint” section, and not the first Endpoint which is used for queries
  * This field is required
  * Example: `https://{hostname}.grafana.net/api/prom/push`
* **GRAFANA\_CLOUD\_PROMETHEUS\_USERNAME** `string` : Username / Instance ID. You can find the Username / Instance ID on the Prometheus page. Value is a number.
  * This field is required
  * Example: `12345678`
* **GRAFANA\_CLOUD\_PROMETHEUS\_PASSWORD** `string` : Password / Api Token. This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `metrics:write` scope.
  * This field is required
  * Example: `glc_eyJvIj...`
* **PROMETHEUS\_RESOURCE\_ATTRIBUTES\_LABELS** `string[]` : Resource Attributes Labels. use these OpenTelemetry resource attributes as prometheus labels for each metric data point
  * This field is optional and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]`
* **PROMETHEUS\_RESOURCE\_EXTERNAL\_LABELS** `{ key: string; value: string; }[]` : External Labels. map of labels names and values to be attached to each metric data point
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Grafana Cloud Prometheus` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `grafanacloudprometheus.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: grafanacloudprometheus-example
  namespace: odigos-system
spec:
  data:
    GRAFANA_CLOUD_PROMETHEUS_RW_ENDPOINT: <Prometheus Remote Write Endpoint>
    GRAFANA_CLOUD_PROMETHEUS_USERNAME: <Username / Instance ID>
    # Note: The commented fields below are optional.
    # PROMETHEUS_RESOURCE_ATTRIBUTES_LABELS: <Resource Attributes Labels (default: ["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"])>
    # PROMETHEUS_RESOURCE_EXTERNAL_LABELS: <External Labels>
  destinationName: grafanacloudprometheus
  secretRef:
    name: grafanacloudprometheus-secret
  signals:
  - METRICS
  type: grafanacloudprometheus

---

apiVersion: v1
data:
  GRAFANA_CLOUD_PROMETHEUS_PASSWORD: <Base64 Password / Api Token>
kind: Secret
metadata:
  name: grafanacloudprometheus-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f grafanacloudprometheus.yaml
```


# Grafana Cloud Tempo
Source: https://docs.odigos.io/oss/backends/grafanacloudtempo

Configuring the Grafana Cloud Tempo backend (Managed)

### Getting Started

![grafanacloudtempo](https://d15jtxgb40qetw.cloudfront.net/grafana.svg)

> [!NOTE]
>
> These instructions are for the Grafana Cloud **managed** Tempo service. If you run a self hosted Tempo instance, please follow the instructions on the [Tempo](../backends/tempo) page.

You can use Grafana Cloud managed service as a backend for Odigos to store your logs, metrics, and traces. This page describes how to configure Odigos to use Grafana Cloud Tempo as a **trace** backend.

All the connection details can be found at [grafana.com](https://grafana.com) under **My Account**

![Grafana labs website](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1e83df72b7b02521187f810d913bbd5f)

Then, under the "GRAFANA CLOUD" title, select your Grafana Stuck, browser to the "Tempo" card and click "Send Traces"

![Tempo managed service configuration](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/grafanacloud4.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=1fd3c17c628b2bead50e17554215305a)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **GRAFANA\_CLOUD\_TEMPO\_ENDPOINT** `string` : Endpoint. This is the URL of the Tempo service, in the form of `host:port`. You can find the endpoint under the "Sending data to Tempo" section.
  * This field is required
  * Example: `tempo-us-central1.grafana.net:443`
* **GRAFANA\_CLOUD\_TEMPO\_USERNAME** `string` : Username. You can find the username in the "Sending data to Tempo" section under the "basic\_auth" yaml field. The username is a number.
  * This field is required
* **GRAFANA\_CLOUD\_TEMPO\_PASSWORD** `string` : Password (Api Token). This field is refered to as "password" or "Grafana.com API Token" in the Grafana Cloud UI. You can manage tokens in your "Account Settings" page under the "SECURITY" section in the "Access Policies" page. Make sure your token scope includes `traces:write` scope.
  * This field is required
  * Example: `glc_eyJvIj...`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Grafana Cloud Tempo` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `grafanacloudtempo.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: grafanacloudtempo-example
  namespace: odigos-system
spec:
  data:
    GRAFANA_CLOUD_TEMPO_ENDPOINT: <Endpoint>
    GRAFANA_CLOUD_TEMPO_USERNAME: <Username>
  destinationName: grafanacloudtempo
  secretRef:
    name: grafanacloudtempo-secret
  signals:
  - TRACES
  type: grafanacloudtempo

---

apiVersion: v1
data:
  GRAFANA_CLOUD_TEMPO_PASSWORD: <Base64 Password (Api Token)>
kind: Secret
metadata:
  name: grafanacloudtempo-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f grafanacloudtempo.yaml
```


# GreptimeDB
Source: https://docs.odigos.io/oss/backends/greptime

Configuring the GreptimeDB backend (Managed)

### Getting Started

![greptime](https://d15jtxgb40qetw.cloudfront.net/greptime.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **GREPTIME\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
  * Example: `https://<unique_id>.<aws_region>.aws.greptime.cloud`
* **GREPTIME\_DB\_NAME** `string` : Database Name.
  * This field is required
* **GREPTIME\_BASIC\_USERNAME** `string` : Basic Auth Username. The left part (from ":") of the Basic Auth Token.
  * This field is required
* **GREPTIME\_BASIC\_PASSWORD** `string` : Basic Auth Password. The right part (from ":") of the Basic Auth Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `GreptimeDB` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `greptime.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: greptime-example
  namespace: odigos-system
spec:
  data:
    GREPTIME_BASIC_USERNAME: <Basic Auth Username>
    GREPTIME_DB_NAME: <Database Name>
    GREPTIME_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: greptime
  secretRef:
    name: greptime-secret
  signals:
  - METRICS
  type: greptime

---

apiVersion: v1
data:
  GREPTIME_BASIC_PASSWORD: <Base64 Basic Auth Password>
kind: Secret
metadata:
  name: greptime-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f greptime.yaml
```


# Groundcover inCloud
Source: https://docs.odigos.io/oss/backends/groundcover

Configuring the Groundcover inCloud backend (Managed)

### Getting Started

![groundcover](https://d15jtxgb40qetw.cloudfront.net/groundcover.svg)

To get an API Key, refer to [these docs](https://docs.groundcover.com/architecture/byoc/ingestion-endpoints#creating-an-ingestion-key).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **GROUNDCOVER\_ENDPOINT** `string` : Groundcover inCloud Site. The format is `host:port`. Host is required, also known as your `inCloud_Site`, it is part of the configuration provided to you by Groundcover when setting up inCloud Managed. Port is optional, and defaults to the default OpenTelemetry gRPC port `4317`.
  * This field is required
* **GROUNDCOVER\_API\_KEY** `string` : Groundcover API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Groundcover inCloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `groundcover.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: groundcover-example
  namespace: odigos-system
spec:
  data:
    GROUNDCOVER_ENDPOINT: <Groundcover inCloud Site>
  destinationName: groundcover
  secretRef:
    name: groundcover-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: groundcover

---

apiVersion: v1
data:
  GROUNDCOVER_API_KEY: <Base64 Groundcover API Key>
kind: Secret
metadata:
  name: groundcover-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f groundcover.yaml
```


# Honeycomb
Source: https://docs.odigos.io/oss/backends/honeycomb

Configuring the Honeycomb backend (Managed)

### Getting Started

![honeycomb](https://d15jtxgb40qetw.cloudfront.net/honeycomb.svg)

**Where to find the data**

* Each Odigos destination send collected telemetry to a **single environment** in Honeycomb.
* The collected telemetry will go into Honeycomb [datasets](https://docs.honeycomb.io/configure/datasets/manage/) based on the `service.name` resource attribute. odigos will set it by default to the k8s workload (deployment) name (or as set in source configuration).

**Login to Honeycomb**\

Go to the **[🔗 website](https://ui.honeycomb.io)** and login/create your honeycomb account.
You can start with the Free plan to test sending traces, logs and metrics without needing to provide a credit card.

**Select Environment**\

Honeycomb uses [environments](https://docs.honeycomb.io/configure/environments/manage/) to partition and organize data.

Common environments includes `staging`, `production`, `development`, `integration`, or you might want to include the cluster/unit name
to group and isolate telemetry that is unrelated.

Each Odigos Honeycomb destination always sends the data to a **single environment**, based on the "API Key" it is configured with.
Before creating your API Key, make sure to select in ui the environment you want your odigos collected telemetry to show up in.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombenv.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=cd8f7b766c1d92d8fb4c18e103f29002)

**Obtaining Honeycomb API Key**

From the top left corner click on the ENVIRONMENT, and click the **Settings** configuration gear icon to go to the settings page.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombenvsettings.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=b072d71a4d9b1202eab8363962b80fb9)

From the Settings page, click on the **API Keys** tab, make sure you are on "Ingest" sub-tab and click on the **Create Ingest API Key** blue button on the right.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombcreateapikey.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=bc15baaf1bada7ba22baf6d8114a9ba6)

Give your API Key a name (include "odigos" in the name to identify it easily later on), make sure the checkbox for "Can create services/datasets" is checked, and click the **Create** button on the modal bottom right.

![API Keys settings page in Honeycomb](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/honeycombcreateapikeymodal.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=2edbf9646142a7956f985b8d367ea03b)

Copy the ingest key to keyboard and paste it in your Honeycomb Odigos destination configuration (ui or kubernetes manifest).

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **HONEYCOMB\_API\_KEY** `string` : API Key.
  * This field is required
* **HONEYCOMB\_ENDPOINT** `string` : Endpoint. Choose one of the endpoints in the dropdown (`api.honeycomb.io` is the US instance, `api.eu1.honeycomb.io` is EU instance)
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Honeycomb` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `honeycomb.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: honeycomb-example
  namespace: odigos-system
spec:
  data:
    HONEYCOMB_ENDPOINT: '<Endpoint (options: [api.honeycomb.io, api.eu1.honeycomb.io])>'
  destinationName: honeycomb
  secretRef:
    name: honeycomb-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: honeycomb

---

apiVersion: v1
data:
  HONEYCOMB_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: honeycomb-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f honeycomb.yaml
```


# HyperDX
Source: https://docs.odigos.io/oss/backends/hyperdx

Configuring the HyperDX backend (Managed)

### Getting Started

![hyperdx](https://d15jtxgb40qetw.cloudfront.net/hyperdx.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **HYPERDX\_API\_KEY** `string` : HyperDX API Key.
  * This field is required
* **HYPERDX\_LOG\_NORMALIZER** `boolean` : Enable Log Normalizer. Enable HyperDX log normalization processor. This parses JSON from log bodies, infers severity levels, and normalizes log attributes for better querying.
  * This field is optional and defaults to `True`

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `in-otel.hyperdx.io:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `HyperDX` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `hyperdx.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: hyperdx-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: hyperdx
  secretRef:
    name: hyperdx-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: hyperdx

---

apiVersion: v1
data:
  HYPERDX_API_KEY: <Base64 HyperDX API Key>
kind: Secret
metadata:
  name: hyperdx-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f hyperdx.yaml
```


# IBM Instana
Source: https://docs.odigos.io/oss/backends/instana

Configuring the IBM Instana backend (Managed)

### Getting Started

![instana](https://d15jtxgb40qetw.cloudfront.net/instana.svg)

**Creating Account**\

Go to the **[🔗 Website](https://www.ibm.com/products/instana?mhsrc=ibmsearch_a\&mhq=instana) > Account** and click **Log In**

**Obtaining Endpoint**\

Go to the **[📖 Docs](https://www.ibm.com/docs/en/instana-observability/current?topic=instana-backend#endpoints-of-the-instana-backend-otlp-acceptor) > Endpoints of the Instana backend otlp-acceptor** and find your **Endpoint**

**Obtaining Agent Key**\

Go to your **📩 Email > Instana Login Details** and copy **Agent-Key**

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **INSTANA\_ENDPOINT** `string` : OTLP gRPC Endpoint. Domain name of the otlp-acceptor component of the Instana backend.
  * This field is required
  * Example: `otlp-{color}-saas.instana.io`
* **INSTANA\_AGENT\_KEY** `string` : Agent Key. The agent key of the Instana for targeting the Instana backend.
  * This field is required

> [!WARNING]
>
> For ingesting logs via OpenTelemetry, an add-on to the Instana license is required. It’s advisable to contact your IBM Sales representative to include this add-on in your account.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `IBM Instana` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `instana.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: instana-example
  namespace: odigos-system
spec:
  data:
    INSTANA_ENDPOINT: <OTLP gRPC Endpoint>
  destinationName: instana
  secretRef:
    name: instana-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: instana

---

apiVersion: v1
data:
  INSTANA_AGENT_KEY: <Base64 Agent Key>
kind: Secret
metadata:
  name: instana-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f instana.yaml
```


# Jaeger
Source: https://docs.odigos.io/oss/backends/jaeger

Configuring the Jaeger backend (Self-Hosted)

### Getting Started

![jaeger](https://d15jtxgb40qetw.cloudfront.net/jaeger.svg)

Version v1.35 of Jaeger introduced the ability to receive OpenTelemetry trace data via the OpenTelemetry Protocol (OTLP). This allows to create a new Jaeger backend by simply specifying the Jaeger OTLP gRPC unencrypted URL.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **JAEGER\_URL** `string` : Jaeger OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`
  * This field is required
* **JAEGER\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **JAEGER\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Jaeger` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `jaeger.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: jaeger-example
  namespace: odigos-system
spec:
  data:
    JAEGER_URL: <Jaeger OTLP gRPC Endpoint>
    # Note: The commented fields below are optional.
    # JAEGER_TLS_ENABLED: <Enable TLS>
    # JAEGER_CA_PEM: <Certificate Authority>
  destinationName: jaeger
  signals:
  - TRACES
  type: jaeger
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f jaeger.yaml
```


# Kafka
Source: https://docs.odigos.io/oss/backends/kafka

Configuring the Kafka backend (Self-Hosted)

### Getting Started

![kafka](https://d15jtxgb40qetw.cloudfront.net/kafka.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **KAFKA\_PROTOCOL\_VERSION** `string` : Protocol Version. Kafka protocol version.
  * This field is required
  * Example: `2.0.0`
* **KAFKA\_BROKERS** `string[]` : Brokers. The list of kafka brokers
  * This field is optional and defaults to `["localhost:9092"]`
* **KAFKA\_RESOLVE\_CANONICAL\_BOOTSTRAP\_SERVERS\_ONLY** `boolean` : Resolve Canonical Bootstrap Servers Only. Whether to resolve then reverse-lookup broker IPs during startup.
  * This field is optional and defaults to `False`
* **KAFKA\_CLIENT\_ID** `string` : Client ID. The client ID to configure the Sarama Kafka client with. The client ID will be used for all produce requests.
  * This field is optional and defaults to `sarama`
* **KAFKA\_TOPIC** `string` : Topic. The name of the default kafka topic to export to (default = otlp\_spans for traces, otlp\_metrics for metrics, otlp\_logs for logs).
  * This field is optional
* **KAFKA\_TOPIC\_FROM\_ATTRIBUTE** `string` : Topic from Attribute. Specify the resource attribute whose value should be used as the message's topic.
  * This field is optional
* **KAFKA\_ENCODING** `string` : Encoding. The encoding of the traces sent to kafka.
  * This field is optional and defaults to `otlp_proto`
* **KAFKA\_PARTITION\_TRACES\_BY\_ID** `boolean` : Partition Traces by ID. Configures the exporter to include the trace ID as the message key in trace messages sent to kafka. Please note: this setting does not have any effect on Jaeger encoding exporters since Jaeger exporters include trace ID as the message key by default.
  * This field is optional and defaults to `False`
* **KAFKA\_PARTITION\_METRICS\_BY\_RESOURCE\_ATTRIBUTES** `boolean` : Partition Metrics by Resource Attributes. Configures the exporter to include the hash of sorted resource attributes as the message partitioning key in metric messages sent to kafka.
  * This field is optional and defaults to `False`
* **KAFKA\_PARTITION\_LOGS\_BY\_RESOURCE\_ATTRIBUTES** `boolean` : Partition Logs by Resource Attributes. Configures the exporter to include the hash of sorted resource attributes as the message partitioning key in log messages sent to kafka.
  * This field is optional and defaults to `False`
* **KAFKA\_AUTH\_METHOD** `string` : Auth Method. The auth method to use.
  * This field is required and defaults to `none`
* **KAFKA\_USERNAME** `string` : Username. The username to use.
  * This field is optional
* **KAFKA\_PASSWORD** `string` : Password. The password to use.
  * This field is optional
* **KAFKA\_METADATA\_FULL** `boolean` : Metadata Full. Whether to maintain a full set of metadata. When disabled, the client does not make the initial request to broker at the startup.
  * This field is optional and defaults to `False`
* **KAFKA\_METADATA\_MAX\_RETRY** `string` : Metadata Max Retry. The number of retries to get metadata.
  * This field is optional and defaults to `3`
* **KAFKA\_METADATA\_BACKOFF\_RETRY** `string` : Metadata Backoff Retry. How long to wait between metadata retries.
  * This field is optional and defaults to `250ms`
* **KAFKA\_TIMEOUT** `string` : Timeout. Is the timeout for every attempt to send data to the backend.
  * This field is optional and defaults to `5s`
* **KAFKA\_RETRY\_ON\_FAILURE\_ENABLED** `boolean` : Enable Retry on Failure.
  * This field is optional and defaults to `True`
* **KAFKA\_RETRY\_ON\_FAILURE\_INITIAL\_INTERVAL** `string` : Initial Interval. Time to wait after the first failure before retrying; ignored if `enabled` is `false`.
  * This field is optional and defaults to `5s`
* **KAFKA\_RETRY\_ON\_FAILURE\_MAX\_INTERVAL** `string` : Max Interval. Is the upper bound on backoff; ignored if `enabled` is `false`.
  * This field is optional and defaults to `30s`
* **KAFKA\_RETRY\_ON\_FAILURE\_MAX\_ELAPSED\_TIME** `string` : Max Elapsed Time. Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false`.
  * This field is optional and defaults to `120s`
* **KAFKA\_PRODUCER\_MAX\_MESSAGE\_BYTES** `string` : Producer Max Message Bytes. The maximum permitted size of a message in bytes.
  * This field is optional and defaults to `1000000`
* **KAFKA\_PRODUCER\_REQUIRED\_ACKS** `string` : Producer Required Acks. Controls when a message is regarded as transmitted.
  * This field is optional and defaults to `1`
* **KAFKA\_PRODUCER\_COMPRESSION** `string` : Producer Compression. The compression used when producing messages to kafka.
  * This field is optional and defaults to `none`
* **KAFKA\_PRODUCER\_FLUSH\_MAX\_MESSAGES** `string` : Producer Flush Max Messages. The maximum number of messages the producer will send in a single broker request.
  * This field is optional and defaults to `0`

> [!NOTE]
>
> The destination topic can be defined in a few different ways and takes priority in the following order:
>
> 1. When `topic_from_attribute` is configured, and the corresponding attribute is found on the ingested data, the value of this attribute is used.
> 2. If a prior component in the collector pipeline sets the topic on the context via the `topic.WithTopic` function (from the [github.com/open-telemetry/opentelemetry-collector-contrib/pkg/kafka/topic](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/kafka/topic) package), the value set in the context is used.
> 3. Finally, the topic configuration is used as a default/fallback destination.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Kafka` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `kafka.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: kafka-example
  namespace: odigos-system
spec:
  data:
    KAFKA_AUTH_METHOD: '<Auth Method (default: none) (options: [none, plain_text])>'
    KAFKA_PROTOCOL_VERSION: <Protocol Version>
    # Note: The commented fields below are optional.
    # KAFKA_BROKERS: <Brokers (default: ["localhost:9092"])>
    # KAFKA_RESOLVE_CANONICAL_BOOTSTRAP_SERVERS_ONLY: <Resolve Canonical Bootstrap Servers Only>
    # KAFKA_CLIENT_ID: <Client ID (default: sarama)>
    # KAFKA_TOPIC: <Topic>
    # KAFKA_TOPIC_FROM_ATTRIBUTE: <Topic from Attribute>
    # KAFKA_ENCODING: <Encoding (default: otlp_proto) (options: [otlp_proto, otlp_json])>
    # KAFKA_PARTITION_TRACES_BY_ID: <Partition Traces by ID>
    # KAFKA_PARTITION_METRICS_BY_RESOURCE_ATTRIBUTES: <Partition Metrics by Resource Attributes>
    # KAFKA_PARTITION_LOGS_BY_RESOURCE_ATTRIBUTES: <Partition Logs by Resource Attributes>
    # KAFKA_USERNAME: <Username>
    # KAFKA_METADATA_FULL: <Metadata Full>
    # KAFKA_METADATA_MAX_RETRY: <Metadata Max Retry (default: 3)>
    # KAFKA_METADATA_BACKOFF_RETRY: <Metadata Backoff Retry (default: 250ms)>
    # KAFKA_TIMEOUT: <Timeout (default: 5s)>
    # KAFKA_RETRY_ON_FAILURE_ENABLED: <Enable Retry on Failure (default: True)>
    # KAFKA_RETRY_ON_FAILURE_INITIAL_INTERVAL: <Initial Interval (default: 5s)>
    # KAFKA_RETRY_ON_FAILURE_MAX_INTERVAL: <Max Interval (default: 30s)>
    # KAFKA_RETRY_ON_FAILURE_MAX_ELAPSED_TIME: <Max Elapsed Time (default: 120s)>
    # KAFKA_PRODUCER_MAX_MESSAGE_BYTES: <Producer Max Message Bytes (default: 1000000)>
    # KAFKA_PRODUCER_REQUIRED_ACKS: <Producer Required Acks (default: 1)>
    # KAFKA_PRODUCER_COMPRESSION: <Producer Compression (default: none) (options: [none, gzip, snappy, lz4, zstd])>
    # KAFKA_PRODUCER_FLUSH_MAX_MESSAGES: <Producer Flush Max Messages (default: 0)>
  destinationName: kafka
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: kafka-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: kafka

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   KAFKA_PASSWORD: <Base64 Password>
# kind: Secret
# metadata:
#   name: kafka-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f kafka.yaml
```


# KloudMate
Source: https://docs.odigos.io/oss/backends/kloudmate

Configuring the KloudMate backend (Managed)

### Getting Started

![kloudmate](https://d15jtxgb40qetw.cloudfront.net/kloudmate.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **KLOUDMATE\_API\_KEY** `string` : KloudMate API Key.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `https://otel.kloudmate.com:4318`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `KloudMate` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `kloudmate.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: kloudmate-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: kloudmate
  secretRef:
    name: kloudmate-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: kloudmate

---

apiVersion: v1
data:
  KLOUDMATE_API_KEY: <Base64 KloudMate API Key>
kind: Secret
metadata:
  name: kloudmate-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f kloudmate.yaml
```


# Last9
Source: https://docs.odigos.io/oss/backends/last9

Configuring the Last9 backend (Managed)

### Getting Started

![last9](https://d15jtxgb40qetw.cloudfront.net/last9.svg)

[Click here](https://app.last9.io/integrations?category=all\&integration=OpenTelemetry) to visit the Last9 OpenTelemetry integration page.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **LAST9\_OTLP\_ENDPOINT** `string` : Last9 OpenTelemetry Endpoint.
  * This field is required
* **LAST9\_OTLP\_BASIC\_AUTH\_HEADER** `string` : Basic Auth Header.
  * This field is required
  * Example: `Basic ...`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Last9` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `last9.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: last9-example
  namespace: odigos-system
spec:
  data:
    LAST9_OTLP_ENDPOINT: <Last9 OpenTelemetry Endpoint>
  destinationName: last9
  secretRef:
    name: last9-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: last9

---

apiVersion: v1
data:
  LAST9_OTLP_BASIC_AUTH_HEADER: <Base64 Basic Auth Header>
kind: Secret
metadata:
  name: last9-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f last9.yaml
```


# Lightstep
Source: https://docs.odigos.io/oss/backends/lightstep

Configuring the Lightstep backend (Managed)

### Getting Started

![lightstep](https://d15jtxgb40qetw.cloudfront.net/lightstep.svg)

**Obtaining Lightstep Access Token**\

Go to **⚙️ > Access Tokens** and click **Create New**

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **LIGHTSTEP\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Lightstep` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `lightstep.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: lightstep-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: lightstep
  secretRef:
    name: lightstep-secret
  signals:
  - TRACES
  type: lightstep

---

apiVersion: v1
data:
  LIGHTSTEP_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: lightstep-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f lightstep.yaml
```


# Logz.io
Source: https://docs.odigos.io/oss/backends/logzio

Configuring the Logz.io backend (Managed)

### Getting Started

![logzio](https://d15jtxgb40qetw.cloudfront.net/logzio.svg)

**Finding Logz.io Region**

You can find your account’s region by selecting **⚙️ > Settings > General** from the top menu. Your two-letter region code is at the start of the region designation.
Another option is to sign in to your Logz.io account and look at the URL in the address bar.

If you see `app.logz.io`, your account is hosted in the US East (Northern Virginia). All other regions have a two-letter region code. For example, if you see `app-eu.logz.io`, then your account is in the Europe (Frankfurt) region.

**Obtaining Tokens**

From your account, go to the **Manage Tokens > Data shipping** tokens tab of your Operations workspace.
It can be reached by selecting **⚙️ > Settings > Manage tokens** in the ADMIN ZONE section of the side navigation menu.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **LOGZIO\_REGION** `string` : Region. The region of your Logz.io account
  * This field is required
* **LOGZIO\_TRACING\_TOKEN** `string` : Tracing token. The token you obtained from Logz.io
  * This field is optional
* **LOGZIO\_METRICS\_TOKEN** `string` : Metrics token. The token you obtained from Logz.io
  * This field is optional
* **LOGZIO\_LOGS\_TOKEN** `string` : Logs token. The token you obtained from Logz.io
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Logz.io` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `logzio.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: logzio-example
  namespace: odigos-system
spec:
  data:
    LOGZIO_REGION: '<Region (options: [listener.logz.io, listener-au.logz.io, listener-ca.logz.io,
      listener-eu.logz.io, listener-nl.logz.io, listener-uk.logz.io, listener-wa.logz.io])>'
  destinationName: logzio
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: logzio-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: logzio

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   LOGZIO_LOGS_TOKEN: <Base64 Logs token>
#   LOGZIO_METRICS_TOKEN: <Base64 Metrics token>
#   LOGZIO_TRACING_TOKEN: <Base64 Tracing token>
# kind: Secret
# metadata:
#   name: logzio-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f logzio.yaml
```


# Loki
Source: https://docs.odigos.io/oss/backends/loki

Configuring the Loki backend (Self-Hosted)

### Getting Started

![loki](https://d15jtxgb40qetw.cloudfront.net/loki.svg)

> [!NOTE]
>
> These instructions are for the **self hosted** Loki instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Loki](../backends/grafanacloudloki) page.

This destination is for self hosted Loki instances.

**Loki Labels**

Loki labels are used to "index" logs and allow efficient filtering in the Grafana UI. You can read more about them [here](https://grafana.com/docs/loki/latest/get-started/labels/).

You can configure your grafana cloud loki labels in the `Labels` multi-input config option in the destination page in the Odigos UI.

* Labels should be names of OpenTelemetry resource attribute or log record attribute keys that you want to index, for example: `"k8s.namespace.name"`.
* To avoid performance issues, it is recommended to limit the number of labels to a reasonable amount.
* [Use only low cardinality values](https://grafana.com/docs/loki/latest/get-started/labels/#cardinality). e.g. use only OpenTelemetry attributes for which you expect a small number of unique values like `http.response.status_code` but **not** `network.peer.address`.
* If the label is not present in a log record, it will be ignored.
* The default labels if not set are: `k8s.container.name, k8s.pod.name, k8s.namespace.name`

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **LOKI\_URL** `string` : Endpoint. If scheme is not provided, it defaults to `http`. If port is not provided, it defaults to `3100`. If path is not provided, it defaults to `/otlp`.
  * This field is required
  * Example: `http://<loki-host>:<loki-port>/otlp`
* **LOKI\_SCOPE\_ORG\_ID** `string` : X-Scope-OrgID. use this X-Scope-OrgID if you need to scope the data to a specific org. Multiple orgs can be chained using `|`
  * This field is optional
  * Example: `org1|org2|org3`
* **LOKI\_USERNAME** `string` : Basic Auth Username. username for basic authentication
  * This field is optional
  * Example: `<loki-username>`
* **LOKI\_PASSWORD** `string` : Basic Auth Password. password for basic authentication
  * This field is optional
  * Example: `<loki-password>`
* **LOKI\_LABELS** `string[]` : Labels. use these OpenTelemetry resource attributes as loki labels for each log record
  * This field is required and defaults to `["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"]`

> [!NOTE]
>
> The supported Loki version is V3. Please make sure you are using the correct version, and that you have enabled native OTLP support. As part of this version, the path `/loki/api/v1/push` has been deprecated and replaced with `/otlp`.

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Loki` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `loki.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: loki-example
  namespace: odigos-system
spec:
  data:
    LOKI_LABELS: '<Labels (default: ["k8s.container.name", "k8s.pod.name", "k8s.namespace.name"])>'
    LOKI_URL: <Endpoint>
    # Note: The commented fields below are optional.
    # LOKI_SCOPE_ORG_ID: <X-Scope-OrgID>
    # LOKI_USERNAME: <Basic Auth Username>
  destinationName: loki
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: loki-secret
  signals:
  - LOGS
  type: loki

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   LOKI_PASSWORD: <Base64 Basic Auth Password>
# kind: Secret
# metadata:
#   name: loki-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f loki.yaml
```


# Lumigo
Source: https://docs.odigos.io/oss/backends/lumigo

Configuring the Lumigo backend (Managed)

### Getting Started

![lumigo](https://d15jtxgb40qetw.cloudfront.net/lumigo.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **LUMIGO\_ENDPOINT** `string` : Lumigo OTLP HTTP Endpoint.
  * This field is required and defaults to `https://ga-otlp.lumigo-tracer-edge.golumigo.com`
* **LUMIGO\_TOKEN** `string` : Lumigo Authorization Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Lumigo` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `lumigo.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: lumigo-example
  namespace: odigos-system
spec:
  data:
    LUMIGO_ENDPOINT: '<Lumigo OTLP HTTP Endpoint (default: https://ga-otlp.lumigo-tracer-edge.golumigo.com)>'
  destinationName: lumigo
  secretRef:
    name: lumigo-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: lumigo

---

apiVersion: v1
data:
  LUMIGO_TOKEN: <Base64 Lumigo Authorization Token>
kind: Secret
metadata:
  name: lumigo-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f lumigo.yaml
```


# Middleware
Source: https://docs.odigos.io/oss/backends/middleware

Configuring the Middleware backend (Managed)

### Getting Started

![middleware](https://d15jtxgb40qetw.cloudfront.net/middleware.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **MW\_TARGET** `string` : Endpoint.
  * This field is required
* **MW\_API\_KEY** `string` : API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Middleware` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `middleware.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: middleware-example
  namespace: odigos-system
spec:
  data:
    MW_TARGET: <Endpoint>
  destinationName: middleware
  secretRef:
    name: middleware-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: middleware

---

apiVersion: v1
data:
  MW_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: middleware-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f middleware.yaml
```


# New Relic
Source: https://docs.odigos.io/oss/backends/newrelic

Configuring the New Relic backend (Managed)

### Getting Started

![newrelic](https://d15jtxgb40qetw.cloudfront.net/newrelic.svg)

**Obtaining a License Key**\

On the left sidebar, click on your account name and select **API Keys** and then click on **Create a key**.

![New Relic API keys](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/newrelic1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=11888f0e65ddb12403e887acd2b8f763)

For key type select **Ingest - License**, give a name to your key and press **Create a key**.

![Create an API key screen in New Relic](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/newrelic2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=4e8c022b81cd8287902d373ac2f2e1b4)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **NEWRELIC\_API\_KEY** `string` : License Key.
  * This field is required
* **NEWRELIC\_ENDPOINT** `string` : Endpoint.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `New Relic` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `newrelic.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: newrelic-example
  namespace: odigos-system
spec:
  data:
    NEWRELIC_ENDPOINT: '<Endpoint (options: [https://otlp.nr-data.net, https://otlp.eu01.nr-data.net])>'
  destinationName: newrelic
  secretRef:
    name: newrelic-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: newrelic

---

apiVersion: v1
data:
  NEWRELIC_API_KEY: <Base64 License Key>
kind: Secret
metadata:
  name: newrelic-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f newrelic.yaml
```


# Observe
Source: https://docs.odigos.io/oss/backends/observe

Configuring the Observe backend (Managed)

### Getting Started

![observe](https://d15jtxgb40qetw.cloudfront.net/observe.svg)

**Creating an ingest token**\

Data is sent to Observe's OTLP collection endpoint, which requires a **Datastream ingest token** (a user/API token will be rejected with `401`). Create one in the Observe UI under **Datastreams**, following [Configure your own OTel collector without Kubernetes](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment).

**Collection endpoint**\

By default the endpoint is `https://<customer_id>.collect.observeinc.com/v2/otel`. For regional tenants or other overrides, set the optional `OBSERVE_ENDPOINT` field. See [Configure your own OTel collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) for the endpoint format.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **OBSERVE\_CUSTOMER\_ID** `string` : Customer ID. The Customer ID is the first part of your Observe URL. For example, if your Observe URL is `https://<customer_id>.observe.com`, then the Customer ID is `<customer_id>`.
  * This field is required
* **OBSERVE\_TOKEN** `string` : Token. A Datastream ingest token (not a user/API token). Create one in the Observe UI under "Datastreams". See [https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment) for how to create an ingest token.
  * This field is required
* **OBSERVE\_ENDPOINT** `string` : Collection Endpoint. Optional. The full OTLP/HTTP collection endpoint. Set this to override the default, for example for a regional tenant such as `https://<customer_id>.collect.eu1.observeinc.com/v2/otel`. When empty, it is derived from the Customer ID. See [https://docs.observeinc.com/docs/configure-your-own-otel-collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) for the endpoint format.
  * This field is optional
  * Example: `https://<customer_id>.collect.observeinc.com/v2/otel`

> [!TIP]
>
> By default the endpoint is derived from your Customer ID as `https://${OBSERVE_CUSTOMER_ID}.collect.observeinc.com/v2/otel`.
> To send to a different endpoint (for example a regional tenant), set the optional `OBSERVE_ENDPOINT` field.
> For details on the collection endpoint and creating an ingest token, see Observe's guides for [configuring your own OTel collector](https://docs.observeinc.com/docs/configure-your-own-otel-collector) and [configuring it without Kubernetes](https://docs.observeinc.com/docs/configure-your-own-otel-collector-in-a-non-kubernetes-environment).

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Observe` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `observe.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: observe-example
  namespace: odigos-system
spec:
  data:
    OBSERVE_CUSTOMER_ID: <Customer ID>
    # Note: The commented fields below are optional.
    # OBSERVE_ENDPOINT: <Collection Endpoint>
  destinationName: observe
  secretRef:
    name: observe-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: observe

---

apiVersion: v1
data:
  OBSERVE_TOKEN: <Base64 Token>
kind: Secret
metadata:
  name: observe-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f observe.yaml
```


# OneUptime
Source: https://docs.odigos.io/oss/backends/oneuptime

Configuring the OneUptime backend (Managed)

### Getting Started

![oneuptime](https://d15jtxgb40qetw.cloudfront.net/oneuptime.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **ONEUPTIME\_INGESTION\_KEY** `string` : Telemetry Ingestion Key. The Telemetry Ingestion Key is used to authenticate your data. You can find it in the OneUptime dashboard under Project Settings > Ingestion Keys.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `https://oneuptime.com/otlp`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OneUptime` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `oneuptime.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: oneuptime-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: oneuptime
  secretRef:
    name: oneuptime-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: oneuptime

---

apiVersion: v1
data:
  ONEUPTIME_INGESTION_KEY: <Base64 Telemetry Ingestion Key>
kind: Secret
metadata:
  name: oneuptime-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f oneuptime.yaml
```


# OpenObserve
Source: https://docs.odigos.io/oss/backends/openobserve

Configuring the OpenObserve backend (Managed)

### Getting Started

![openobserve](https://d15jtxgb40qetw.cloudfront.net/openobserve.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **OPEN\_OBSERVE\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
  * Example: `https://api.openobserve.ai/api/<organization_id>`
* **OPEN\_OBSERVE\_API\_KEY** `string` : API Key. Do not include the `Basic` prefix.
  * This field is required
* **OPEN\_OBSERVE\_STREAM\_NAME** `string` : Stream Name.
  * This field is required and defaults to `default`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OpenObserve` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `openobserve.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: openobserve-example
  namespace: odigos-system
spec:
  data:
    OPEN_OBSERVE_ENDPOINT: <OTLP HTTP Endpoint>
    OPEN_OBSERVE_STREAM_NAME: '<Stream Name (default: default)>'
  destinationName: openobserve
  secretRef:
    name: openobserve-secret
  signals:
  - TRACES
  - LOGS
  type: openobserve

---

apiVersion: v1
data:
  OPEN_OBSERVE_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: openobserve-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f openobserve.yaml
```


# Oracle Cloud
Source: https://docs.odigos.io/oss/backends/oracle

Configuring the Oracle Cloud backend (Managed)

### Getting Started

![oracle](https://d15jtxgb40qetw.cloudfront.net/oracle.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **ORACLE\_ENDPOINT** `string` : OTLP HTTP Endpoint. The data upload endpoint of the Oracle APM domain.
  * This field is required
* **ORACLE\_DATA\_KEY** `string` : Data Key.
  * This field is required
* **ORACLE\_DATA\_KEY\_TYPE** `string` : Data Key Type. The type of the data key, relevant for Traces. Metrics are always authenticated with a private data key.
  * This field is required and defaults to `private`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Oracle Cloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `oracle.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: oracle-example
  namespace: odigos-system
spec:
  data:
    ORACLE_DATA_KEY_TYPE: '<Data Key Type (default: private) (options: [private, public])>'
    ORACLE_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: oracle
  secretRef:
    name: oracle-secret
  signals:
  - TRACES
  - METRICS
  type: oracle

---

apiVersion: v1
data:
  ORACLE_DATA_KEY: <Base64 Data Key>
kind: Secret
metadata:
  name: oracle-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f oracle.yaml
```


# OTLP gRPC
Source: https://docs.odigos.io/oss/backends/otlp

Configuring the OTLP gRPC backend (Self-Hosted)

### Getting Started

![otlp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg)

For advanced users trying to implement complex observability pipelines, Odigos support sending data to any OTLP gRPC unencrypted endpoint.

> [!NOTE]
>
> Note that if your backend expects [OTLP over http](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) you should use the [OTLP http](../backends/otlphttp) destination instead.

> [!TIP]
>
> If your backend is supported natively in Odigos, it's recommended to use the native integration.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ✅ Profiles

* **OTLP\_GRPC\_ENDPOINT** `string` : OTLP gRPC Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP gRPC port `4317`.
  * This field is required
  * Example: `host:port`
* **OTLP\_GRPC\_COMPRESSION** `string` : Destination Compression Type. Compression type to use for the destination. The default is `none`. The compression type must be supported by the destination.
  * This field is optional and defaults to `none`
* **OTLP\_GRPC\_HEADERS** `{ key: string; value: string; }[]` : Headers. Headers is the option to set custom GRPC headers for OTLP GRPC destination. If specified, please provide each header in the format: key:value. Multiple headers can be added. Keys must be non-empty strings and follow standard HTTP header conventions. Values must be non-empty strings and may include alphanumerics, whitespace, and standard punctuation.
  * This field is optional
* **OTLP\_GRPC\_OAUTH2\_ENABLED** `boolean` : Enable OAuth2. Enable OAuth2 client credentials authentication
  * This field is optional and defaults to `False`
* **OTLP\_GRPC\_OAUTH2\_CLIENT\_ID** `string` : OAuth2 Client ID. OAuth2 client identifier for client credentials flow
  * This field is optional
  * Example: `your-client-id`
* **OTLP\_GRPC\_OAUTH2\_CLIENT\_SECRET** `string` : OAuth2 Client Secret. OAuth2 client secret for client credentials flow
  * This field is optional
  * Example: `your-client-secret`
* **OTLP\_GRPC\_OAUTH2\_TOKEN\_URL** `string` : OAuth2 Token URL. OAuth2 token endpoint URL for obtaining access tokens
  * This field is optional
  * Example: `https://example.com/oauth2/token`
* **OTLP\_GRPC\_OAUTH2\_SCOPES** `string` : OAuth2 Scopes. Comma-separated list of OAuth2 scopes to request (e.g., "api.metrics,api.traces")
  * This field is optional
  * Example: `api.metrics,api.traces`
* **OTLP\_GRPC\_OAUTH2\_AUDIENCE** `string` : OAuth2 Audience. OAuth2 audience parameter for token requests
  * This field is optional
  * Example: `api.example.com`
* **OTLP\_GRPC\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **OTLP\_GRPC\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_GRPC\_MTLS\_ENABLED** `boolean` : Enable mTLS. Use a client certificate and key for mutual TLS authentication with the destination
  * This field is optional and defaults to `False`
* **OTLP\_GRPC\_CLIENT\_CERT\_PEM** `string` : Client Certificate. Client certificate in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_GRPC\_CLIENT\_KEY\_PEM** `string` : Client Key. Client private key in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN PRIVATE KEY-----`
* **OTLP\_GRPC\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OTLP gRPC` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `otlp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: otlp-example
  namespace: odigos-system
spec:
  data:
    OTLP_GRPC_ENDPOINT: <OTLP gRPC Endpoint>
    # Note: The commented fields below are optional.
    # OTLP_GRPC_COMPRESSION: <Destination Compression Type (default: none) (options: [none, gzip, snappy, lz4, zlib, deflate, zstd])>
    # OTLP_GRPC_HEADERS: <Headers>
    # OTLP_GRPC_OAUTH2_ENABLED: <Enable OAuth2>
    # OTLP_GRPC_OAUTH2_CLIENT_ID: <OAuth2 Client ID>
    # OTLP_GRPC_OAUTH2_TOKEN_URL: <OAuth2 Token URL>
    # OTLP_GRPC_OAUTH2_SCOPES: <OAuth2 Scopes>
    # OTLP_GRPC_OAUTH2_AUDIENCE: <OAuth2 Audience>
    # OTLP_GRPC_TLS_ENABLED: <Enable TLS>
    # OTLP_GRPC_CA_PEM: <Certificate Authority>
    # OTLP_GRPC_MTLS_ENABLED: <Enable mTLS>
    # OTLP_GRPC_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
  destinationName: otlp
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: otlp-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  - PROFILES
  type: otlp

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   OTLP_GRPC_CLIENT_CERT_PEM: <Base64 Client Certificate>
#   OTLP_GRPC_CLIENT_KEY_PEM: <Base64 Client Key>
#   OTLP_GRPC_OAUTH2_CLIENT_SECRET: <Base64 OAuth2 Client Secret>
# kind: Secret
# metadata:
#   name: otlp-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f otlp.yaml
```


# OTLP http
Source: https://docs.odigos.io/oss/backends/otlphttp

Configuring the OTLP http backend (Self-Hosted)

### Getting Started

![otlphttp](https://d15jtxgb40qetw.cloudfront.net/opentelemetry.svg)

For advanced users trying to implement complex observability pipelines, Odigos support sending data to any OTLP http endpoint.

> [!NOTE]
>
> Note that if your backend expects [OTLP over gRPC](https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options) you should use the [OTLP gRPC](../backends/otlp) destination instead.

> [!TIP]
>
> If your backend is supported natively in Odigos, it's recommended to use the native integration.

#### Using Basic Authentication

This section is relevant if your OTLP http endpoint requires [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) (username and password).

To configure basic authentication, use the optional config options `Basic Auth Username` and `Basic Auth Password`.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ✅ Profiles

* **OTLP\_HTTP\_ENDPOINT** `string` : OTLP http Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`.
  * This field is required
  * Example: `http://host:port`
* **OTLP\_HTTP\_BASIC\_AUTH\_USERNAME** `string` : Basic Auth Username. in case the otlp receiver requires basic auth, this is the username
  * This field is optional
  * Example: `username`
* **OTLP\_HTTP\_BASIC\_AUTH\_PASSWORD** `string` : Basic Auth Password. in case the otlp receiver requires basic auth, this is the password
  * This field is optional
  * Example: `password`
* **OTLP\_HTTP\_OAUTH2\_ENABLED** `boolean` : Enable OAuth2. Enable OAuth2 client credentials authentication
  * This field is optional and defaults to `False`
* **OTLP\_HTTP\_OAUTH2\_CLIENT\_ID** `string` : OAuth2 Client ID. OAuth2 client identifier for client credentials flow
  * This field is optional
  * Example: `your-client-id`
* **OTLP\_HTTP\_OAUTH2\_CLIENT\_SECRET** `string` : OAuth2 Client Secret. OAuth2 client secret for client credentials flow
  * This field is optional
  * Example: `your-client-secret`
* **OTLP\_HTTP\_OAUTH2\_TOKEN\_URL** `string` : OAuth2 Token URL. OAuth2 token endpoint URL for obtaining access tokens
  * This field is optional
  * Example: `https://example.com/oauth2/token`
* **OTLP\_HTTP\_OAUTH2\_SCOPES** `string` : OAuth2 Scopes. Comma-separated list of OAuth2 scopes to request (e.g., "api.metrics,api.traces")
  * This field is optional
  * Example: `api.metrics,api.traces`
* **OTLP\_HTTP\_OAUTH2\_AUDIENCE** `string` : OAuth2 Audience. OAuth2 audience parameter for token requests
  * This field is optional
  * Example: `api.example.com`
* **OTLP\_HTTP\_COMPRESSION** `string` : Destination Compression Type. Compression type to use for the destination. The default is `none`. The compression type must be supported by the destination.
  * This field is optional and defaults to `none`
* **OTLP\_HTTP\_HEADERS** `{ key: string; value: string; }[]` : Headers. Headers is the option to set custom HTTP headers for OTLP HTTP destination. If specified, please provide each header in the format: key:value. Multiple headers can be added. Keys must be non-empty strings and follow standard HTTP header conventions. Values must be non-empty strings and may include alphanumerics, whitespace, and standard punctuation.
  * This field is optional
* **OTLP\_HTTP\_TRACES\_ENDPOINT** `string` : OTLP http traces Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/traces` will be used.
  * This field is optional
  * Example: `http://host:port/v1/traces`
* **OTLP\_HTTP\_METRICS\_ENDPOINT** `string` : OTLP http metrics Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/metrics` will be used.
  * This field is optional
  * Example: `http://host:port/v1/metrics`
* **OTLP\_HTTP\_LOGS\_ENDPOINT** `string` : OTLP http logs Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/logs` will be used.
  * This field is optional
  * Example: `http://host:port/v1/logs`
* **OTLP\_HTTP\_PROFILES\_ENDPOINT** `string` : OTLP http profiles Endpoint. The format is `host:port`, host is required, port is optional and defaults to the default OTLP HTTP port `4318`, if missing `http://host:port/v1/profiles` will be used.
  * This field is optional
  * Example: `http://host:port/v1/profiles`
* **OTLP\_HTTP\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **OTLP\_HTTP\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_HTTP\_MTLS\_ENABLED** `boolean` : Enable mTLS. Use a client certificate and key for mutual TLS authentication with the destination
  * This field is optional and defaults to `False`
* **OTLP\_HTTP\_CLIENT\_CERT\_PEM** `string` : Client Certificate. Client certificate in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **OTLP\_HTTP\_CLIENT\_KEY\_PEM** `string` : Client Key. Client private key in PEM format for mutual TLS authentication
  * This field is optional
  * Example: `-----BEGIN PRIVATE KEY-----`
* **OTLP\_HTTP\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `OTLP http` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `otlphttp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: otlphttp-example
  namespace: odigos-system
spec:
  data:
    OTLP_HTTP_ENDPOINT: <OTLP http Endpoint>
    # Note: The commented fields below are optional.
    # OTLP_HTTP_BASIC_AUTH_USERNAME: <Basic Auth Username>
    # OTLP_HTTP_OAUTH2_ENABLED: <Enable OAuth2>
    # OTLP_HTTP_OAUTH2_CLIENT_ID: <OAuth2 Client ID>
    # OTLP_HTTP_OAUTH2_TOKEN_URL: <OAuth2 Token URL>
    # OTLP_HTTP_OAUTH2_SCOPES: <OAuth2 Scopes>
    # OTLP_HTTP_OAUTH2_AUDIENCE: <OAuth2 Audience>
    # OTLP_HTTP_COMPRESSION: <Destination Compression Type (default: none) (options: [none, gzip, snappy, lz4, zlib, deflate, zstd])>
    # OTLP_HTTP_HEADERS: <Headers>
    # OTLP_HTTP_TRACES_ENDPOINT: <OTLP http traces Endpoint>
    # OTLP_HTTP_METRICS_ENDPOINT: <OTLP http metrics Endpoint>
    # OTLP_HTTP_LOGS_ENDPOINT: <OTLP http logs Endpoint>
    # OTLP_HTTP_PROFILES_ENDPOINT: <OTLP http profiles Endpoint>
    # OTLP_HTTP_TLS_ENABLED: <Enable TLS>
    # OTLP_HTTP_CA_PEM: <Certificate Authority>
    # OTLP_HTTP_MTLS_ENABLED: <Enable mTLS>
    # OTLP_HTTP_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
  destinationName: otlphttp
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: otlphttp-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  - PROFILES
  type: otlphttp

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   OTLP_HTTP_BASIC_AUTH_PASSWORD: <Base64 Basic Auth Password>
#   OTLP_HTTP_CLIENT_CERT_PEM: <Base64 Client Certificate>
#   OTLP_HTTP_CLIENT_KEY_PEM: <Base64 Client Key>
#   OTLP_HTTP_OAUTH2_CLIENT_SECRET: <Base64 OAuth2 Client Secret>
# kind: Secret
# metadata:
#   name: otlphttp-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f otlphttp.yaml
```


# Prometheus
Source: https://docs.odigos.io/oss/backends/prometheus

Configuring the Prometheus backend (Self-Hosted)

### Getting Started

![prometheus](https://d15jtxgb40qetw.cloudfront.net/prometheus.svg)

> [!NOTE]
>
> These instructions are for the **self hosted** Prometheus instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Prometheus](../backends/grafanacloudprometheus) page.

**Enabling Remote Write in Prometheus**\

Currently, Odigos supports Prometheus via the Remote Write API. In order to enable remote write in Prometheus, set [the following feature flag](https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver).

#### Using Authentication

To enable authentication for Prometheus remote write, check the `Use Authentication` checkbox. Odigos supports two authentication methods:

* **Bearer Token Authentication**: Use the `Bearer Token` field to provide a bearer token for authentication. This method is used if basic auth username is not provided.
* **Basic Authentication**: Use the `Basic Auth Username` and `Basic Auth Password` fields if your Prometheus remote write endpoint requires [basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). Basic auth takes precedence if username is provided.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **PROMETHEUS\_REMOTEWRITE\_URL** `string` : Remote Write URL. The remote write endpoint. If the URL ends with a known remote write path (`/api/v1/write` for Prometheus/Thanos or `/api/v1/push` for Mimir) it is used as-is; otherwise `/api/v1/write` is appended.
  * This field is required
* **PROMETHEUS\_RESOURCE\_ATTRIBUTES\_LABELS** `string[]` : Resource Attributes Labels (Deprecated). deprecated. will be removed soon.
  * This field is optional
* **PROMETHEUS\_USE\_AUTHENTICATION** `boolean` : Use Authentication.
  * This field is optional and defaults to `False`
* **PROMETHEUS\_BEARER\_TOKEN** `string` : Bearer Token. use this bearer token if your prometheus remote write endpoint requires authentication
  * This field is optional
  * Example: `your-bearer-token`
* **PROMETHEUS\_BASIC\_AUTH\_USERNAME** `string` : Basic Auth Username. username for basic authentication (if your prometheus remote write endpoint requires basic auth)
  * This field is optional
  * Example: `username`
* **PROMETHEUS\_BASIC\_AUTH\_PASSWORD** `string` : Basic Auth Password. password for basic authentication (if your prometheus remote write endpoint requires basic auth)
  * This field is optional
  * Example: `password`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Prometheus` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `prometheus.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: prometheus-example
  namespace: odigos-system
spec:
  data:
    PROMETHEUS_REMOTEWRITE_URL: <Remote Write URL>
    # Note: The commented fields below are optional.
    # PROMETHEUS_RESOURCE_ATTRIBUTES_LABELS: <Resource Attributes Labels (Deprecated)>
    # PROMETHEUS_USE_AUTHENTICATION: <Use Authentication>
    # PROMETHEUS_BASIC_AUTH_USERNAME: <Basic Auth Username>
  destinationName: prometheus
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: prometheus-secret
  signals:
  - METRICS
  type: prometheus

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   PROMETHEUS_BASIC_AUTH_PASSWORD: <Base64 Basic Auth Password>
#   PROMETHEUS_BEARER_TOKEN: <Base64 Bearer Token>
# kind: Secret
# metadata:
#   name: prometheus-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f prometheus.yaml
```


# Pyroscope
Source: https://docs.odigos.io/oss/backends/pyroscope

Send continuous profiles from Odigos to Grafana Pyroscope

### Getting Started

![pyroscope](https://d15jtxgb40qetw.cloudfront.net/pyroscope.svg)

[Grafana Pyroscope](https://grafana.com/oss/pyroscope/) is an open-source continuous profiling backend. Odigos exports profile data over OTLP to Pyroscope so the same eBPF profiler that powers in-product views can also feed long-term, queryable storage.

> [!NOTE]
>
> This destination only emits the **Profiles** signal. The destination must include `PROFILES`, and continuous profiling must be enabled for the agent you use:
>
> * **K8s Agent**: [Continuous Profiling overview](/enterprise/instrumentations/profiling/overview) and [getting started](/enterprise/instrumentations/profiling/getting-started)
> * **VM Agent**: [Continuous Profiling overview](/vmagent/instrumentations/profiling/overview) and [getting started](/vmagent/instrumentations/profiling/getting-started)

**Pyroscope version**\

Use a Pyroscope build that accepts OTLP profiles on the endpoint you configure. If you are unsure, upgrade to a recent release before configuring Odigos.

**Quickstart: self-hosted Pyroscope**\

If you don't already run Pyroscope, the upstream Helm chart gets you a working instance in a couple of commands:

```bash theme={null}
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install pyroscope grafana/pyroscope \
  --namespace pyroscope --create-namespace
```

The distributor will be reachable in-cluster at `pyroscope.pyroscope:4040` — use that as the `PYROSCOPE_URL` value below.

> [!WARNING]
>
> Provide the distributor's `host:port` only. Do **not** add `/otlp` or other path suffixes — Odigos sends OTLP profiles to `/v1development/profiles` on the host you configure.

**TLS**\

Toggle `PYROSCOPE_TLS_ENABLED` when Pyroscope is fronted by TLS. For plain traffic, leave it off.

**Verifying the export**\

After applying the destination and sending traffic to a profiled source, open the Pyroscope UI (`kubectl -n pyroscope port-forward svc/pyroscope 4040`) and confirm that profiles appear under your service.

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ❌ Metrics
> ❌ Logs
> ✅ Profiles

* **PYROSCOPE\_URL** `string` : Pyroscope Endpoint. Hostname and port of the Pyroscope distributor HTTP API (e.g. `pyroscope.pyroscope:4040`). Odigos sends OTLP profiles to `/v1development/profiles` on this host; do not add an `/otlp` path.
  * This field is required
  * Example: `host:port`
* **PYROSCOPE\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Pyroscope` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `pyroscope.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: pyroscope-example
  namespace: odigos-system
spec:
  data:
    PYROSCOPE_URL: <Pyroscope Endpoint>
    # Note: The commented fields below are optional.
    # PYROSCOPE_TLS_ENABLED: <Enable TLS>
  destinationName: pyroscope
  signals:
  - PROFILES
  type: pyroscope
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f pyroscope.yaml
```


# qryn
Source: https://docs.odigos.io/oss/backends/qryn

Configuring the qryn backend (Self-Hosted)

### Getting Started

![qryn-oss](https://d15jtxgb40qetw.cloudfront.net/qryn.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **QRYN\_OSS\_URL** `string` : API Url. The API Endpoint for for your qryn instance
  * This field is required
  * Example: `http://qryn.local:3100`
* **QRYN\_OSS\_PASSWORD** `string` : Basic auth password. The HTTP Basic Auth password for your qryn instance
  * This field is optional
* **QRYN\_OSS\_USERNAME** `string` : Basic auth username. The HTTP Basic Auth username for your qryn instance
  * This field is optional
* **QRYN\_OSS\_RESOURCE\_TO\_TELEMETRY\_CONVERSION** `boolean` : Convert container attributes to labels.
  * This field is optional and defaults to `True`
* **QRYN\_OSS\_ADD\_EXPORTER\_NAME** `boolean` : Add exporter name to labels.
  * This field is optional and defaults to `True`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `qryn` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `qryn-oss.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: qryn-oss-example
  namespace: odigos-system
spec:
  data:
    QRYN_OSS_URL: <API Url>
    # Note: The commented fields below are optional.
    # QRYN_OSS_USERNAME: <Basic auth username>
    # QRYN_OSS_RESOURCE_TO_TELEMETRY_CONVERSION: <Convert container attributes to labels (default: True)>
    # QRYN_OSS_ADD_EXPORTER_NAME: <Add exporter name to labels (default: True)>
  destinationName: qryn-oss
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: qryn-oss-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: qryn-oss

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   QRYN_OSS_PASSWORD: <Base64 Basic auth password>
# kind: Secret
# metadata:
#   name: qryn-oss-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f qryn-oss.yaml
```


# Quickwit
Source: https://docs.odigos.io/oss/backends/quickwit

Configuring the Quickwit backend (Self-Hosted)

### Getting Started

![quickwit](https://d15jtxgb40qetw.cloudfront.net/quickwit.svg)

You'll find more details here: [https://quickwit.io/docs/log-management/otel-service](https://quickwit.io/docs/log-management/otel-service)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **QUICKWIT\_URL** `string` : Quickwit OTLP gRPC Endpoint. The unencrypted gRPC endpoint of the Quickwit OTLP receiver (indexer component). The default Quickwit gRPC port if not changed is `7281`
  * This field is required
  * Example: `quickwit.example.com:7281`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Quickwit` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `quickwit.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: quickwit-example
  namespace: odigos-system
spec:
  data:
    QUICKWIT_URL: <Quickwit OTLP gRPC Endpoint>
  destinationName: quickwit
  signals:
  - TRACES
  - LOGS
  type: quickwit
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f quickwit.yaml
```


# Sentry
Source: https://docs.odigos.io/oss/backends/sentry

Configuring the Sentry backend

## Obtaining a Sentry DSN

To use Sentry, you need to obtain a DSN (Data Source Name) from your Sentry account.
You can find this in the Sentry UI under "Client Keys (DSN)".

## Configuring Sentry Backend

Add a new backend with the DSN from your Sentry account.


# Seq
Source: https://docs.odigos.io/oss/backends/seq

Configuring the Seq backend (Self-Hosted)

### Getting Started

![seq](https://d15jtxgb40qetw.cloudfront.net/seq.svg)

* [Deploy Seq into Kubernetes using Docker](https://docs.datalust.co/docs/getting-started-with-docker)
* [Deploy Seq into Kubernetes using Helm](https://docs.datalust.co/docs/using-helm)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ✅ Logs
> ❌ Profiles

* **SEQ\_ENDPOINT** `string` : OTLP HTTP Endpoint. The format is `http://host:port`, `host` is required and equals the cluster internal DNS record of the Seq instance, `port` is required and equals any of the exposed ports of the Seq instance (defaults to 5341), do not add the rest of the path `/ingest/otlp/v1/*`.
  * This field is required
* **SEQ\_API\_KEY** `string` : API Key. The API key is generated in the Seq instance. The API key must have the `Ingest` permission.
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Seq` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `seq.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: seq-example
  namespace: odigos-system
spec:
  data:
    SEQ_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: seq
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: seq-secret
  signals:
  - TRACES
  - LOGS
  type: seq

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   SEQ_API_KEY: <Base64 API Key>
# kind: Secret
# metadata:
#   name: seq-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f seq.yaml
```


# SignalFx
Source: https://docs.odigos.io/oss/backends/signalfx

Configuring the SignalFx backend (Managed)

### Getting Started

![signalfx](https://d15jtxgb40qetw.cloudfront.net/signalfx.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **SIGNALFX\_ACCESS\_TOKEN** `string` : Access Token. The authentication token provided by Splunk Observability Cloud.
  * This field is required
* **SIGNALFX\_REALM** `string` : Realm. The SignalFx realm where your data will be sent (e.g., us0, us1, eu0).
  * This field is required
* **SIGNALFX\_CA\_PEM** `string` : CA Certificate (PEM). Optional CA certificate in PEM format for TLS verification.
  * This field is optional
* **SIGNALFX\_INSECURE\_SKIP\_VERIFY** `boolean` : Skip TLS Verification. If enabled, TLS certificate verification will be skipped. Not recommended for production.
  * This field is optional

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `SignalFx` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `signalfx.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: signalfx-example
  namespace: odigos-system
spec:
  data:
    SIGNALFX_REALM: <Realm>
    # Note: The commented fields below are optional.
    # SIGNALFX_INSECURE_SKIP_VERIFY: <Skip TLS Verification>
  destinationName: signalfx
  # Uncomment the 'secretRef' below if you are using the optional Secret.
  # secretRef:
  #   name: signalfx-secret
  signals:
  - TRACES
  - METRICS
  type: signalfx

---

# The following Secret is optional. Uncomment the entire block if you need to use it.
# apiVersion: v1
# data:
#   SIGNALFX_ACCESS_TOKEN: <Base64 Access Token>
#   SIGNALFX_CA_PEM: <Base64 CA Certificate (PEM)>
# kind: Secret
# metadata:
#   name: signalfx-secret
#   namespace: odigos-system
# type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f signalfx.yaml
```


# SigNoz
Source: https://docs.odigos.io/oss/backends/signoz

Configuring the SigNoz backend (Self-Hosted)

### Getting Started

![signoz](https://d15jtxgb40qetw.cloudfront.net/signoz.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **SIGNOZ\_URL** `string` : Collector Endpoint.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `SigNoz` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `signoz.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: signoz-example
  namespace: odigos-system
spec:
  data:
    SIGNOZ_URL: <Collector Endpoint>
  destinationName: signoz
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: signoz
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f signoz.yaml
```


# Splunk (Legacy) (Deprecated)
Source: https://docs.odigos.io/oss/backends/splunk

Configuring the Splunk (Legacy) (Deprecated) backend (Managed)

### Getting Started

![splunk](https://d15jtxgb40qetw.cloudfront.net/splunk.svg)

**Obtaining Access Token**\

Go to **Settings** then **Access Tokens** and click **Create New Token**

![Splunk Access Tokens menu](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/splunk1.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=3a14b36ce4e63b2b7de2ffb6c679a36d)

Give the token a name and select **Ingest Token** for the authorization scope and click create.

![Create Access Token form in Splunk](https://mintcdn.com/odigos/pHE1teVWLrD8x6Wa/snippets/shared/backends/images/splunk2.png?fit=max\&auto=format\&n=pHE1teVWLrD8x6Wa\&q=85\&s=aca44406c34343662a6c0c0654a2e47c)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required
* **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`.
  * This field is required

> [!NOTE]
>
> This legacy destination now sends traces using OTLP/HTTP because the SAPM exporter was removed upstream.
> Existing `splunk` destinations retain their realm and access token. For new destinations, use [Splunk (OTLP)](/backends/splunkotlp).

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Splunk (Legacy) (Deprecated)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `splunk.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: splunk-example
  namespace: odigos-system
spec:
  data:
    SPLUNK_REALM: <Realm>
  destinationName: splunk
  secretRef:
    name: splunk-secret
  signals:
  - TRACES
  type: splunk

---

apiVersion: v1
data:
  SPLUNK_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: splunk-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f splunk.yaml
```


# Splunk (OTLP)
Source: https://docs.odigos.io/oss/backends/splunkotlp

Configuring the Splunk (OTLP) backend (Managed)

### Getting Started

![splunkotlp](https://d15jtxgb40qetw.cloudfront.net/splunk.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required
* **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`.
  * This field is required
* **SPLUNK\_OTLP\_COMPRESSION** `string` : Compression Type. Compression type to use for the destination. The default is `none`.
  * This field is optional and defaults to `none`
* **SPLUNK\_OTLP\_TLS\_ENABLED** `boolean` : Enable TLS. Secure connection
  * This field is optional and defaults to `False`
* **SPLUNK\_OTLP\_CA\_PEM** `string` : Certificate Authority. When using TLS, provide the CA certificate in PEM format to verify the server. If empty uses system root CA
  * This field is optional
  * Example: `-----BEGIN CERTIFICATE-----`
* **SPLUNK\_OTLP\_INSECURE\_SKIP\_VERIFY** `boolean` : Insecure Skip Verify. Skip TLS certificate verification
  * This field is optional and defaults to `False`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Splunk (OTLP)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `splunkotlp.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: splunkotlp-example
  namespace: odigos-system
spec:
  data:
    SPLUNK_REALM: <Realm>
    # Note: The commented fields below are optional.
    # SPLUNK_OTLP_COMPRESSION: <Compression Type (default: none) (options: [none, gzip])>
    # SPLUNK_OTLP_TLS_ENABLED: <Enable TLS>
    # SPLUNK_OTLP_CA_PEM: <Certificate Authority>
    # SPLUNK_OTLP_INSECURE_SKIP_VERIFY: <Insecure Skip Verify>
  destinationName: splunkotlp
  secretRef:
    name: splunkotlp-secret
  signals:
  - TRACES
  type: splunkotlp

---

apiVersion: v1
data:
  SPLUNK_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: splunkotlp-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f splunkotlp.yaml
```


# Splunk (SAPM) (Deprecated)
Source: https://docs.odigos.io/oss/backends/splunksapm

Configuring the Splunk (SAPM) (Deprecated) backend (Managed)

### Getting Started

![splunksapm](https://d15jtxgb40qetw.cloudfront.net/splunk.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs

* **SPLUNK\_ACCESS\_TOKEN** `string` : Access Token.
  * This field is required
* **SPLUNK\_REALM** `string` : Realm. The Splunk realm in use. This is the second part of the URL you use to access Splunk/SignalFX. For example, if you access Splunk at `https://app.us1.signalfx.com`, then the realm is `us1`.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](../cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Splunk (SAPM) (Deprecated)` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `splunksapm.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: splunksapm-example
  namespace: odigos-system
spec:
  data:
    SPLUNK_REALM: <Realm>
  destinationName: splunksapm
  secretRef:
    name: splunksapm-secret
  signals:
  - TRACES
  type: splunksapm

---

apiVersion: v1
data:
  SPLUNK_ACCESS_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: splunksapm-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f splunksapm.yaml
```


# Sumo Logic
Source: https://docs.odigos.io/oss/backends/sumologic

Configuring the Sumo Logic backend (Managed)

### Getting Started

![sumologic](https://d15jtxgb40qetw.cloudfront.net/sumologic.svg)

Follow the [following instructions](https://help.sumologic.com/docs/send-data/hosted-collectors/http-source/otlp/) to obtain a OTLP/HTTP Source URL.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **SUMOLOGIC\_COLLECTION\_URL** `string` : Source URL.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Sumo Logic` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `sumologic.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: sumologic-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: sumologic
  secretRef:
    name: sumologic-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: sumologic

---

apiVersion: v1
data:
  SUMOLOGIC_COLLECTION_URL: <Base64 Source URL>
kind: Secret
metadata:
  name: sumologic-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f sumologic.yaml
```


# TelemetryHub
Source: https://docs.odigos.io/oss/backends/telemetryhub

Configuring the TelemetryHub backend (Managed)

### Getting Started

![telemetryhub](https://d15jtxgb40qetw.cloudfront.net/telemetryhub.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **TELEMETRY\_HUB\_API\_KEY** `string` : API Key.
  * This field is required

> [!TIP]
>
> We handle the endpoint internally, so you don't need to provide it.
>
> * The endpoint is `https://otlp.telemetryhub.com:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `TelemetryHub` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `telemetryhub.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: telemetryhub-example
  namespace: odigos-system
spec:
  data: {}
  destinationName: telemetryhub
  secretRef:
    name: telemetryhub-secret
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: telemetryhub

---

apiVersion: v1
data:
  TELEMETRY_HUB_API_KEY: <Base64 API Key>
kind: Secret
metadata:
  name: telemetryhub-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f telemetryhub.yaml
```


# Tempo
Source: https://docs.odigos.io/oss/backends/tempo

Configuring the Tempo backend (Self-Hosted)

### Getting Started

![tempo](https://d15jtxgb40qetw.cloudfront.net/tempo.svg)

> [!NOTE]
>
> These instructions are for the **self hosted** Tempo instance. If you need a managed service, please follow the instructions on the [Grafana Cloud Tempo](../backends/grafanacloudtempo) page.

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ❌ Metrics
> ❌ Logs
> ❌ Profiles

* **TEMPO\_URL** `string` : Endpoint.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Tempo` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `tempo.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: tempo-example
  namespace: odigos-system
spec:
  data:
    TEMPO_URL: <Endpoint>
  destinationName: tempo
  signals:
  - TRACES
  type: tempo
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f tempo.yaml
```


# Tingyun 基调听云
Source: https://docs.odigos.io/oss/backends/tingyun

Configuring the Tingyun 基调听云 backend (Managed)

### Getting Started

![tingyun](https://d15jtxgb40qetw.cloudfront.net/tingyun.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **TINGYUN\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
* **TINGYUN\_LICENSE\_KEY** `string` : License Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Tingyun 基调听云` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `tingyun.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: tingyun-example
  namespace: odigos-system
spec:
  data:
    TINGYUN_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: tingyun
  secretRef:
    name: tingyun-secret
  signals:
  - TRACES
  - METRICS
  type: tingyun

---

apiVersion: v1
data:
  TINGYUN_LICENSE_KEY: <Base64 License Key>
kind: Secret
metadata:
  name: tingyun-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f tingyun.yaml
```


# Traceloop
Source: https://docs.odigos.io/oss/backends/traceloop

Configuring the Traceloop backend (Managed)

### Getting Started

![traceloop](https://d15jtxgb40qetw.cloudfront.net/traceloop.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **TRACELOOP\_ENDPOINT** `string` : Traceloop OTLP HTTP Endpoint.
  * This field is required and defaults to `https://api.traceloop.com`
* **TRACELOOP\_API\_KEY** `string` : Traceloop API Key.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Traceloop` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `traceloop.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: traceloop-example
  namespace: odigos-system
spec:
  data:
    TRACELOOP_ENDPOINT: '<Traceloop OTLP HTTP Endpoint (default: https://api.traceloop.com)>'
  destinationName: traceloop
  secretRef:
    name: traceloop-secret
  signals:
  - TRACES
  - METRICS
  type: traceloop

---

apiVersion: v1
data:
  TRACELOOP_API_KEY: <Base64 Traceloop API Key>
kind: Secret
metadata:
  name: traceloop-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f traceloop.yaml
```


# Uptrace
Source: https://docs.odigos.io/oss/backends/uptrace

Configuring the Uptrace backend (Managed)

### Getting Started

![uptrace](https://d15jtxgb40qetw.cloudfront.net/uptrace.svg)

### Configuring Destination Fields

> **Supported Signals:**
>
> ✅ Traces
> ✅ Metrics
> ✅ Logs
> ❌ Profiles

* **UPTRACE\_DSN** `string` : Data Source Name (DSN).
  * This field is required
* **UPTRACE\_ENDPOINT** `string` : Endpoint. Overwrite Uptrace endpoint when self-hosting
  * This field is optional and defaults to `https://otlp.uptrace.dev:4317`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `Uptrace` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `uptrace.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: uptrace-example
  namespace: odigos-system
spec:
  data:
    UPTRACE_DSN: <Data Source Name (DSN)>
    # Note: The commented fields below are optional.
    # UPTRACE_ENDPOINT: <Endpoint (default: https://otlp.uptrace.dev:4317)>
  destinationName: uptrace
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: uptrace
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f uptrace.yaml
```


# VictoriaMetrics Cloud
Source: https://docs.odigos.io/oss/backends/victoriametricscloud

Configuring the VictoriaMetrics Cloud backend (Managed)

### Getting Started

![victoriametricscloud](https://d15jtxgb40qetw.cloudfront.net/victoriametrics.svg)

**Creating Account**\

Go to the **[🔗 Cloud Console](https://console.victoriametrics.cloud) > Sign Up**

**Setting Up**\

Go to **⚙️ > Deployments** and click **Start using VictoriaMetrics Cloud**

**Obtaining Access Token**\

Go to **⚙️ > Deployments** and click **Overview**\

Then click **Access Tokens** and copy one of the keys

**Obtaining Endpoint**\

Go to **⚙️ > Integrations** and click **OpenTelemetry > Operator**\

Scroll to **Kubernetes commandline command** and copy the **endpoint**

### Configuring Destination Fields

> **Supported Signals:**
>
> ❌ Traces
> ✅ Metrics
> ❌ Logs
> ❌ Profiles

* **VICTORIA\_METRICS\_CLOUD\_ENDPOINT** `string` : OTLP HTTP Endpoint.
  * This field is required
* **VICTORIA\_METRICS\_CLOUD\_TOKEN** `string` : Access Token.
  * This field is required

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

**Step 1**

Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

```bash theme={null}
odigos ui
```

**Step 2**

Click on `Add Destination`, select `VictoriaMetrics Cloud` and follow the on-screen instructions

##### **Using Kubernetes manifests**

**Step 1**

Save the YAML below to a file (e.g. `victoriametricscloud.yaml`)

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: victoriametricscloud-example
  namespace: odigos-system
spec:
  data:
    VICTORIA_METRICS_CLOUD_ENDPOINT: <OTLP HTTP Endpoint>
  destinationName: victoriametricscloud
  secretRef:
    name: victoriametricscloud-secret
  signals:
  - METRICS
  type: victoriametricscloud

---

apiVersion: v1
data:
  VICTORIA_METRICS_CLOUD_TOKEN: <Base64 Access Token>
kind: Secret
metadata:
  name: victoriametricscloud-secret
  namespace: odigos-system
type: Opaque
```

**Step 2**

Apply the YAML using `kubectl`

```bash theme={null}
kubectl apply -f victoriametricscloud.yaml
```


# odigos
Source: https://docs.odigos.io/oss/cli/odigos



## odigos

Automate OpenTelemetry Observability in Kubernetes

### Synopsis

Odigos is primarily focused on automating OpenTelemetry observability pipelines for traces, metrics, and logs, without the need for extensive code changes. The core of Odigos functionality lies in the Kubernetes operators it deploys within your cluster, enabling seamless observability.

Key Features of Odigos:

* Automatic creation of OpenTelemetry observability pipelines.
* Simplified tracing, metrics, and log collection.
* Enhanced visibility into your Kubernetes services.
* Streamlined Kubernetes operations with observability at the forefront.

Get started with Odigos today to effortlessly improve the observability of your Kubernetes services!

### Options

```
  -h, --help                  help for odigos
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos cleanup](/cli/odigos_cleanup)	 - Remove Odigos Sources created by the user.
* [odigos describe](/cli/odigos_describe)	 - Show details of a specific odigos entity
* [odigos diagnose](/cli/odigos_diagnose)	 - Diagnose Client Cluster
* [odigos install](/cli/odigos_install)	 - Install and upgrade Odigos
* [odigos pro](/cli/odigos_pro)	 - Manage Odigos onprem tier for enterprise users
* [odigos profile](/cli/odigos_profile)	 - Manage presets of applied profiles to your odigos installation
* [odigos sources](/cli/odigos_sources)	 - Manage Odigos Sources in a cluster
* [odigos ui](/cli/odigos_ui)	 - Start the Odigos UI
* [odigos uninstall](/cli/odigos_uninstall)	 - Uninstall Odigos
* [odigos uninstall-deprecated](/cli/odigos_uninstall-deprecated)	 - Revert all the changes made by the `odigos install` command.
  This command will uninstall Odigos from your cluster. It will delete all Odigos objects
  and rollback any metadata changes made to your objects.
* [odigos upgrade](/cli/odigos_upgrade)	 - Upgrade Odigos
* [odigos version](/cli/odigos_version)	 - Print odigos version.


# odigos describe
Source: https://docs.odigos.io/oss/cli/odigos_describe



## odigos describe

Show details of a specific odigos entity

### Synopsis

This command can be used for troubleshooting and observing odigos state for its various entities.\
It is similar to `kubectl describe` but for Odigos entities.

```
odigos describe [flags]
```

### Examples

```

# Describe a source of kind deployment and name myservice in the default namespace
odigos describe source deployment myservice -n default

# Describe a source of kind deploymentconfig and name myservice in the default namespace (OpenShift)
odigos describe source deploymentconfig myservice -n default

Output:

Name:  myservice
Kind:  deployment
Namespace:  default

Labels:
  Instrumented:  true
  Workload:  instrumented
  Namespace: unset
  Decision: Workload is instrumented because there is an enabled Source for this workload
  Troubleshooting: https://docs.odigos.io/architecture/troubleshooting#1-source-configuration

Instrumentation Config:
  Created at 2024-07-30 19:00:40 +0300 IDT

Runtime inspection details:
  Created at 2024-07-30 19:00:40 +0300 IDT
  Detected Containers:
    - Container Name: myservice
      Language:       javascript
      Relevant Environment Variables:
        - NODE_OPTIONS : --require /var/odigos/nodejs/autoinstrumentation.js

Instrumentation Device:
  Status: Successfully applied instrumentation device to pod template
  - Container Name: myservice
    Instrumentation Devices: javascript-native-community

Pods (Total 1, Running 1):
  Pod Name: myservice-ffd68d8c-qqmxl
  Pod Phase: Running
  Pod Node Name: kind-control-plane
  Containers:
  - Container Name: myservice
    Instrumentation Devices: javascript-native-community
    Instrumentation Instances:
    - Healthy: true
      Reason: Healthy

```

### Options

```
  -h, --help     help for describe
  -r, --remote   use odigos ui service in the cluster to describe the entity
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes
* [odigos describe source](/cli/odigos_describe_source)	 - Show details of a specific odigos source


# odigos diagnose
Source: https://docs.odigos.io/oss/cli/odigos_diagnose



## odigos diagnose

Diagnose Client Cluster

### Synopsis

Retrieves Logs of all Odigos components in the odigos-system namespace and CRDs of Actions, instrumentation resources and more.
The results will be saved in a compressed file for further troubleshooting.
The file will be saved in this format: odigos\_debug\_ddmmyyyyhhmmss.tar.gz

```
odigos diagnose [flags]
```

### Options

```
      --all-workload-namespaces              Collect instrumented source workloads from all namespaces
  -h, --help                                 help for diagnose
      --odigos-configmaps                    Include Odigos ConfigMaps in the diagnose output (default true)
      --odigos-crds                          Include Odigos CRDs in the diagnose output (default true)
      --odigos-logs                          Include Odigos component pod logs in the diagnose output (default true)
      --odigos-metrics                       Include Odigos Prometheus metrics in the diagnose output (default true)
      --odigos-profiles                      Include Odigos pprof profiles in the diagnose output (default true)
      --source-workload-namespaces strings   Collect instrumented source workloads from specific namespaces (comma-separated)
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos install
Source: https://docs.odigos.io/oss/cli/odigos_install



## odigos install

Install and upgrade Odigos

### Synopsis

This subcommand installs or upgrades Odigos in your Kubernetes cluster.
It installs Kubernetes components that auto-instrument your applications with OpenTelemetry
and send traces, metrics, and logs to any telemetry backend.

```
odigos install [flags]
```

### Examples

```

# Install or upgrade Odigos open-source in your cluster with the default values
odigos install

# Install or upgrade Odigos on-prem tier for enterprise users
odigos install --set onPremToken=${ODIGOS_TOKEN}

# Install or upgrade Odigos and set specific values
odigos install --set collectorGateway.minReplicas=5 --set collectorGateway.maxReplicas=10

# Install or upgrade Odigos using a local values.yaml file
odigos install --values ./values.yaml

# Reset all values to chart defaults (opt out of reuse)
odigos install --reset-then-reuse-values=false

```

### Options

```
      --chart string              Helm chart to install (repo/name, local path, or URL) (default "odigos/odigos")
      --chart-version string      Override Helm chart version (defaults to CLI-baked version)
  -h, --help                      help for install
      --ns string                 Target Kubernetes namespace (default "odigos-system")
      --release-name string       Helm release name (default "odigos")
      --reset-then-reuse-values   Reset to chart defaults, then reuse values from the previous release (default: true). (default true)
      --set strings               Set values on the command line (key=value)
  -f, --values string             Path to a custom values.yaml file
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos profile
Source: https://docs.odigos.io/oss/cli/odigos_profile



## odigos profile

Manage presets of applied profiles to your odigos installation

### Synopsis

This command can be used to interact with the applied profiles in your odigos installation.

```
odigos profile [flags]
```

### Examples

```

# Enable payload collection for all supported workloads and instrumentation libraries in the cluster
odigos profile add full-payload-collection

# Remove the full-payload-collection profile from the cluster
odigos profile remove full-payload-collection

```

### Options

```
  -a, --available   list all available profiles to use
  -h, --help        help for profile
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos sources
Source: https://docs.odigos.io/oss/cli/odigos_sources



## odigos sources

Manage Odigos Sources in a cluster

### Synopsis

This command can be used to create, delete, or update Sources to configure workload or namespace auto-instrumentation

### Examples

```
# Create a Source "foo-source" for deployment "foo" in namespace "default"
odigos sources create foo-source --workload-kind=Deployment --workload-name=foo --workload-namespace=default -n default

# Update all existing Sources in namespace "default" to disable instrumentation
odigos sources update --disable-instrumentation -n default

# Delete all Sources in group "mygroup"
odigos sources delete --group mygroup --all-namespaces
	
```

### Options

```
  -h, --help   help for sources
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes
* [odigos sources create](/cli/odigos_sources_create)	 - Create an Odigos Source
* [odigos sources delete](/cli/odigos_sources_delete)	 - Delete Odigos Sources
* [odigos sources disable](/cli/odigos_sources_disable)	 - Disable a source for Odigos instrumentation.
* [odigos sources enable](/cli/odigos_sources_enable)	 - Enable a source for Odigos instrumentation.
* [odigos sources status](/cli/odigos_sources_status)	 - Show the status of all Odigos Sources
* [odigos sources update](/cli/odigos_sources_update)	 - Update Odigos Sources


# odigos ui
Source: https://docs.odigos.io/oss/cli/odigos_ui



## odigos ui

Start the Odigos UI

### Synopsis

Start the Odigos UI. This will start a web server that you can access in your browser and enables you to manage and configure Odigos.

```
odigos ui [flags]
```

### Examples

```

# Start the Odigos UI on http://localhost:3000
odigos ui

# Start the Odigos UI on specific port if 3000 is already in use
odigos ui --port 3456

# Start the Odigos UI and have it manage and configure a specific cluster
odigos ui --kubeconfig <path-to-kubeconfig>

```

### Options

```
      --address string    Address to serve the UI on (default "localhost")
  -h, --help              help for ui
      --port int          Port to listen on (default 3000)
      --remote-port int   Port to forward to the remote UI (default 3000)
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos uninstall
Source: https://docs.odigos.io/oss/cli/odigos_uninstall



## odigos uninstall

Uninstall Odigos

### Synopsis

Revert all the changes made by the odigos install command. This command will uninstall Odigos from your cluster. It will delete all Odigos objects and rollback any metadata changes made to your objects.

```
odigos uninstall [flags]
```

### Examples

```

# Uninstall Odigos
odigos uninstall

```

### Options

```
  -h, --help                  help for uninstall
      --ns string             Target Kubernetes namespace (default "odigos-system")
      --release-name string   Helm release name (default "odigos")
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos upgrade
Source: https://docs.odigos.io/oss/cli/odigos_upgrade



## odigos upgrade

Upgrade Odigos

### Synopsis

Upgrades (or installs) Odigos in your Kubernetes cluster.
This command behaves identically to 'odigos install' and uses the same Helm-based flow.

```
odigos upgrade [flags]
```

### Examples

```

# Upgrade Odigos
odigos upgrade

# Upgrade Odigos with custom values
odigos upgrade --set collectorGateway.maxReplicas=10

# Reset all values to chart defaults (opt out of reuse)
odigos upgrade --reset-then-reuse-values=false

```

### Options

```
      --chart string              Helm chart to install (repo/name, local path, or URL) (default "odigos/odigos")
      --chart-version string      Override Helm chart version (defaults to CLI-baked version)
  -h, --help                      help for upgrade
      --ns string                 Target Kubernetes namespace (default "odigos-system")
      --release-name string       Helm release name (default "odigos")
      --reset-then-reuse-values   Reset to chart defaults, then reuse values from the previous release (default: true). (default true)
      --set strings               Set values on the command line (key=value)
  -f, --values string             Path to a custom values.yaml file
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# odigos version
Source: https://docs.odigos.io/oss/cli/odigos_version



## odigos version

Print odigos version.

### Synopsis

This command is used to print the Odigos version.
Both the CLI version and the Odigos components in your cluster will be printed.

```
odigos version [flags]
```

### Examples

```

# Print the version of odigos CLI and Odigos deployment in your cluster
odigos version

```

### Options

```
      --cli       prints only the CLI version
      --cluster   prints only the version of odigos deployed in the cluster
  -h, --help      help for version
```

### Options inherited from parent commands

```
      --kube-context string   (optional) name of the kubeconfig context to use
      --kubeconfig string     (optional) absolute path to the kubeconfig file (default "KUBECONFIG")
  -v, --verbose               enable verbose output
```

### SEE ALSO

* [odigos](/cli/odigos)	 - Automate OpenTelemetry Observability in Kubernetes


# Custom Resources
Source: https://docs.odigos.io/oss/custom-resources



The different components of the Odigos observability control plane work together
to achieve observability for the cluster. The interaction between the components
is performed via the Kubernetes API server.

The instrumentor is a
[Kubernetes Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/).
It coordinates by writing and watching the odigos custom resources in the
Kubernetes API server.

Odigos adds the following custom resources to the Kubernetes API server:

## Source

This resource defines namespaces and workloads to instrument. It has the following `spec` fields:

| Field | Description | Type | Default | Required |
| - | - | - | - | - |
| `disableInstrumentation` | Excludes the specified workload from instrumentation. | `bool` | `false` | Optional |
| `workload.name` | Name of the workload for this Source. | `string` | n/a | Required |
| `workload.namespace` | Namespace of the workload for this Source. | `string` | n/a | Required |
| `workload.kind` | Kind of the workload for this Source. | `WorkloadKind` | n/a | Required |

Valid values for `WorkloadKind` are:

* `Deployment`
* `DaemonSet`
* `StatefulSet`
* `Namespace`

These values are case-sensitive.

## Destination

This custom resource is used to define the backend destinations for the
observability data. Destinations can be either vendors (Datadog, Honeycomb, etc)
or on premise.

The destination object holds any data that is needed to connect to the backend
system. Notice that sensitive fields such as API keys are stored in a Kubernetes
secret and referenced by the destination object.

## CollectorsGroup

The CollectorsGroup object is used to define a group of collectors that have a
shared role. For example, a collectors group might include collectors that are
deployed as DaemonSet that are responsible for collecting telemetry data, or
collectors that are deployed as Deployment and are responsible for shipping the
data to the selected destinations.


# Debugging Odigos
Source: https://docs.odigos.io/oss/debugging



Different components of Odigos can be debugged in different ways. This page describes how to debug each component.

## Odiglet

First make sure you have a running installation of Odigos on a local Kind cluster.
From the root of the Odigos repository, run:

```bash theme={null}
TAG=<your-odigos-version> make debug-odiglet
```

This command will replace the running Odiglet binary and run it in a container on the Kind cluster.

> [!NOTE]
>
> The new Odiglet will not start until a debug session is attached to it

Now all that left is to attach a remote debugger (using VSCode / GoLand) to `localhost:2345`


# Istio Service Mesh Tracing
Source: https://docs.odigos.io/oss/instrumentations/advanced/Istio



Odigos can collect the distributed traces emitted by the [Istio](https://istio.io/) service mesh and route them through the same pipeline as your application traces. Once enabled, the Envoy sidecars report a span for every meshed hop;connection-level latency, retries, response codes, and mTLS — over OTLP to the Odigos collector, where they are correlated with the spans Odigos already generates for your applications. The result is a single trace that spans both the network and application layers.

> [!NOTE]
>
> This page covers how to point an **existing** Istio mesh at Odigos. It is not a guide to installing or operating Istio. For mesh setup, see the [Istio installation guide](https://istio.io/latest/docs/setup/install/helm/).

## Prerequisites

**1. Kubernetes 1.26 or newer**

**Kubernetes 1.26+ is required** — the `odigos-data-collection-local-traffic` service depends on `internalTrafficPolicy: Local`, which became GA in 1.26.

**2. A running Istio mesh**

Istio's control plane (`istiod`) must be installed in your cluster. Follow the [Istio installation guide](https://istio.io/latest/docs/setup/install/helm/).

**3. Workloads enrolled in the mesh**

Istio only emits spans for workloads that have an Envoy sidecar. Label the namespaces you want traced and restart their workloads so the sidecar is injected:

```bash theme={null}
kubectl label namespace <namespace> istio-injection=enabled
kubectl rollout restart deployment -n <namespace>
```

For the full set of injection options, see Istio's [sidecar injection](https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/) documentation.

## Configuration

**1. Register Odigos as an OpenTelemetry provider**

Add an OpenTelemetry extension provider to Istio's `MeshConfig` that points at the Odigos node collector's OTLP/HTTP endpoint. For background on this field, see Istio's reference for [enabling OpenTelemetry tracing](https://istio.io/latest/docs/tasks/observability/distributed-tracing/opentelemetry/).

Apply it using whichever method you use to manage your mesh:

**Helm**

Add the `meshConfig` block to the values for the `istiod` chart (for example, `istiod-values.yaml`):

```yaml istiod-values.yaml theme={null}
meshConfig:
  enableTracing: true
  extensionProviders:
    - name: odigos-otel
      opentelemetry:
        service: odigos-data-collection-local-traffic.odigos-system.svc.cluster.local
        port: 4318
        http:
          path: "/v1/traces"
          timeout: 5s
        resource_detectors:
          environment: {}
```

```bash theme={null}
helm upgrade istiod istio/istiod -n istio-system --reuse-values -f istiod-values.yaml
```

**istioctl**

Define the `meshConfig` block in an `IstioOperator` configuration file (for example, `istio-otel.yaml`):

```yaml istio-otel.yaml theme={null}
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    extensionProviders:
      - name: odigos-otel
        opentelemetry:
          service: odigos-data-collection-local-traffic.odigos-system.svc.cluster.local
          port: 4318
          http:
            path: "/v1/traces"
            timeout: 5s
          resource_detectors:
            environment: {}
```

```bash theme={null}
istioctl install -f istio-otel.yaml
```

Defining the provider on its own does not start any tracing — it only tells Istio where Odigos is.

> [!NOTE]
>
> Requires **Kubernetes 1.26+**. Below that, `odigos-data-collection-local-traffic` does not exist and Istio has no endpoint to reach.

**2. Enable tracing with the Telemetry API**

Create a `Telemetry` resource in the Istio root namespace (usually `istio-system`) that references the provider. This is what actually turns on span reporting. See the [Istio Telemetry API](https://istio.io/latest/docs/tasks/observability/telemetry/) for scoping options.

```yaml theme={null}
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: odigos-otel
      randomSamplingPercentage: 100
```

> [!TIP]
>
> Leave `randomSamplingPercentage` at `100` so every meshed hop is reported. If you need to reduce volume, do it with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) rather than at the Envoy layer — that way the decision sees the full trace (errors, latency, and downstream spans) instead of being made blindly per request at the sidecar.

**3. Confirm a destination on the default data stream**

The Odigos collector only exports data when at least one destination is configured. Because Envoy emits Istio spans on behalf of the mesh rather than any specific Odigos-instrumented workload, the collector routes them through the **default data stream** — so make sure the destination you want the Istio traces to reach is assigned to the default data stream.

> [!NOTE]
>
> Istio produces spans only for L7 HTTP traffic on Service ports named `http` / `http-*` (or with `appProtocol: http`). Plain TCP protocols such as Redis or AMQP will not generate mesh spans.

## Validation

After traffic flows through a meshed workload, look for Envoy spans in your tracing backend. They appear under service names in the form `<workload>.<namespace>` (for example, `frontend.default`), distinct from the application service names Odigos reports. You can identify them by:

* `otel.scope.name` = `envoy`
* `component` = `proxy`
* `istio.*` resource attributes such as `istio.canonical_service` and `istio.mesh_id`

### Example

Real meshed traces interleave Istio and Odigos spans — every meshed hop adds a pair of Istio sidecar spans (one on each side of the wire) that wrap the callee's application work, which itself can fan out into more application spans and another mesh hop. A two-hop call through `frontend → checkout → payments` lays out like this:

Horizontal position is wall-clock time; label indentation follows the true parent/child nesting. A child bar always starts after and ends within its parent.

```
                                0ms      50       100      150      200ms
                                 ├────────┼────────┼────────┼────────┤
  frontend · POST /checkout       ████████████████████████████████████
    frontend · Istio→checkout          ┏━━━━━━━━━━━━━━━━━━━━━━━━━┓
      checkout · Istio server           ┏━━━━━━━━━━━━━━━━━━━┓
        checkout · POST /checkout        ██████████████████████
          checkout · SELECT cart         ██████
          checkout · Istio→payments              ┏━━━━━━━━━┓
            payments · Istio server               ┏━━━━━━━┓
              payments · POST /charge             █████████
                payments · INSERT txn              ██████

  Legend:  █ Odigos (app)   ┏━┓ Istio (mesh)
```

Each Istio client/server pair brackets the next service's application work. The gap between the two Istio spans is **on-the-wire time** (DNS, TCP, mTLS handshakes, retries); the gap between the Istio server span and the application span is **proxy overhead**. With each additional meshed hop, another sidecar pair is added — both costs become directly attributable instead of being absorbed into a single application-latency number.

> [!NOTE]
>
> Istio spans are additive — they do not replace the application spans Odigos generates. Each meshed HTTP hop yields an Istio client span, an Istio server span, and the application span(s), all sharing the same trace.


# NGINX Ingress Tracing
Source: https://docs.odigos.io/oss/instrumentations/advanced/Nginx



Odigos can collect the distributed traces emitted by the [Ingress-NGINX Controller](https://kubernetes.github.io/ingress-nginx/) and route them through the same pipeline as your application traces. Once enabled, the controller reports a span for every request that enters the cluster through it — route, upstream, response code, and latency — over OTLP to the Odigos collector, where they are correlated with the spans Odigos already generates for your applications. The result is a single trace that begins at the edge and follows the request all the way through your services.

> [!WARNING]
>
> The community Ingress-NGINX project has been **retired** as of March 2026 — it no longer receives releases, bug fixes, or security patches, so running it carries growing security risk. See the [Kubernetes retirement announcement](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/). For new deployments, migrate to the [Gateway API](https://gateway-api.sigs.k8s.io/) instead.

> [!NOTE]
>
> This page covers how to point an **existing** Ingress-NGINX deployment at Odigos. It is not a guide to installing or operating the controller. For controller setup, see the [Ingress-NGINX installation guide](https://kubernetes.github.io/ingress-nginx/deploy/).

> [!NOTE]
>
> Tracing happens at the **Ingress-NGINX Controller**, not in your application's nginx pods. Stock `nginx` images lack the OpenTelemetry module, but the controller bundles it and turns a few ConfigMap keys into the right OTel directives. Put the workloads you want traced behind an Ingress and enable OTel on the controller.

## Prerequisites

**1. Kubernetes 1.26 or newer**

The `odigos-data-collection-local-traffic` service depends on `internalTrafficPolicy: Local`, which became GA in 1.26.

**2. A running Ingress-NGINX Controller**

The Ingress-NGINX Controller must be installed in your cluster, with the OpenTelemetry module available. Follow the [Ingress-NGINX installation guide](https://kubernetes.github.io/ingress-nginx/deploy/).

**3. Workloads exposed through an Ingress**

The controller only emits spans for traffic that flows through it. Make sure the workloads you want traced are fronted by an `Ingress` resource that routes to their `Service`. For background on routing, see the [Ingress-NGINX user guide](https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/).

## Configuration

**1. Point the controller at the Odigos OTLP endpoint**

Configure the Ingress-NGINX Controller to export OpenTelemetry traces to the Odigos node collector's OTLP/gRPC endpoint. For background on these keys, see the controller's [OpenTelemetry documentation](https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/).

Choose one of the methods below based on how you manage the controller. If you installed it with Helm, use Helm values so a later `helm upgrade` does not overwrite the settings; otherwise apply the ConfigMap directly.

**Helm**

Enable the OTel module and set the exporter through the `ingress-nginx` chart values:

```bash theme={null}
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.opentelemetry.enabled=true \
  --set controller.config.enable-opentelemetry="true" \
  --set controller.config.otlp-collector-host="odigos-data-collection-local-traffic.odigos-system.svc.cluster.local" \
  --set controller.config.otlp-collector-port="4317" \
  --set controller.config.otel-service-name="nginx-ingress" \
  --set controller.config.opentelemetry-trust-incoming-span="true" \
  --set controller.config.opentelemetry-operation-name="HTTP \$request_method \$uri"
```

**ConfigMap**

Add the exporter keys to the controller's ConfigMap (named `ingress-nginx-controller` in the `ingress-nginx` namespace by default — match whatever your controller reads via its `--configmap` flag):

```yaml ingress-nginx-controller-cm.yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-opentelemetry: "true"
  otlp-collector-host: "odigos-data-collection-local-traffic.odigos-system.svc.cluster.local"
  otlp-collector-port: "4317"
  otel-service-name: "nginx-ingress"
  opentelemetry-trust-incoming-span: "true"
  opentelemetry-operation-name: "HTTP $request_method $uri"
```

After changing the ConfigMap, roll the controller so it reloads:

```bash theme={null}
kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller
```

> [!WARNING]
>
> **OTLP/gRPC only.** The Ingress-NGINX OTel module speaks OTLP gRPC on port **4317**. There is no HTTP/protobuf (4318) option — pointing it at 4318 will silently fail to export. The `odigos-data-collection-local-traffic` service accepts OTLP/gRPC on 4317.

> [!TIP]
>
> By default the controller samples every request through the ingress, so no sampler settings are needed. If you need to reduce volume, do it with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) rather than at the controller — that way the decision sees the full trace (errors, latency, and downstream spans) instead of being made blindly per request at the edge.

**2. Enable tracing per route**

Enabling OTel on the controller does not trace every route by default. Activate tracing on a route with per-`Ingress` annotations. For the full set of options, see the controller's [OpenTelemetry documentation](https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/).

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/enable-opentelemetry: "true"
    nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true"
spec:
  rules:
    - host: frontend.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend
                port:
                  number: 80
```

**3. Confirm a destination on the default data stream**

The Odigos collector only exports data when at least one destination is configured. Because the controller emits ingress spans on behalf of the edge rather than any specific Odigos-instrumented workload, the collector routes them through the **default data stream** — so make sure the destination you want the NGINX traces to reach is assigned to the default data stream.

## Validation

After traffic flows through the ingress, look for the controller's spans in your tracing backend. They appear under the service name you set in `otel-service-name` (for example, `nginx-ingress`), distinct from the application service names Odigos reports.

Confirm the path end to end. Neither the NGINX OTel module nor the Odigos collector logs anything per span, so **no OTel errors in the logs is the healthy state** — any OTel output is an error. To confirm spans are actually arriving, check your destination.

* **Controller logs** — confirm there are no export errors:

```bash theme={null}
kubectl -n ingress-nginx logs deployment/ingress-nginx-controller | grep -i otel
```

If the collector is unreachable, the gRPC exporter logs an `UNAVAILABLE` error on every flush:

```text theme={null}
[Error] File: .../otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "failed to connect to all addresses; last error: ... ipv4:<host>:4317: Connection refused"
```

That almost always means a wrong `otlp-collector-host`/`otlp-collector-port`, port 4318 instead of 4317, or a NetworkPolicy blocking egress to `odigos-system`.

* **Collector logs** — the collector is also quiet on success. Watch for export errors toward your destination:

```bash theme={null}
kubectl -n odigos-system logs -l odigos.io/collector-role=DATA_COLLECTION --tail=50
```

A misconfigured or unreachable destination shows up as repeated exporter errors. Depending on where the failure happens, you will see either `Exporting failed. Dropping data.` or `Exporting failed. Rejecting data.`:

```text theme={null}
error   exporterhelper/queue_sender.go:50   Exporting failed. Dropping data.   {"otelcol.component.id": "otlp/<destination>", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "...", "dropped_items": 5}
```

* **Your destination** — open your backend and look for a service named after `otel-service-name` (for example, `nginx-ingress`).

### Example

Real traces interleave the NGINX ingress span with the Odigos application spans — the ingress span wraps the upstream's application work, which itself can fan out into more application spans. A request through `ingress → frontend → checkout` lays out like this:

Horizontal position is time; indentation shows parent/child nesting.

```
                                0ms      50       100      150      200ms
                                 ├────────┼────────┼────────┼────────┤
  nginx-ingress · GET /            ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    frontend · GET /                ███████████████████████████████████
      frontend · GET /checkout            ████████████████████████
        checkout · GET /checkout            ██████████████████
          checkout · SELECT cart            ██████

  Legend:  █ Odigos (app)   ┏━┓ NGINX (ingress)
```

The ingress span wraps all the application work behind it. The gap before the first application span is **time spent at the edge** (routing and connecting upstream); everything inside is application latency.

> [!NOTE]
>
> NGINX ingress spans are additive — they do not replace the application spans Odigos generates. Each traced request yields an ingress span plus the application span(s), all sharing the same trace, as long as `opentelemetry-trust-incoming-span` is enabled so context propagates to your workloads.

## Tuning

| Key | Effect |
| - | - |
| `otel-service-name` | The service name spans show up under. Set per environment. |
| `opentelemetry-operation-name` | Template for the span name. `HTTP $request_method $uri` produces names like `HTTP GET /checkout`; supports NGINX variables. |
| `otel-sampler` | Leave at the default (`AlwaysOn`), which captures 100%, and configure sampling with [Odigos tail sampling](../../pipeline/sampling/head-and-tail-sampling) instead. |
| `opentelemetry-trust-incoming-span` | Whether to continue trace context from inbound requests. Keep `true` so ingress spans link to upstream callers and downstream workloads. |
| `nginx.ingress.kubernetes.io/enable-opentelemetry` (annotation) | Per-`Ingress` enablement. Remove to stop tracing a specific route. |

## Troubleshooting

* **No spans at all** — the most common cause is a wrong `otlp-collector-host`. The `odigos-data-collection-local-traffic` service must exist in `odigos-system` and accept OTLP/gRPC. Verify with `kubectl get svc -n odigos-system`.
* **Set 4318 and nothing arrives** — the module is gRPC-only; use 4317.
* **OTel keys added to an application nginx ConfigMap did nothing** — expected; that is the stock-nginx ConfigMap, not the controller's. OTel config belongs in the controller ConfigMap.
* **Config changes ignored** — roll the controller: `kubectl -n ingress-nginx rollout restart deployment ingress-nginx-controller`.


# Agent Mount Method
Source: https://docs.odigos.io/oss/instrumentations/configuration/mount-method

For Odigos agents to run inside instrumented pod containers, certain files must be mounted into the container. In Kubernetes, these files are mounted under `/var/odigos` directory, with subdirectories for each agent. There are few mechanisms to achieve this, which are explained in this document.

## Mount Method

> [!WARNING]
>
> This section is for advanced users and odigos administrators.
>
> It is recommended to use the default settings, unless you have specific requirements or run into an issue.

## Supported Mount Methods

Odigos supports 4 mount methods, which can be used depending on the user preference, cluster policies, integration with existing tools, etc.

### 1. VirtualDevice

> [!NOTE]
>
> This is the default mount method

#### Pod Manifest Additions

For agents and languages that requires filesystem mounts, odigos webhook will add the following:

* For each instrumented container in the pod spec, odigos will add a resource requirement under the `spec.containers[].resources` field:

```yaml theme={null}
    resources:
      limits:
        instrumentation.odigos.io/generic: "1"
      requests:
        instrumentation.odigos.io/generic: "1"
```

#### Caveats

* With [Karpenter](../../setup/odigos-with-karpenter), provisioning can fail because Karpenter does not know about the `instrumentation.odigos.io/generic` extended resource when simulating new nodes. Support depends on your Karpenter version and provider: when [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is available and enabled, apply an overlay that adds this capacity (see the [Karpenter guide](../../setup/odigos-with-karpenter#virtual-device-nodeoverlay)). Otherwise use [`k8s-init-container`](#3-InitContainer), or [`k8s-host-path`](#2-HostPath) / [`k8s-csi-driver`](#4-CSI-Driver) with the Karpenter integration steps.
* Odiglet daemonset needs to be running on a node for instrumented pods to be scheduled on that node.

### 2. HostPath

This method is an opt-in configuration option which can be used if the virtual device method is not suitable for your cluster.

If it is supported in the cluster, it is preferred to use over the VirtualDevice method which requires odiglet component to run.

**Enabling HostPath**

* Profile: `odigos profile add mount-method-k8s-host-path`

* Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-host-path`

* Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-host-path`, or use helm cli `--set instrumentor.mountMethod=k8s-host-path` flag with helm upgrade.

* Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-host-path`.

#### Pod Manifest Additions

For agents and languages that requires filesystem mounts, odigos webhook will add the following:

* Volume to the pod spec, under the `spec.volumes` field:

```yaml theme={null}
  - hostPath:
      path: /var/odigos
      type: ""
    name: odigos-agent
```

* VolumeMount to the instrumented container specs, under the `spec.containers[].volumeMounts` field:

```yaml theme={null}
    - mountPath: /var/odigos/{agent_sub_dir}
      name: odigos-agent
      readOnly: true
      subPath: {agent_sub_dir}
```

#### Caveats

* The "HostPath" option should be enabled in the cluster. Some policy tools may block this, like [Open Policy Agent](https://www.openpolicyagent.org/) or [Kyverno](https://kyverno.io/). If blocked, use the VirtualDevice method, or ask the cluster administrator to whitelist the Odigos agent host path.
* With [Karpenter](../../setup/odigos-with-karpenter), additional configuration is required. By default Odigos adds node affinity for the odiglet-installed label, which blocks Karpenter from provisioning nodes. Enable `karpenter.enabled` and configure the NodePool `startupTaints` as described in the [Karpenter guide](../../setup/odigos-with-karpenter).

### 3. InitContainer

This is an opt-in configuration method designed for scenarios where using hostPath is not feasible due to security policies or virtual device limitations.

For agents and languages that require filesystem mounts, Odigos will inject an InitContainer that copies the necessary files into a shared volume mounted by the application container.

**Enabling InitContainer**

* Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-init-container`
* Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-init-container`, or use helm cli `--set instrumentor.mountMethod=k8s-init-container` flag with helm upgrade.
* Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-init-container`.

#### Pod Manifest Additions

Odigos will add an InitContainer that copies the necessary files for instrumentation, depending on the application language. These files will be placed into a shared volume that is also mounted by the application container.

* Volume to the pod spec, under the `spec.volumes` field:

```yaml theme={null}
  - hostPath:
      path: /var/odigos
      type: ""
    name: odigos-agent
```

#### Caveats

The InitContainer method must be enabled in the cluster and introduces a slight delay during pod initialization.
This approach is ideal for users who prioritize avoiding hostPath or virtual device usage over startup speed.

Works with [Karpenter](../../setup/odigos-with-karpenter) out of the box. No additional Karpenter configuration is required.

#### Init container image pull secrets

Kubernetes 1.35+ enables [`KubeletEnsureSecretPulledImages`](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) by default. The kubelet [re-verifies pull credentials](https://kubernetes.io/docs/concepts/containers/images/#ensure-image-pull-credential-verification) even when the agents image is already cached on the node. The injected `odigos-agents` init container therefore needs the same pull secret in the **application** namespace, not only in the Odigos namespace.

This applies when you use `k8s-init-container` with a private registry:

* the enterprise registry (`registry.odigos.io` via `odigos-enterprise-registry`, including `externalOnpremPullSecret` / `externalOnpremTokenSecret`)
* a private mirrored registry (`imagePrefix` + `imagePullSecrets`)

On Kubernetes 1.35+, the pod webhook adds those secret **names** to instrumented pods. You must create the same-named dockerconfig secret in every instrumented namespace yourself (External Secrets, Reflector, or a GitOps manifest). Below 1.35, odiglet pre-pulls the agents image onto each node and kubelet will use that cache without per-pod credentials.

### 4. CSI Driver

This is an opt-in configuration method that uses the Container Storage Interface (CSI) standard for mounting instrumentation files. It provides improved user experience over the VirtualDevice method by ensuring pods wait gracefully when the mount system isn't ready.

**Enabling CSI Driver**

* Odigos CLI: `odigos install --set instrumentor.mountMethod=k8s-csi-driver`
* Helm Chart: in your values file, set `instrumentor.mountMethod` to `k8s-csi-driver`, or use helm cli `--set instrumentor.mountMethod=k8s-csi-driver` flag with helm upgrade.
* Kubernetes Manifest: under the `odigos-configuration` ConfigMap in odigos namespace, set the value in the `mountMethod` field of `config.yaml` to `k8s-csi-driver`.

#### Pod Manifest Additions

For agents and languages that requires filesystem mounts, odigos webhook will add the following:

* Volume to the pod spec, under the `spec.volumes` field:

```yaml theme={null}
  - name: odigos-agent
    csi:
      driver: odigos.csi.driver
      volumeAttributes:
        type: instrumentation
```

* VolumeMount to the instrumented container specs, under the `spec.containers[].volumeMounts` field:

```yaml theme={null}
    - mountPath: /var/odigos/{agent_sub_dir}
      name: odigos-agent
      readOnly: true
      subPath: {agent_sub_dir}
```

#### Benefits

* **Better User Experience**: Pods wait in "Creating" state until CSI driver is ready.
* **Standard Compliance**: Uses Kubernetes CSI standard for storage integration.
* **Same Functionality**: Provides identical mount behavior as the VirtualDevice method.

#### Caveats

* Requires CSI support in the cluster (available in Kubernetes 1.13+).
* Like VirtualDevice, requires odiglet daemonset to be running on nodes for instrumented pods to be scheduled.
* With [Karpenter](../../setup/odigos-with-karpenter), additional configuration is required. By default Odigos adds node affinity for the odiglet-installed label, which blocks Karpenter from provisioning nodes. Enable `karpenter.enabled` and configure the NodePool `startupTaints` as described in the [Karpenter guide](../../setup/odigos-with-karpenter).


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/dotnet/enrichment



Odigos will automatically instrument your .NET services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project by running:

```bash theme={null}
dotnet add package OpenTelemetry
```

## Creating Activity

To create a new `Activity`, use the `ActivitySource` class.

```csharp theme={null}
using System.Diagnostics;

public static class Telemetry
{
    // Name it after the service name for your app.
    // It can come from a config file, constants file, etc.
    public static readonly ActivitySource MyActivitySource = new(TelemetryConstants.ServiceName);

    public static void MyMethod()
    {
        using (var activity = MyActivitySource.StartActivity("MyMethod"))
        {
            // Your code here
        }
    }
}
```

## Additional Information

For more use cases, see the [OpenTelemetry .NET API documentation](https://opentelemetry.io/docs/languages/net/instrumentation/#creating-activities).


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/dotnet/native



> [!NOTE]
>
> Native Instrumentation is currently the default setting of Odigos **Open-Source** version.
>
> eBPF-Based Instrumentation is currently not supported for this language.

## Supported Versions

Supported versions of the .NET Framework range from **4.6.2** to **10.0.3**.

The following CPU architectures are supported:

* amd64 (x86\_64)
* arm64

## Instrumentation Libraries

The following .NET modules will be auto instrumented by Odigos:

### Application Frameworks

| Framework | Tracing Support | Metrics Support | Notes |
| - | :-: | :-: | - |
| [ASP.NET](https://docs.microsoft.com/aspnet/overview) | Yes | Yes | |
| [ASP.NET Core](https://docs.microsoft.com/aspnet/core/introduction-to-aspnet-core?view=aspnetcore-6.0) | Yes | Yes | |

### Databases

| Library | Tracing Support | Metrics Support | Notes |
| - | - | - | - |
| [Microsoft.Data.SqlClient](https://www.nuget.org/packages/Microsoft.Data.SqlClient) | Yes | | |
| [System.Data.SqlClient](https://www.nuget.org/packages/System.Data.SqlClient) | Yes | | |
| MySql.Data | Partial | | This is the official MySQL library. |
| [MySqlConnector](https://mysqlconnector.net/) | Yes | | Seems to be the [recommended library for MariaDB](https://mariadb.com/docs/connectors/mariadb-connector-net/mariadb-connector-net-guide). |
| [Npgsql](https://www.npgsql.org/) | Yes | | |
| [Microsoft.Data.SqlLite](https://docs.microsoft.com/dotnet/standard/data/sqlite/?tabs=netcore-cli) | | | |
| [MongoDB.Driver](https://www.nuget.org/packages/mongodb.driver) | Yes | | |
| [StackExchange.Redis](https://stackexchange.github.io/StackExchange.Redis/) | Partial | | |
| [Elasticsearch-net](https://github.com/elastic/elasticsearch-net) | Yes | | 8.0.0+ only. |
| [Oracle.ManagedDataAccess.Core](https://www.nuget.org/packages/Oracle.ManagedDataAccess.Core) | Yes | | 23.4.0+ only. |
| [Oracle.ManagedDataAccess](https://www.nuget.org/packages/Oracle.ManagedDataAccess) | Yes | | 23.4.0+ only. |
| [AWSSDK.DynamoDBv2](https://www.nuget.org/packages/AWSSDK.DynamoDBv2) | | | |
| [Microsoft.Azure.Cosmos](https://www.nuget.org/packages/Microsoft.Azure.Cosmos) | | | |

### Inter-process communication (IPC)

| Library | Tracing Support | Metrics Support | Notes |
| - | - | - | - |
| [HttpClient](https://docs.microsoft.com/dotnet/api/system.net.http.httpclient) | Yes | Yes | |
| [HttpWebRequest](https://docs.microsoft.com/dotnet/api/system.net.httpwebrequest) | Yes | Yes | |
| [WCF](https://docs.microsoft.com/dotnet/framework/wcf/whats-wcf) | Yes (Client and Service side on .NET Framework) | | |
| [CoreWCF](https://github.com/CoreWCF/CoreWCF) | | | |
| [RestSharp](https://restsharp.dev/) | | | This library may be implicitly supported by instrumenting the underlying HttpClient or HttpWebRequest. |
| [gRPC-dotnet](https://github.com/grpc/grpc-dotnet) | Partial | | Client and service support should be added. |
| [GraphQL](https://www.nuget.org/packages/GraphQL/) | Partial | | |
| [GraphQL Client](https://github.com/graphql-dotnet/graphql-client) | | | |
| [RabbitMQ](https://www.nuget.org/packages/RabbitMQ.Client) | Yes | | `RabbitMQ.Client` 7.0.0+ has native support for traces. |
| [Kafka](https://www.nuget.org/packages/Confluent.Kafka) | | | |
| [NServiceBus](https://docs.particular.net/nservicebus/) | Yes | Yes | |
| [MassTransit](https://masstransit-project.com/) | Yes | | |

## Automatic Testing

The following scenarios are automatically tested with every release:

| .NET Version | LibC type | CPU |
| - | - | - |
| 6.0 | glibc | amd64 |
| 6.0 | musl | amd64 |
| 8.0 | glibc | amd64 |
| 8.0 | musl | amd64 |


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/golang/ebpf



> [!NOTE]
>
> eBPF-Based Instrumentation is currently the default setting of Odigos **Open-Source** version, and is also available with the Odigos **Enterprise** plan.
>
> Native Instrumentation is currently not supported for this language.

## Supported Versions

Odigos uses the official [opentelemetry-go-instrumentation](https://github.com/open-telemetry/opentelemetry-go-instrumentation) OpenTelemetry Auto Instrumentation using eBPF, thus it supports the same golang versions as this project.

* Go runtime versions **1.19** through **1.26** are supported.

> [!NOTE]
>
> Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.
>
> The required Linux capabilities depend on your kernel version:
>
> * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`.
> * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`.
>
> We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`.

## Traces

Odigos will automatically instrument your golang services to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Go official auto Instrumentation Libraries.

## Instrumentation Libraries

The following go modules will be auto instrumented by Odigos:

### HTTP Frameworks

* [`github.com/gin-gonic/gin`](https://pkg.go.dev/github.com/gin-gonic/gin) HTTP web framework ⭐️
* [`net/http`](https://pkg.go.dev/net/http) go standard library for http server and client

> [!NOTE]
>
> Please note that in order for the `http.route` parameter to be properly extracted by Go auto instrumentation, you should use Go 1.22 or above and
> declare a template handler as described in [this article](https://go.dev/blog/routing-enhancements).

### HTTP Clients

* [`net/http`](https://pkg.go.dev/net/http) go standard library for http server and client

### Database Clients, ORMs, and data access libraries

* [`github.com/gocql/gocql`](https://pkg.go.dev/github.com/gocql/gocql) ⭐️ versions `>= v1.0.0`. database client library for Apache Cassandra
* [`github.com/elastic/go-elasticsearch/v8`](https://pkg.go.dev/github.com/elastic/go-elasticsearch/v8) ⭐️ versions >= v8.0.0
* [`database/sql`](https://pkg.go.dev/database/sql) go standard library for SQL database access

### Messaging Systems Clients

* [`github.com/rabbitmq/amqp091-go`](https://pkg.go.dev/github.com/rabbitmq/amqp091-go) ⭐️ versions `>= v1.4.0`. messaging client for RabbitMQ
* [`github.com/segmentio/kafka-go`](https://pkg.go.dev/github.com/segmentio/kafka-go) versions `>= v0.4.1`. messaging client for Apache Kafka
* [`github.com/apache/pulsar-client-go`](https://pkg.go.dev/github.com/apache/pulsar-client-go) ⭐️ versions `>= v0.12.0`. messaging client for Apache Pulsar
* [`github.com/IBM/sarama`](https://github.com/IBM/sarama) ⭐️ versions `>= v1.40.0`. messaging client for Apache Kafka

### RPC (Remote Procedure Call)

* [`google.golang.org/grpc`](https://pkg.go.dev/google.golang.org/grpc) versions `>= v1.14.0`. rpc client and server for gRPC framework

## Custom Instrumentations

We support adding custom eBPF-based instrumentations for arbitrary Go functions and invokable sites that exist in your application code or its dependencies.
By defining custom instrumentation rules, you can extend Odigos's auto-instrumentation capabilities and cover your specific use cases and application logic.
To learn more about creating custom eBPF-based instrumentations for Go applications, refer to the [Custom eBPF Instrumentation for Go Applications](../../pipeline/rules/custominstrumentation) guide.

## About Go Offsets

Auto-instrumentation for Go with eBPF works by using eBPF uprobes to dynamically read variables in memory at runtime. For example,
when instrumenting the `net/http` package Odigos is able to determine the `http.request.method` by probing the [`Method`](https://pkg.go.dev/net/http#Request)
field in the current `Request`.

Doing so relies on knowing the exact memory location of this field. This is known as the field's "offset" (because it refers to the field's location as a
number of bytes offset from the start of the current struct). With this information, our eBPF code can access the value directly.

However, most production Go programs do not preserve this offset information at compile time. Only Go programs compiled with [debug information](https://go.dev/doc/gdb)
will include this information by default. To work with all Go programs, Odigos ships with a precompiled list of known offsets for all currently supported
versions of instrumented libraries.

This precompiled list is updated to be up-to-date with every release of Odigos. But, this means that you may encounter errors instrumenting applications
that use very newly released versions of dependencies. If this occurs, it will be resolved by updating to the next release of Odigos, or alternatively
upgrading the locally-stored offsets with the [`odigos pro update-offsets`](../../../enterprise/cli/odigos_pro_update-offsets) command.

### Updating Offsets Between Versions

It is possible to pull the latest Go library support for Odigos without updating the entire Odigos installation. There are 3 ways to do so:

1. Manual updates (with [`odigos pro update-offsets`](../../../enterprise/cli/odigos_pro_update-offsets)).
2. Direct automatic updates (recommended, see below).
3. Mirrored image automatic updates (for disconnected environments, see below).

In both automatic modes, a Kubernetes CronJob is created which updates the locally stored Go library offsets used by the Odiglet. The differences
between these modes and how to enable them are described below.

#### Direct automatic updates

In "direct" mode, the CronJob will pull the latest Go library offsets directly from the Odigos cloud server and apply it to the cluster. This
option consumes the fewest resources but requires that your cluster has direct internet access to this file hosted on the web. If you have
restricted Internet egress policies, see "mirrored image updates" below.

Enable automatic updates on any standard cron schedule by setting the `goAutoOffsetsCron` option during installation or upgrade.
For example, enable nightly updates at midnight with `odigos install --set goAutoOffsetsCron="0 0 * * *"`.

To disable automatic updates, set the cron value to an empty string with `odigos install --set goAutoOffsetsCron=""`.

Alternatively, you can set these values in your `values.yaml` file for Helm installations:

```yaml theme={null}
goAutoOffsetsCron: "0 0 * * *"  # Enable nightly updates at midnight
goAutoOffsetsMode: "direct"     # Use direct mode
```

#### Mirrored image automatic updates

For clusters with restricted traffic policies, you have the option to [mirror the Go library updates as an image](../../../setup/docker-registry#install-odigos-images-from-a-custom-docker-registry).
In this mode, the CronJob will run an image that contains the latest Go library offsets bundled into the image itself. This mode
does not require direct Internet access from the cluster, but it does require you to mirror this image into your own registry and maintain
the latest version on your nodes.

To enable mirrored image updates, run `odigos install --set goAutoOffsetsMode="image"`. Then use the `goAutoOffsetsCron` option described above to set the schedule.

To disable mirrored image updates, run `odigos install --set goAutoOffsetsMode="direct"` (to switch back to direct mode), or set the `goAutoOffsetsCron` setting
to an empty string (`""`) as described above to disable automatic updates entirely.


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/golang/enrichment



Odigos will automatically instrument your Go services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Odigos fully supports the OpenTelemetry Go eBPF Auto SDK. See the [official docs on the Auto SDK](https://opentelemetry.io/docs/zero-code/go/autosdk/)
for information on using it.

Add the following dependencies to your project by running:

```bash theme={null}
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace
```

## Spans

Spans are the OpenTelemetry representation of operations in the execution of your code. You should record a span if you think the operation is meaningful and that observing it will help you understand the behavior of your system and operate it.

A span includes a name, start and end time, and key-value attributes for detailed information about the execution.

## Tracer

In OpenTelemetry, spans are created using the `Tracer` object which you can acquire from the OpenTelemetry API.

Each tracer is associated with a name and optional version.
These name and version are recorded as "instrumentation scope" into all spans created by this tracer.
Humans viewing the trace and data processors downstream can use this information to understand the origin of each span.

It is best practice to use the go module and package name for the tracer name.

```go theme={null}
package main

import (
   "context"
   "go.opentelemetry.io/otel"
)

var tracer = otel.Tracer("github.com/<yourusername>/<reponame>/<packagename>")
```

> [!WARNING]
>
> **Note**: Manually initializing a global TracerProvider will conflict with the Go Auto SDK's ability to link
> eBPF spans to custom spans and lead to broken trace IDs. If you are enriching your Odigos application with
> custom spans, do not initialize a global TracerProvider. See the [OpenTelemetry docs](https://opentelemetry.io/docs/zero-code/go/autosdk/)
> for more information.

## Creating Spans

```go theme={null}
func my_function(ctx context.Context) {
   // Span start time is set here
   ctx, span := tracer.Start(ctx, "my_function")
   // Span end time is set here
   defer span.End()
   // ...Do some work, use ctx if needed when calling next functions
}
```

Important Notes:

1. **Always End a span**:
   Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function.
2. **Propagate and use a valid context object**:
   When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace.
3. **Pass the context object downstream**:
   When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow.
4. **Assign meaningful names to spans**:
   Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context.
5. **Avoid dynamic, high cardinality data in span names**:
   Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

### Recording Span Attributes

Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting

Examples:

* User ID, organization ID, Account ID or other identifiers.
* Inputs - the relevant parameters or configuration that influenced the operation.
* Outputs - the result of the operation.
* Entities - the entities or resources that the operation interacted with.

Attribute names are lowercased strings in the form `my_application.my_attribute`, example: `my_service.user_id`.
Read more [here](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/)

To record attributes, use the `SetAttributes` method on the span object.

```go theme={null}
import (
   "context"
   "go.opentelemetry.io/otel/attribute"
)

func my_function(ctx context.Context, userId string) {
   _, span := tracer.Start(ctx, "my_function")
   defer span.End()
   span.SetAttributes(attribute.String("my_service.user_id", userId))
   // ...Do some work
}
```

Important Notes:

1. **Be cautious when recording data**:
   Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
2. **Attribute cost considerations**:
   Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
3. **Use static names for attributes**:
   Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute\_name) to provide clear context for downstream consumers.
4. **Adhere to OpenTelemetry semantic conventions**:
   Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.

### Recording Errors

To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts.

If an operation you are tracing fails, you can mark the span's status as an error and record the error details within the span. Here's how you can do it:

```go theme={null}
import (
   "context"
   "go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

func my_function(ctx context.Context, userId string) {
   _, span := tracer.Start(ctx, "my_function")
   defer span.End()
   err := doSomeWork() // ...Do some work
   if err != nil {
      span.SetStatus(codes.Error, "Operation failed")
      span.RecordError(err)
   }
}
```

## Additional Information

For more use cases, see the [OpenTelemetry Go API documentation](https://opentelemetry.io/docs/languages/go/instrumentation/#creating-spans).


# Go Hooks
Source: https://docs.odigos.io/oss/instrumentations/golang/go-hooks



> [!NOTE]
>
> Go hooks are currently available with the Odigos **Enterprise** plan.\\
>
> [Contact us](https://odigos.io/) for more information.

Odigos provides Go functions to augment and enhance auto-instrumentation. These functions are intended
to be used in conjunction with auto-instrumentation to provide enriched functionality and interaction for
developers running in an Odigos environment.


# eBPF-Based Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/java/ebpf



> [!NOTE]
>
> eBPF-Based Instrumentation is currently available with the Odigos **Enterprise** plan.\\
>
> [Contact us](https://odigos.io/) for more information.
>
> For Native Instrumentation [click here](../../instrumentations/java/native).


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/java/enrichment



Odigos will automatically instrument your Java services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project:

**Maven**

```xml theme={null}
<project>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry</groupId>
                <artifactId>opentelemetry-bom</artifactId>
                <version>1.35.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-api</artifactId>
        </dependency>
    </dependencies>
</project>
```

**Gradle**

```groovy theme={null}
dependencyManagement {
    imports {
        mavenBom("io.opentelemetry:opentelemetry-bom:1.35.0")
    }
}

dependencies {
implementation("org.springframework.boot:spring-boot-starter-web");
implementation("io.opentelemetry:opentelemetry-api");
}

```

If you are not using Spring and its `io.spring.dependency-management` dependency management plugin, install the OpenTelemetry BOM and API using Gradle dependencies only.

```groovy theme={null}
dependencies {
    implementation(platform("io.opentelemetry:opentelemetry-bom:1.35.0"));
    implementation("io.opentelemetry:opentelemetry-api");
}
```

## Creating Spans

To create a span, use the `Tracer` interface. The `Tracer` interface is the main entry point for the OpenTelemetry API. It allows you to create spans and manage the current span.
Acquire a `Tracer` instance using the `OpenTelemetry` class:

```java theme={null}
import io.opentelemetry.api.GlobalOpenTelemetry;

Tracer tracer = GlobalOpenTelemetry.getTracer("instrumentation-scope-name", "instrumentation-scope-version");
```

Now you can create a span using the `Tracer` instance:

```java theme={null}
Span span = tracer.spanBuilder("span-name").startSpan();

try (Scope scope = span.makeCurrent()) {
    // Your code goes here.
} finally {
    span.end();
}
```

## Additional Information

For more use cases, see the [OpenTelemetry Java API documentation](https://opentelemetry.io/docs/languages/java/instrumentation/#create-spans).


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/java/native



> [!NOTE]
>
> Native Instrumentation is currently the default setting of Odigos **Open-Source** version.
>
> eBPF-Based Instrumentation is currently available with the Odigos Enterprise plan.
> Contact us for more information.

## Supported Versions

Odigos uses the official OpenTelemetry instrumentation libraries and SDK components, thus it supports the same Java versions as this project.

* Java Virtual Machine runtime versions **8** and above are supported.

## Traces

Odigos will automatically instrument your Java sources to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Java Official Instrumentation Libraries.

If you use one of the supported logger libraries, the trace id and span id will be auto-injected into your log records allowing signals correlations between logs and traces.

## Instrumentation Libraries

The following Java modules will be auto instrumented by Odigos:

### HTTP Frameworks

* [`play-ws`](https://central.sonatype.com/artifact/com.typesafe.play/play-ws_3) `versions >=1.0.0 <2.1.0`
* [`jaxrs`](https://central.sonatype.com/artifact/com.jslsolucoes/jaxrs) `versions >=0.5`
* [`resteasy-jaxrs`](https://central.sonatype.com/artifact/org.jboss.resteasy/resteasy-jaxrs) `versions >=3.0.0 <4.0.0`
* [`jersey-server`](https://central.sonatype.com/artifact/com.sun.jersey/jersey-server) `versions >=2.0.0 <3.0.0`
* [`jersey-container-servlet`](https://central.sonatype.com/artifact/org.glassfish.jersey.containers/jersey-container-servlet) `versions >=2.0.0 <3.0.0`
* [`spring-web`](https://central.sonatype.com/artifact/org.springframework/spring-web) `versions >=3.1.0`
* [`spring-webflux`](https://central.sonatype.com/artifact/org.springframework/spring-webflux) `versions >=5.0.0`
* [`spring-webmvc`](https://central.sonatype.com/artifact/org.springframework/spring-webmvc) `versions >=3.1.0`
* [`javalin`](https://central.sonatype.com/artifact/io.javalin/javalin) `versions >=5.0.0`
* [`finatra-http`](https://central.sonatype.com/artifact/com.twitter/finatra-http_2.13) `versions >=2.9.0`
* [`ktor-server-core`](https://central.sonatype.com/artifact/io.ktor/ktor-server-core) `versions >=2.0.0`

### HTTP Clients

* [`okhttp`](https://central.sonatype.com/artifact/com.squareup.okhttp/okhttp) `versions >=3.0.0`
* [`httpasyncclient`](https://central.sonatype.com/artifact/org.apache.httpcomponents/httpasyncclient) `versions >=4.1`
* [`reactor-netty-http`](https://central.sonatype.com/artifact/io.projectreactor.netty/reactor-netty-http) `versions >=1.0.0`
* [`cxf-rt-frontend-jaxrs`](https://central.sonatype.com/artifact/org.apache.cxf/cxf-rt-frontend-jaxrs) `versions >=3.2 <4.0.0`

### Database Clients, ORMs, and Data Access Libraries

* [`jedis`](https://central.sonatype.com/artifact/redis.clients/jedis) `versions >=1.4.0 <4.0.0`
* [`couchbase-java-client`](https://central.sonatype.com/artifact/org.testcontainers/couchbase) `versions >=2.0.0`
* [`spring-data-commons`](https://central.sonatype.com/artifact/org.springframework.data/spring-data-commons) `versions >=1.8.0`
* [`spring-batch-core`](https://central.sonatype.com/artifact/org.springframework.batch/spring-batch-core) `versions >=3.0.0 <5.0.0`
* [`elasticsearch-java`](https://central.sonatype.com/artifact/co.elastic.clients/elasticsearch-java) `versions >=7.16 <7.17.20`, `versions >=8 <8.10`
* [`elasticsearch-rest-client`](https://central.sonatype.com/artifact/org.elasticsearch.client/elasticsearch-rest-client) `versions >=6.4 <7.0`
* [`elasticsearch-transport`](https://central.sonatype.com/artifact/org.elasticsearch/elasticsearch-transport-wares) `versions >=5.0 <8.0`
* [`elasticsearch-rest`](https://central.sonatype.com/artifact/org.apache.camel/camel-elasticsearch-rest) `versions >=7.0`

### Messaging Systems Clients

* [`spring-rabbit`](https://central.sonatype.com/artifact/org.springframework.amqp/spring-rabbit) `versions >=1.6.4 <2.1.1`
* [`spring-kafka`](https://central.sonatype.com/artifact/org.springframework.kafka/spring-kafka) `versions >=2.7`
* [`rocketmq-client-java`](https://central.sonatype.com/artifact/org.apache.rocketmq/rocketmq-client-java) `versions >=4.0.0`
* [`reactor-kafka`](https://central.sonatype.com/artifact/io.projectreactor.kafka/reactor-kafka) `versions >=1.0.0`
* [`spring-jms`](https://central.sonatype.com/artifact/org.springframework/spring-jms) `versions >=2.0`

### RPC (Remote Procedure Call)

* [`dubbo`](https://central.sonatype.com/artifact/com.alibaba/dubbo) `versions >=2.7`
* [`grpc-core`](https://central.sonatype.com/artifact/io.grpc/grpc-core) `versions >=1.6.0`
* [`finagle-http`](https://central.sonatype.com/artifact/com.twitter/finagle-http_2.12) `versions >=23.11.0`

### Security

* [`spring-security-config`](https://central.sonatype.com/artifact/org.springframework.security/spring-security-config) `versions >=6.0.0`

### Container Orchestration

* [`kubernetes-client`](https://central.sonatype.com/artifact/io.fabric8/kubernetes-client) `versions >=7.0.0`

### Java Core Modules

* [`spring-aop`](https://central.sonatype.com/artifact/org.springframework/spring-aop) `versions >=1.2`
* [`spring-context`](https://central.sonatype.com/artifact/org.springframework/spring-context) `versions >=4.0.0 <6.0.0`
* [`spring-core`](https://central.sonatype.com/artifact/org.springframework/spring-core) `versions >=2.0.`
* [`spring-boot-actuator-autoconfigure`](https://central.sonatype.com/artifact/org.springframework.boot/spring-boot-actuator-autoconfigure) `versions >=2.0.0`
* [`spring-cloud-gateway-core`](https://central.sonatype.com/artifact/org.springframework.cloud/spring-cloud-gateway-core) `versions >=2.0.0`
* [`spring-ws-core`](https://central.sonatype.com/artifact/org.springframework.ws/spring-ws-core) `versions >=2.0.0`
* [`spring-integration-core`](https://central.sonatype.com/artifact/org.springframework.integration/spring-integration-core) `versions >=4.1.0`
* [`reactor-core`](https://central.sonatype.com/artifact/io.projectreactor/reactor-core) `versions >=3.1.0`
* [`reactor-netty`](https://central.sonatype.com/artifact/io.projectreactor.netty/reactor-netty) `versions >=0.7.0`


# eBPF-Based Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/nodejs/ebpf



> [!NOTE]
>
> eBPF-Based Instrumentation is currently available with the Odigos **Enterprise** plan.\\
>
> [Contact us](https://odigos.io/) for more information.
>
> For Native Instrumentation [click here](../../instrumentations/nodejs/native).


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/nodejs/enrichment



Odigos will automatically instrument your Node.js services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following npm packages to your service by running:

```bash theme={null}
npm install @opentelemetry/api@1.9.0
```

> [!WARNING]
>
> Odigos agent implements OpenTelemetry API version 1.9.0. Any version greater than 1.9.0 may not be compatible with Odigos agent and fail to produce data.\\
>
> Please do not use caret range ~~`@opentelemetry/api@^1.9.0`~~ for this dependency in your package.json to avoid pulling in incompatible version.

## Creating Spans

To enhance your traces with spans for your custom business logic operations, you need to create a span in your code and wrap the operation you want to observe.

The `tracer` object is the entry point for the OpenTelemetry API.

You can create a `tracer` object using the `getTracer` method from the `@opentelemetry/api` package.

**JavaScript**

```javascript theme={null}
const opentelemetry = require('@opentelemetry/api');

const tracer = opentelemetry.trace.getTracer(
   'instrumentation-scope-name',
   'instrumentation-scope-version',
);

function myFunction() {
   const span = tracer.startSpan('myFunction');
   // Your code here
   span.end();
}
```

**TypeScript**

```typescript theme={null}
import opentelemetry from '@opentelemetry/api';

const tracer = opentelemetry.trace.getTracer(
   'instrumentation-scope-name',
   'instrumentation-scope-version',
);

function myFunction() {
   const span = tracer.startSpan('myFunction');
   // Your code here
   span.end();
}
```

Make sure to replace `instrumentation-scope-name` and `instrumentation-scope-version` with the name and version of your instrumented file.

Important Notes:

1. **Always End a span**:
   Ensure that every span is ended to appear in your trace. Defer the End method of the span to guarantee that the span is always ended, even with multiple return paths in the function.
2. **Propagate and use a valid context object**:
   When calling tracer.Start, use a valid context object instead of context.Background(). This makes the new span a child of the active span, ensuring it appears correctly in the trace.
3. **Pass the context object downstream**:
   When calling downstream functions, pass the context object returned from tracer.Start() to ensure any spans created within these functions are children of the current span. This maintains the hierarchical relationship between spans and provides a clear trace of the request flow.
4. **Assign meaningful names to spans**:
   Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context.
5. **Avoid dynamic, high cardinality data in span names**:
   Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

## Additional Information

For more use cases, see the [OpenTelemetry JavaScript API documentation](https://opentelemetry.io/docs/languages/js/instrumentation/#create-spans).


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/nodejs/native



> [!NOTE]
>
> Native Instrumentation is currently the default setting of Odigos **Open-Source** version.
>
> eBPF-Based Instrumentation is currently available with the Odigos Enterprise plan.
> Contact us for more information.

## Supported Versions

Odigos uses the official OpenTelemetry instrumentation libraries and SDK components, thus it supports the same Node.js versions as the OpenTelemetry Node.js.

* Node.js runtime versions **14** and above are supported.

## Traces

Odigos will automatically instrument your Node.js sources to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Node.js Official Instrumentation Libraries.

If you use one of the supported logger libraries, the trace id and span id will be auto-injected into your log records allowing signals correlations between logs and traces.

## Instrumentation Libraries

The following npm packages will be auto instrumented by Odigos:

### Node.js Core Modules:

* `node:dns` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:fs` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:http` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:https` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:net` [`Node.js`](https://nodejs.org/) versions `>=14`

### HTTP Frameworks:

* [`connect`](https://www.npmjs.com/package/connect) versions `>=3.0.0 <4`
* [`express`](https://www.npmjs.com/package/express) versions `>=4.0.0 <5`
* [`fastify`](https://www.npmjs.com/package/fastify) versions `>=3.0.0 <5`
* [`graphql`](https://www.npmjs.com/package/graphql) versions `>=14.0.0 <17`
* [`@hapi/hapi`](https://www.npmjs.com/package/@hapi/hapi) versions `>=17.0.0 <22`
* [`koa`](https://www.npmjs.com/package/koa) versions `>=2.0.0 <3`
* [`@koa/router`](https://www.npmjs.com/package/@koa/router) versions `>=8.0.0`
* [`@nestjs/core`](https://www.npmjs.com/package/@nestjs/core) versions `>=4.0.0 <11`
* `node:http` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:https` [`Node.js`](https://nodejs.org/) versions `>=14`
* [`restify`](https://www.npmjs.com/package/restify) versions `>=4.0.0 <12`
* [`router`](https://www.npmjs.com/package/router) versions `>=1.0.0 <2`

### HTTP Clients:

* `node:http` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:https` [`Node.js`](https://nodejs.org/) versions `>=14`
* [`undici`](https://www.npmjs.com/package/undici) versions `>=5.12.0`

### RPC (Remote Procedure Call):

* [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) versions `>=2.308.0 <3`
* [`@aws-sdk/client-*`](https://www.npmjs.com/package/@aws-sdk/client-*) versions `>=3.0.0 <4`
* [`@grpc/grpc-js`](https://www.npmjs.com/package/@grpc/grpc-js) versions `^1.0.0`

### Messaging Systems Clients:

* [`amqplib`](https://www.npmjs.com/package/amqplib) versions `>=0.5.5 <1`
* [`kafkajs`](https://www.npmjs.com/package/kafkajs) versions `>=0.3.0 <3`
* `node:http` [`Node.js`](https://nodejs.org/) versions `>=14`
* `node:https` [`Node.js`](https://nodejs.org/) versions `>=14`
* [`socket.io`](https://www.npmjs.com/package/socket.io) versions `>=2.0.0 <5`

### Database Clients, ORMs, and data access libraries:

* [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) versions `>=2.308.0 <3`
* [`@aws-sdk/client-*`](https://www.npmjs.com/package/@aws-sdk/client-*) versions `>=3.0.0 <4`
* [`cassandra-driver`](https://www.npmjs.com/package/cassandra-driver) versions `>=4.4.0 <5`
* [`dataloader`](https://www.npmjs.com/package/dataloader) versions `>=2.0.0 <3`
* [`generic-pool`](https://www.npmjs.com/package/generic-pool) versions `>=2.0.0 <4`
* [`ioredis`](https://www.npmjs.com/package/ioredis) versions `>=2.0.0 <6`
* [`knex`](https://www.npmjs.com/package/knex) versions `>=0.10.0 <4`
* [`lru-memoizer`](https://www.npmjs.com/package/lru-memoizer) versions `>=1.3.0 <3`
* [`memcached`](https://www.npmjs.com/package/memcached) versions `>=2.2.0 <3`
* [`mongodb`](https://www.npmjs.com/package/mongodb) versions `>=3.3.0 <7`
* [`mongoose`](https://www.npmjs.com/package/mongoose) versions `>=5.9.7 <7`
* [`mysql`](https://www.npmjs.com/package/mysql) versions `>=2.0.0 <3`
* [`mysql2`](https://www.npmjs.com/package/mysql2) versions `>=1.4.2 <4`
* [`pg-pool`](https://www.npmjs.com/package/pg-pool) versions `>=2.0.0 <4`
* [`pg`](https://www.npmjs.com/package/pg) versions `>=8.0.0 <9`
* [`redis`](https://www.npmjs.com/package/redis) versions `>=2.6.0`
* [`tedious`](https://www.npmjs.com/package/tedious) versions `>=1.11.0 <18`

### Loggers:

Automatic injection of trace context (trace id and span id) into log records for the following loggers:

* [`bunyan`](https://www.npmjs.com/package/bunyan) versions `>=1.0.0 <2`
* [`pino`](https://www.npmjs.com/package/pino) versions `>=5.14.0 <10`
* [`winston`](https://www.npmjs.com/package/winston) versions `>=1.0.0 <4`


# OpenTelemetry eBPF Instrumentation (OBI)
Source: https://docs.odigos.io/oss/instrumentations/obi

OBI is OpenTelemetry’s eBPF-based auto-instrumentation.

Odigos can run **[OpenTelemetry eBPF Instrumentation (OBI)](https://opentelemetry.io/docs/zero-code/obi)** for workloads where the `opentelemetry-ebpf-instrumentation` distro is selected or a Metrics `InstrumentationRule` is configured.

> **[OpenTelemetry OBI](https://opentelemetry.io/docs/zero-code/obi)**
>
> Upstream concepts, requirements, and capabilities.

> **[OBI on GitHub](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation)**
>
> Source, releases, and issues.

## Traces

OBI tracing can be enabled by selecting it as a container override for specific workloads. This can be configured either
from the Odigos UI or directly on a [Source object](/oss/pipeline/sources/introduction).

### Enabling OBI from the UI

**1. Open the source drawer and edit the detected container**

Select the source you want to instrument. In the source drawer, click the edit (pencil) icon next to the
detected container.

![Edit detected container in the source drawer](https://mintcdn.com/odigos/hTGQ8BCEhhNz2oGB/images/oss/instrumentations/obi/obi1.png?fit=max\&auto=format\&n=hTGQ8BCEhhNz2oGB\&q=85\&s=4a1edef1046f0ab1a579c4edb4f7d9ce)

**2. Choose the opentelemetry-ebpf-instrumentation distro**

In the **Otel Distro Name** drop-down, choose **`opentelemetry-ebpf-instrumentation`** and click the check
mark to save.

![Select opentelemetry-ebpf-instrumentation as the Otel distro](https://mintcdn.com/odigos/hTGQ8BCEhhNz2oGB/images/oss/instrumentations/obi/obi2.png?fit=max\&auto=format\&n=hTGQ8BCEhhNz2oGB\&q=85\&s=a3def545718649f08138f82e1c66e1c5)

**3. Verify the source is instrumented with OBI**

After refreshing, the detected container should report that it is instrumented with
`opentelemetry-ebpf-instrumentation`.

![Source instrumented with OBI](https://mintcdn.com/odigos/hTGQ8BCEhhNz2oGB/images/oss/instrumentations/obi/obi3.png?fit=max\&auto=format\&n=hTGQ8BCEhhNz2oGB\&q=85\&s=7976a4c854bf7810c1f856e7ff47d713)

**4. Remove OBI instrumentation**

To remove OBI instrumentation, open the runtime override config again and click the delete (trash) icon
next to the **Otel Distro Name** field.

![Remove OBI override from runtime config](https://mintcdn.com/odigos/hTGQ8BCEhhNz2oGB/images/oss/instrumentations/obi/obi4.png?fit=max\&auto=format\&n=hTGQ8BCEhhNz2oGB\&q=85\&s=eaff4da5ca010e1635b49a5125dcc99e)

> [!NOTE]
>
> By default, OBI does not need to restart your containers to begin instrumenting them. Because of this,
> choosing OBI as your container override will not trigger a pod restart. In some cases, however, your
> specific application may still require a pod restart (for example, when switching from a pre-existing
> instrumentation to OBI or vice-versa). You can trigger a pod restart using the **Restart** button in the
> source drawer.

### Enabling OBI from a Source object

If editing a [Source object](/oss/pipeline/sources/introduction) directly, OBI is enabled through the
`containerOverrides` field. For example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-workload
  namespace: default
spec:
  workload:
    kind: DaemonSet
    name: sample-app
    namespace: default
  containerOverrides:
    - containerName: foo
      otelDistroName: opentelemetry-ebpf-instrumentation
```

In this case, `containerName` is the name of the container to instrument in this workload, and the required
`otelDistroName` is **`opentelemetry-ebpf-instrumentation`**.

### Trace support

When OBI is the selected distro, odiglet attaches the process to OBI's trace selector when instrumentation loads and
removes it when instrumentation stops. OBI captures distributed traces for supported protocols (HTTP, gRPC, databases,
messaging, and more) without code changes. DNS lookup traces are enabled by default when using the OBI distro.

Learn more in the upstream docs:

> **[Distributed traces](https://opentelemetry.io/docs/zero-code/obi/distributed-traces/)**
>
> Context propagation, compatibility, and Kubernetes requirements.

> **[Supported protocols](https://opentelemetry.io/docs/zero-code/obi/configure/export-data/)**
>
> Protocol, database, messaging, and Go library instrumentation coverage.

Odigos configures OBI header-based context propagation (`context_propagation: headers`). For encrypted traffic and
multi-service propagation limits, see the upstream distributed tracing guide linked above.

### Limitations

OBI's instrumentation is based on HTTP header propagation. For encrypted traffic, context propagation will not work unless both
workloads (client and server) are instrumented with OBI as explained in [the official OBI docs](https://opentelemetry.io/docs/zero-code/obi/distributed-traces/#context-propagation-at-network-level).

## Metrics

Upstream OBI is split into several observability pipelines. In Odigos, each signal is wired independently so
traces and metrics can be enabled without requiring the same distro or lifecycle on every process.

| Signal | OBI pipeline | Enabled in Odigos when | Applies to |
| - | - | - | - |
| **Traces** | [AppO11y](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main/pkg/appolly) (application protocol tracing) | The `opentelemetry-ebpf-instrumentation` distro is selected for the container. DNS lookup traces are included by default for the OBI distro | Processes instrumented with the OBI distro only |
| **Network metrics** | [NetO11y](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main/pkg/netolly) (point-to-point flow bytes) and [StatsO11y](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/tree/main/pkg/statsolly) (TCP statistics such as RTT and failed connections) | An [InstrumentationRule](/oss/pipeline/rules/introduction) with `networkMetrics` set (optionally scoped via `scopes`) | Any process, regardless of language or distro (Go, Java, OBI, etc.) |

All OBI telemetry that Odigos collects is exported over OTLP to the odiglet node collector (`localhost:4317`), then
follows the same Odigos pipeline as other workload telemetry.

> [!NOTE]
>
> For consistency and discoverability alongside other Odigos telemetry, the node collector rewrites the
> `obi` prefix of OBI metric names to `odigos`. Metrics named `obi.*` are exported as `odigos.*` (and the
> Prometheus form `obi_*` as `odigos_*`). For example, `obi.network.flow.bytes` is exported as
> `odigos.network.flow.bytes`. Metric names referenced in the OBI documentation below use the `obi`
> prefix; account for this rename when building dashboards or queries.

OBI does not need to be selected as the traces distro for an app to use OBI metrics, see below.

### Configuration

**Network metrics infrastructure (cluster-wide):** OBI network flow collection requires odiglet `hostNetwork`. Enable the
infrastructure in Helm by setting `metricsSources.networkMetrics.enabled` to `true` (it defaults to disabled):

```yaml theme={null}
metricsSources:
  networkMetrics:
    enabled: true
```

**Enabling network metrics:** Create an `InstrumentationRule` in the Odigos installation namespace (typically
`odigos-system`) with a `networkMetrics` block. See [Network Metrics](/oss/pipeline/rules/networkmetrics) for configuration options,
examples, and verification steps.

Multiple rules merge with OR semantics (if any rule enables a signal, it is enabled). Workload scoping via `scopes` is
optional — omit it to apply cluster-wide.

### Network metrics

Network metrics describe bytes sent and received between endpoints (pods, services, nodes, and related Kubernetes
metadata), along with TCP-level statistics such as round-trip time and failed connection attempts. Enable cluster
infrastructure with `metricsSources.networkMetrics.enabled: true`, then enable collection with an
InstrumentationRule `networkMetrics` block (cluster-wide or scoped via `scopes`).

> [!WARNING]
>
> Network metrics are collected per process. In environments where a workload runs many processes,
> enabling network metrics can significantly increase the number of emitted metric series and the
> overall cardinality of your metrics. Monitor your metrics cardinality after enabling network metrics,
> and scope the rule via `scopes` to limit it to the workloads that need it.

> **[Network metrics overview](https://opentelemetry.io/docs/zero-code/obi/network/#network-metrics)**
>
> Flow and inter-zone metric families, attributes, and aggregation behavior.

> **[Exported metric names](https://opentelemetry.io/docs/zero-code/obi/metrics/)**
>
> OpenTelemetry and Prometheus names for `obi.network.flow.bytes` and related series.

### Stats metrics

Stats metrics (`obi.stat.tcp.rtt`, `obi.stat.tcp.failed.connections`, and related series) are collected together with network flow metrics when the Network Metrics rule is enabled.

> **[Exported stats metrics](https://opentelemetry.io/docs/zero-code/obi/metrics/)**
>
> Metric names and attributes for `obi.stat.tcp.rtt`, `obi.stat.tcp.failed.connections`, and related series.

> **[Metrics export features](https://opentelemetry.io/docs/zero-code/obi/configure/export-data/)**
>
> How OBI groups metrics into `network`, `stats`, `stats_tcp_rtt`, and related feature flags.

Network flow and TCP stats metrics export interval defaults to 60 seconds.


# Overview
Source: https://docs.odigos.io/oss/instrumentations/overview

Odigos will automatically apply instrumentation to your selected workloads to record observability signals from your services such as collecting traces, metrics, and logs without any code changes.

## Supported Runtimes

Odigos provides automatic instrumentations for the following runtimes:

> **[Go](../instrumentations/golang/ebpf)**
>
> Versions 1.19 - 1.26 are supported

> **[Javascript](../instrumentations/nodejs/native)**
>
> Versions 14 - 25 are supported

> **[Python](../instrumentations/python/native)**
>
> Versions 3.8 - 3.14 are supported

> **[Java](../instrumentations/java/native)**
>
> Versions 8 - 25 are supported

> **[PHP](../instrumentations/php/native)**
>
> Versions 8.1.0 - 8.4.x are supported

> **[Ruby](../instrumentations/ruby/native)**
>
> Versions 3.1.0 - 3.5.x are supported

> **[.NET](../instrumentations/dotnet/native)**
>
> Versions 4.6.2 - 10.0.3 are supported

## Enrich with OpenTelemetry APIs

Odigos automatically generates data for open-source libraries and frameworks.
Additional trace spans, metrics datapoints and log records can be added using the OpenTelemetry APIs.
Odigos will automatically capture this data and deliver it to the chosen [destination](../backends-overview) alongside the automatically generated data.

> [!NOTE]
>
> No need to configure the OpenTelemetry SDK, Odigos will automatically
> configure it for you.

Select a language to learn how to enrich your data with OpenTelemetry APIs:

> **[Go](../instrumentations/golang/enrichment)**
>
> For applications written in Go

> **[Javascript](../instrumentations/nodejs/enrichment)**
>
> For applications written in JavaScript or Node.js

> **[Python](../instrumentations/python/enrichment)**
>
> For applications written in Python

> **[Java](../instrumentations/java/enrichment)**
>
> For applications written in Java or JVM-based languages

> **[PHP](../instrumentations/php/enrichment)**
>
> For applications written in PHP

> **[Ruby](../instrumentations/ruby/enrichment)**
>
> For applications written in Ruby

> **[.NET](../instrumentations/dotnet/enrichment)**
>
> For applications written in C# or .NET

## Instrumentation Flow:

1. User selects a workload to auto instrument by creating a `Source` object (per workload or per namespace).
2. Odigos `instrumentor` component watches for changes with `Source` objects, and creates a relative `InstrumentationConfig` object.
3. Odigos `odiglet` component watches for changes with the `InstrumentationConfig` objects, and runs a runtime inspection on running pods to detect the programming language for every container.
4. Odigos `instrumentor` component watches for changes with the `InstrumentationConfig` object, and adds a resource request into each pod spec of all relevant containers in the workload manifest. These resource requests are called `Instrumentation Devices`.
5. Kubernetes detects the changes in the manifest, and rollout-restarts the pods with the new `Instrumentation Devices`.
6. New pods are scheduled and started. Odiglet resolves the resource request by mounting the auto instrumentation code and relevant environment variables into the container.
7. The auto instrumentation code starts the OpenTelemetry SDK and sends telemetry data to the odigos pipeline.

## Auto-rollback and Recover

If pods enter CrashLoopBackOff or ImagePullBackOff after instrumentation is injected, Odigos **disables the agent** and restarts the workload **without** instrumentation. The UI may show that Odigos detected a crash and rolled back instrumentation to protect the workload.

That rollback is **not** a Kubernetes Deployment undo. Odigos does not restore a previous ReplicaSet or Helm revision. It only removes Odigos agents from the workload.

**Recover** (in the UI) means **try instrumentation again**. It clears the rollback latch and rolls the workload **with** agents. Clicking Recover does not remove remaining instrumentation or roll back the Deployment a second time.

Disable automatic uninstrumentation with:

```bash theme={null}
odigos install --set autoRollback.disabled=true
```

or Helm `autoRollback.disabled=true`.


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/php/enrichment



Odigos will automatically instrument your PHP services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project by running:

```bash theme={null}
composer require open-telemetry/api
```

## Creating a Span

To create a new span, use the `TracerProvider`.

```php theme={null}
<?php

use OpenTelemetry\API\Globals;
use Psr\Http\Message\ResponseInterface as Response;
use Psr\Http\Message\ServerRequestInterface as Request;
use Slim\Factory\AppFactory;

require __DIR__ . '/vendor/autoload.php';

$tracer = Globals::tracerProvider()->getTracer('demo');

$app = AppFactory::create();
$app->get('/rolldice', function (Request $request, Response $response) use ($tracer) {
    $span = $tracer
        ->spanBuilder('manual-span')
        ->startSpan();

    $result = random_int(1,6);
    $response->getBody()->write(strval($result));

    $span
        ->addEvent('rolled dice', ['result' => $result])
        ->end();
    return $response;
});

$app->run();
```

## Additional Information

For more use cases, see the [OpenTelemetry PHP API documentation](https://opentelemetry.io/docs/languages/php).


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/php/native



> [!NOTE]
>
> This language instrumentation is using OpenTelemetry native-php SDK and instrumentation libraries.

## Supported Versions

The minimal supported version of PHP is **8.1.0**.
The maximal supported version of PHP is **8.4.x**, when PHP releases a new version we'll add the support for it.

> [!CAUTION]
>
> PHP version **8.0** has been deprecated and is no longer supported.

> [!WARNING]
>
> PHP version **8.1** has limited library support and will be the next version to be deprecated.

> [!TIP]
>
> PHP version **8.2+** is the recommended version for new projects.

## Instrumentation Libraries

The following PHP modules will be auto instrumented by Odigos:

### PHP Standards Recommendations (PSR):

* [`psr/3 (log)`](https://packagist.org/packages/psr/log) versions `>=1`
* [`psr/6 (cache)`](https://packagist.org/packages/psr/cache) versions `>=1`
* [`psr/16 (simple-cache)`](https://packagist.org/packages/psr/simple-cache) versions `>=1`
* [`psr/14 (event-dispatcher)`](https://packagist.org/packages/psr/event-dispatcher) versions `>=1`
* [`psr/15 (http-server-middleware)`](https://packagist.org/packages/psr/http-server-middleware) versions `>=1`
* [`psr/18 (http-client)`](https://packagist.org/packages/psr/http-client) versions `>=1`

### HTTP Clients:

* [`php-http/httplug`](https://packagist.org/packages/php-http/httplug) versions `>=2`
* [`guzzlehttp/guzzle`](https://packagist.org/packages/guzzlehttp/guzzle) versions `>=7`
* [`ext-curl`](https://www.php.net/manual/en/book.curl.php) versions `*`

> [!WARNING]
>
> `ext-curl` not supported for PHP 8.1

### Frameworks:

> [!NOTE]
>
> We're working on adding support for `laravel`, `cakephp`, `wordpress`, `yii2` and `codeigniter` frameworks.

* [`slim/slim`](https://packagist.org/packages/slim/slim) versions `>=4`
* [`symfony/http-kernel`](https://packagist.org/packages/symfony/http-kernel) versions `*`
* [`symfony/http-client-contracts`](https://packagist.org/packages/symfony/http-client-contracts) versions `*`

### Databases:

* [`ext-mongodb`](https://www.php.net/manual/en/book.mongodb.php) versions `>=1.13`
* [`ext-mysqli`](https://www.php.net/manual/en/book.mysqli.php) versions `*`

> [!WARNING]
>
> `ext-mysqli` not supported for PHP 8.1

* [`ext-pdo`](https://www.php.net/manual/en/book.pdo.php) versions `*`

> [!WARNING]
>
> `ext-pdo` not supported for PHP 8.1

* [`doctrine/dbal`](https://packagist.org/packages/doctrine/dbal) versions `>=3`

> [!WARNING]
>
> `doctrine/dbal` not supported for PHP 8.1

### Messaging Systems:

* [`ext-amqp`](https://pecl.php.net/package/amqp) versions `*`

> [!WARNING]
>
> `ext-amqp` not supported for PHP 8.1

* [`ext-rdkafka`](https://pecl.php.net/package/rdkafka) versions `*`

> [!WARNING]
>
> `ext-rdkafka` not supported for PHP 8.1

### Atificial Intelligence:

* [`openai-php/client`](https://packagist.org/packages/openai-php/client) versions `*`


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/python/enrichment



Odigos will automatically instrument your Python services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Install the API from PyPI using pip:

```bash theme={null}
pip install opentelemetry-api
```

## Creating Spans

To create a span, use the `tracer` object from the `opentelemetry.trace` module. The `tracer` object is a factory for creating spans.

```python theme={null}
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def my_function():
   with tracer.start_as_current_span("my_function") as span:
      print("Hello world!")
```

Important Notes:

1. **Assign meaningful names to spans**:
   Use descriptive names for spans, (such as the function name) to clearly describe the operations they represent. This helps anyone examining the trace to easily understand the span's purpose and context.
2. **Avoid dynamic, high cardinality data in span names**:
   Do not include dynamic data such as IDs in the span name, as this can cause performance issues and make the trace harder to read. Instead, use static, descriptive names for spans and record dynamic information in span attributes. This ensures better performance and readability of the trace.

### Recording Span Attributes

Span attributes are key-value pairs that record additional information about an operation, which can be useful for debugging, performance analysis, or troubleshooting

Examples:

* User ID, organization ID, Account ID or other identifiers.
* Inputs - the relevant parameters or configuration that influenced the operation.
* Outputs - the result of the operation.
* Entities - the entities or resources that the operation interacted with.

Attribute names are lowercased strings in the form `my_application.my_attribute`, example: `my_service.user_id`.
Read more [here](https://opentelemetry.io/docs/specs/semconv/general/attribute-naming/)

To record attributes, use the `set_attribute` method on the span object.

```python theme={null}
def my_function(arg: str):
   with tracer.start_as_current_span("my_function") as span:
      span.set_attribute("argument_name", arg)
```

Important Notes:

1. **Be cautious when recording data**:
   Avoid including PII (personally identifiable information) or any data you do not wish to expose in your traces.
2. **Attribute cost considerations**:
   Each attribute affects performance and processing time. Record only what is necessary and avoid superfluous data.
3. **Use static names for attributes**:
   Avoid dynamic content such as IDs in attribute keys. Use static names and properly namespace them (scope.attribute\_name) to provide clear context for downstream consumers.
4. **Adhere to OpenTelemetry semantic conventions**:
   Prefer using namespaces and attribute names from the OpenTelemetry semantic conventions to enhance data interoperability and make it easier for others to understand.

### Recording Errors

To easily identify and monitor errors in your traces, the Span object includes a status field that can be used to mark the span as an error. This helps in spotting errors in trace viewers, sampling, and setting up alerts.

If an operation you are tracing fails, you can mark the span's status as an error and record the error details within the span. Here's how you can do it:

* An exception has been raised to demonstrate an error that occurred in your code.

```python theme={null}
def my_function():
   with tracer.start_as_current_span("my_function") as span:
      try:
         print("Hello world!")
         raise Exception("Some Exception")
      except Exception as e:
         span.record_exception(e)
         span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
```

## Advanced Capabilities

The Odigos Python agent extends the standard OpenTelemetry SDK with capabilities that apply to **all spans** — including custom spans you create with the API above. These features are configured remotely through OpAMP and take effect without restarting your application.

### Head Sampling

The Python agent supports [head sampling](../../pipeline/sampling/head-and-tail-sampling#head-sampling-agent) at the root span. Odigos pushes [Noisy Operation](../../pipeline/sampling/rules/noisy-operation) rules to the agent, which evaluates them through a custom `OdigosSampler` wired into the `TracerProvider`.

Custom spans participate in the same sampling pipeline as auto-instrumented spans:

* **Root spans** — If your custom span is the entry point of a trace (no active parent), head-sampling rules can match it and decide whether the entire trace is kept or dropped.
* **Child spans** — Follow parent-based sampling: if the parent trace is sampled, custom child spans are recorded; if not, they are dropped (or recorded for metrics only — see [Span metrics mode](#span-metrics-mode) below).
* **Context propagation** — Use `tracer.start_as_current_span(...)` (as shown above) so custom spans attach to the active trace and inherit the sampling decision.

Head-sampling rules match HTTP server spans (`SpanKind.SERVER`) and HTTP client spans (`SpanKind.CLIENT`) using OpenTelemetry semantic conventions. The sampler accepts both stable and legacy attribute names — for example `http.request.method` and `http.method`, `server.address` and `http.host`. Server rules match on route (with `{param}` wildcard support), route prefix, and method. Client rules match on server address, templated path, path prefix, and method.

When multiple rules match, the **lowest keep percentage** wins. Even when a trace is dropped, the agent attaches an `odigos` [tracestate](https://www.w3.org/TR/trace-context/#tracestate-header) entry so downstream services and the Odigos tail sampler can coordinate on the same decision.

For rule configuration and evaluation order, see [Sampling rules overview](../../pipeline/sampling/rules/overview).

### Dry Run Mode

When cluster-wide dry run (`samplingDryRun` in [OdigosConfiguration](../../api-reference/odigos.io.v1alpha1)) is enabled, the Python agent still exports every span but annotates the sampling decision in tracestate (`dry:t` if the trace would have been kept, `dry:f` if it would have been dropped). Use dry run to validate Noisy rules before committing to real drops.

### Span Metrics Mode

The global `spanMetricsMode` setting controls what happens to spans that head sampling marks as not sampled:

| Mode | Not-sampled spans |
| - | - |
| `sampled-spans-only` (default) | Dropped entirely — no trace export, no span metrics |
| `all-spans` | Recorded with `RECORD_ONLY` — excluded from trace export but forwarded for span metrics computation |

This applies to custom spans the same way it applies to auto-instrumented spans. If you rely on span metrics for RED dashboards, set `spanMetricsMode: all-spans` in your Odigos configuration so unsampled traffic still contributes to rate and error metrics.

### Agent Diagnostics

Debug logging for the Odigos Python agent and its OpenTelemetry components can be toggled at runtime through an [InstrumentationRule](../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-InstrumentationRule) with `agentDiagnostics`:

```yaml theme={null}
spec:
  agentDiagnostics:
    odigosLogLevel: debug
    openTelemetryComponentsLogLevel: debug
```

Supported levels are `error`, `warn`, `info`, and `debug`. Remote config from OpAMP **fully overrides** the startup environment variables — if a level is not set in the remote config, that logger is silenced.

For the window before the first OpAMP message arrives, you can set startup log levels with environment variables:

| Variable | Logger | Values |
| - | - | - |
| `ODIGOS_LOG_LEVEL` | Odigos agent (`odigos`) | `ALL`, `VERBOSE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `NONE` |
| `OTEL_LOG_LEVEL` | OpenTelemetry SDK components | Same as above |

At `debug` level, the sampler logs each rule evaluation — useful when troubleshooting why a custom or auto-instrumented span was kept or dropped.

## Additional Information

For more use cases, see the [OpenTelemetry Python API documentation](https://opentelemetry.io/docs/languages/python/instrumentation/#creating-spans).


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/python/native



> [!NOTE]
>
> Native Instrumentation is currently the default setting of Odigos **Open-Source** version.
>
> eBPF-Based Instrumentation is currently not supported for this language.

## Supported Versions

Odigos uses the official [opentelemetry-python-instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation) OpenTelemetry Auto Instrumentation, thus it supports the same Python versions as this project.

* In the enterprise version, Odigos leverages eBPF to enhance performance in the Python instrumentation process.

* Odigos ships **two separate Python distros** — one for Python 3.8 and one for Python 3.9 and above. Odigos selects the correct distro automatically based on the runtime version of your workload.

* **Python 3.8** uses a dedicated legacy distro with OpenTelemetry instrumentation `0.54b1`.

* **Python 3.9 through 3.14** uses the main distro with the latest OpenTelemetry instrumentation.

## Traces

Odigos will automatically instrument your Python services to record and collect spans for distributed tracing, by utilizing the OpenTelemetry Python official auto Instrumentation Libraries.

## Instrumentation Libraries

The following Python modules will be auto instrumented by Odigos. Supported libraries differ between the Python 3.8 and Python 3.9+ distros:

**Python 3.8**

The Python 3.8 distro includes the libraries below. It does not include Gen AI instrumentations other than OpenAI.

### Database Clients, ORMs, and Data Access Libraries

* [`aiopg`](https://pypi.org/project/aiopg/) versions `aiopg >= 0.13.0, < 2.0.0`
* [`dbapi`](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/dbapi/dbapi.html)
* [`mysql`](https://pypi.org/project/mysql-connector-python/) version `mysql-connector-python >= 8.0, < 10.0`
* [`mysqlclient`](https://pypi.org/project/mysqlclient/) version `mysqlclient < 3`
* [`psycopg`](https://pypi.org/project/psycopg/) versions `psycopg >= 3.1.0`
* [`psycopg2`](https://pypi.org/project/psycopg2/) versions `psycopg2 >= 2.7.3.1`, `psycopg2-binary >= 2.7.3.1`
* [`pymemcache`](https://pypi.org/project/pymemcache/) versions `pymemcache >= 1.3.5, < 5`
* [`pymssql`](https://pypi.org/project/pymssql/) versions `pymssql >= 2.1.5, < 3`
* [`pymongo`](https://pypi.org/project/pymongo/) versions `pymongo >= 3.1, < 5.0`
* [`pymysql`](https://pypi.org/project/PyMySQL/) versions `PyMySQL < 2`
* [`redis`](https://pypi.org/project/redis/) versions `redis >= 2.6`
* [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) versions `sqlalchemy >= 1.0.0, < 2.1.0`
* [`sqlite3`](https://docs.python.org/3/library/sqlite3.html)
* [`tortoiseorm`](https://pypi.org/project/tortoise-orm/) versions `tortoise-orm >= 0.17.0`
* [`cassandra`](https://pypi.org/project/cassandra-driver/) versions `cassandra-driver ~= 3.25`, `scylla-driver ~= 3.25`
* [`elasticsearch`](https://pypi.org/project/elasticsearch/) versions `elasticsearch >= 6.0`
* [`asyncpg`](https://pypi.org/project/asyncpg/) versions `asyncpg >= 0.12.0`

### HTTP Frameworks

* [`asgi`](https://pypi.org/project/asgiref/) versions `asgiref ~= 3.0`
* [`django`](https://pypi.org/project/Django/) versions `django >= 1.10`
  > [!NOTE]
  >
  > Please note that for Django instrumentation to work, you need to set the
  > [DJANGO\_SETTINGS\_MODULE](https://docs.djangoproject.com/en/5.1/topics/settings/#envvar-DJANGO_SETTINGS_MODULE) environment variable. Make sure to
  > set this in your Kubernetes manifest or Dockerfile to ensure proper configuration.
* [`fastapi`](https://pypi.org/project/fastapi/) versions `fastapi ~= 0.58`
* [`flask`](https://pypi.org/project/Flask/) versions `flask >= 1.0`
* [`pyramid`](https://pypi.org/project/pyramid/) versions `pyramid >= 1.7`
* [`starlette`](https://pypi.org/project/starlette/) versions `starlette >= 0.13, < 0.15`
* [`falcon`](https://pypi.org/project/falcon/) versions `falcon >= 1.4.1, < 5.0.0`
* [`tornado`](https://pypi.org/project/tornado/) versions `tornado >= 5.1.1`

### HTTP Clients

* [`aiohttp-client`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`httpx`](https://pypi.org/project/httpx/) versions `httpx >= 0.18.0`
* [`requests`](https://pypi.org/project/requests/) versions `requests ~= 2.0`
* [`urllib`](https://docs.python.org/3/library/urllib.html)
* [`urllib3`](https://pypi.org/project/urllib3/) versions `urllib3 >= 1.0.0, < 3.0.0`

### Messaging Systems Clients

* [`aio-pika`](https://pypi.org/project/aio-pika/) versions `aio_pika >= 7.2.0, < 10.0.0`
* [`celery`](https://pypi.org/project/celery/) versions `celery >= 4.0, < 6.0`
* [`confluent-kafka`](https://pypi.org/project/confluent-kafka/) versions `confluent-kafka >= 1.8.2, <= 2.7.0`
* [`kafka-python`](https://pypi.org/project/kafka-python/) versions `kafka-python >= 2.0, < 3.0`, `kafka-python-ng >= 2.0, < 3.0`
* [`pika`](https://pypi.org/project/pika/) versions `pika >= 0.12.0`
* [`remoulade`](https://pypi.org/project/remoulade/) versions `remoulade >= 0.50`

### RPC (Remote Procedure Call)

* [`grpc`](https://pypi.org/project/grpcio/) versions `grpcio >= 1.42.0`

### Web Servers

* [`aiohttp-server`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`wsgi`](https://docs.python.org/3/library/wsgiref.html)

### Cloud Services and SDKs

* [`boto`](https://pypi.org/project/boto/) versions `boto ~= 2.0`
* [`boto3sqs`](https://pypi.org/project/boto3/) versions `boto3 ~= 1.0`
* [`botocore`](https://pypi.org/project/botocore/) versions `botocore ~= 1.0`

### Framework and Library Utilities

* [`jinja2`](https://pypi.org/project/Jinja2/) versions `jinja2 >= 2.7, < 4.0`

### Other

* [`asyncio`](https://pypi.org/project/asyncio/)
* [`click`](https://pypi.org/project/click/) versions `click >= 8.1.3, < 9.0.0`
* [`threading`](https://docs.python.org/3/library/threading.html)

### Gen AI

* [`openai`](https://pypi.org/project/openai/) versions `openai >= 1.26.0`

### Loggers

Automatic injection of trace context (trace id and span id) into log records for the following loggers:

* [`logging`](https://docs.python.org/3/library/logging.html)

**Python 3.9+**

The Python 3.9+ distro includes the libraries below, including additional Gen AI instrumentations not available on Python 3.8.

### Database Clients, ORMs, and Data Access Libraries

* [`aiopg`](https://pypi.org/project/aiopg/) versions `aiopg >= 0.13.0, < 2.0.0`
* [`dbapi`](https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/dbapi/dbapi.html)
* [`mysql`](https://pypi.org/project/mysql-connector-python/) version `mysql-connector-python >= 8.0, < 10.0`
* [`mysqlclient`](https://pypi.org/project/mysqlclient/) version `mysqlclient < 3`
* [`psycopg`](https://pypi.org/project/psycopg/) versions `psycopg >= 3.1.0`
* [`psycopg2`](https://pypi.org/project/psycopg2/) versions `psycopg2 >= 2.7.3.1`, `psycopg2-binary >= 2.7.3.1`
* [`pymemcache`](https://pypi.org/project/pymemcache/) versions `pymemcache >= 1.3.5, < 5`
* [`pymssql`](https://pypi.org/project/pymssql/) versions `pymssql >= 2.1.5, < 3`
* [`pymongo`](https://pypi.org/project/pymongo/) versions `pymongo >= 3.1, < 5.0`
* [`pymysql`](https://pypi.org/project/PyMySQL/) versions `PyMySQL < 2`
* [`redis`](https://pypi.org/project/redis/) versions `redis >= 2.6`
* [`sqlalchemy`](https://pypi.org/project/SQLAlchemy/) versions `sqlalchemy >= 1.0.0, < 2.1.0`
* [`sqlite3`](https://docs.python.org/3/library/sqlite3.html)
* [`tortoiseorm`](https://pypi.org/project/tortoise-orm/) versions `tortoise-orm >= 0.17.0`
* [`cassandra`](https://pypi.org/project/cassandra-driver/) versions `cassandra-driver ~= 3.25`, `scylla-driver ~= 3.25`
* [`elasticsearch`](https://pypi.org/project/elasticsearch/) versions `elasticsearch >= 6.0`
* [`asyncpg`](https://pypi.org/project/asyncpg/) versions `asyncpg >= 0.12.0`

### HTTP Frameworks

* [`asgi`](https://pypi.org/project/asgiref/) versions `asgiref ~= 3.0`
* [`django`](https://pypi.org/project/Django/) versions `django >= 2.0`
  > [!NOTE]
  >
  > Please note that for Django instrumentation to work, you need to set the
  > [DJANGO\_SETTINGS\_MODULE](https://docs.djangoproject.com/en/5.1/topics/settings/#envvar-DJANGO_SETTINGS_MODULE) environment variable. Make sure to
  > set this in your Kubernetes manifest or Dockerfile to ensure proper configuration.
* [`fastapi`](https://pypi.org/project/fastapi/) versions `fastapi ~= 0.92`
* [`flask`](https://pypi.org/project/Flask/) versions `flask >= 1.0`
* [`pyramid`](https://pypi.org/project/pyramid/) versions `pyramid >= 1.7`
* [`starlette`](https://pypi.org/project/starlette/) versions `starlette >= 0.13`
* [`falcon`](https://pypi.org/project/falcon/) versions `falcon >= 1.4.1, < 5.0.0`
* [`tornado`](https://pypi.org/project/tornado/) versions `tornado >= 5.1.1`

### HTTP Clients

* [`aiohttp-client`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`httpx`](https://pypi.org/project/httpx/) versions `httpx >= 0.18.0`
* [`requests`](https://pypi.org/project/requests/) versions `requests ~= 2.0`
* [`urllib`](https://docs.python.org/3/library/urllib.html)
* [`urllib3`](https://pypi.org/project/urllib3/) versions `urllib3 >= 1.0.0, < 3.0.0`

### Messaging Systems Clients

* [`aio-pika`](https://pypi.org/project/aio-pika/) versions `aio_pika >= 7.2.0, < 10.0.0`
* [`celery`](https://pypi.org/project/celery/) versions `celery >= 4.0, < 6.0`
* [`confluent-kafka`](https://pypi.org/project/confluent-kafka/) versions `confluent-kafka >= 1.8.2, < 3.0.0`
* [`kafka-python`](https://pypi.org/project/kafka-python/) versions `kafka-python >= 2.0, < 3.0`, `kafka-python-ng >= 2.0, < 3.0`
* [`pika`](https://pypi.org/project/pika/) versions `pika >= 0.12.0`
* [`remoulade`](https://pypi.org/project/remoulade/) versions `remoulade >= 0.50`

### RPC (Remote Procedure Call)

* [`grpc`](https://pypi.org/project/grpcio/) versions `grpcio >= 1.42.0`

### Web Servers

* [`aiohttp-server`](https://pypi.org/project/aiohttp/) versions `aiohttp ~= 3.0`
* [`wsgi`](https://docs.python.org/3/library/wsgiref.html)

### Cloud Services and SDKs

* [`boto3sqs`](https://pypi.org/project/boto3/) versions `boto3 ~= 1.0`
* [`botocore`](https://pypi.org/project/botocore/) versions `botocore ~= 1.0`, `aiobotocore ~= 2.0`

### Framework and Library Utilities

* [`jinja2`](https://pypi.org/project/Jinja2/) versions `jinja2 >= 2.7, < 4.0`

### Other

* [`asyncio`](https://pypi.org/project/asyncio/)
* [`click`](https://pypi.org/project/click/) versions `click >= 8.1.3, < 9.0.0`
* [`threading`](https://docs.python.org/3/library/threading.html)

### Gen AI

* [`anthropic`](https://pypi.org/project/anthropic/) versions `anthropic >= 0.3.11`
* [`google-genai`](https://pypi.org/project/google-genai/) versions `google-genai >= 1.32.0`
* [`langchain`](https://pypi.org/project/langchain-core/) versions `langchain-core > 0.1.0`
* [`openai-agents`](https://pypi.org/project/openai-agents/) versions `openai-agents >= 0.3.3`
* [`openai`](https://pypi.org/project/openai/) versions `openai >= 1.26.0`
* [`vertexai`](https://pypi.org/project/google-cloud-aiplatform/) versions `google-cloud-aiplatform >= 1.64`
* [`weaviate`](https://pypi.org/project/weaviate-client/) versions `weaviate-client >= 3.26.0, < 5`

### Loggers

Automatic injection of trace context (trace id and span id) into log records for the following loggers:

* [`logging`](https://docs.python.org/3/library/logging.html)


# Enrich with OpenTelemetry API
Source: https://docs.odigos.io/oss/instrumentations/ruby/enrichment



Odigos will automatically instrument your Ruby services and record semantic spans from popular modules. Many users find the automatic instrumentation data sufficient for their needs.

However, if there is anything specific to your application that you want to record, you can enrich the data by adding custom spans to your code.

This is sometimes referred to as `manual instrumentation`.

## Use Cases

Examples of custom spans you might want to add to your code include:

* Spans for the execution of some potentially heavy or slow computation in you service.
* Tracing for internal or third party libraries for which you don't have automatic or integrated instrumentation.
* Spans to describe some logical operations in your business logic that are meaningful in your domain.

## Required Dependencies

Add the following dependencies to your project by running:

```bash theme={null}
gem install opentelemetry-sdk
```

## Creating a Span

To acquire a tracer:

```ruby theme={null}
# If in a Rails app, this lives in config/initializers/opentelemetry.rb
require "opentelemetry/sdk"

OpenTelemetry::SDK.configure do |c|
  c.service_name = '<YOUR_SERVICE_NAME>'
end

# 'MyAppTracer' can be used throughout your code now
MyAppTracer = OpenTelemetry.tracer_provider.tracer('<YOUR_TRACER_NAME>')
```

To create a new span:

```ruby theme={null}
require "opentelemetry/sdk"

def do_work
  MyAppTracer.in_span("do_work") do |span|
    # do some work that the 'do_work' span tracks!
  end
end
```

## Additional Information

For more use cases, see the [OpenTelemetry Ruby API documentation](https://opentelemetry.io/docs/languages/ruby).


# Automatic Instrumentation
Source: https://docs.odigos.io/oss/instrumentations/ruby/native



> [!NOTE]
>
> This language instrumentation is using OpenTelemetry native-ruby SDK and
> instrumentation libraries.

## Supported Versions

The minimal supported version of Ruby is **3.1.0**.
The maximal supported version of Ruby is **3.5.x**, when Ruby releases a new version we'll add the support for it.

## Instrumentation Libraries

The following Ruby gems will be auto instrumented by Odigos:

### HTTP Clients:

* [`http`](https://rubygems.org/gems/http) versions `*`
* [`httpclient`](https://rubygems.org/gems/httpclient) versions `*`
* [`restclient`](https://rubygems.org/gems/restclient) versions `*`
* [`net-http`](https://rubygems.org/gems/net-http) versions `*`
* [`faraday`](https://rubygems.org/gems/faraday) versions `*`
* [`excon`](https://rubygems.org/gems/excon) versions `*`
* [`ethon`](https://rubygems.org/gems/ethon) versions `*`

### gRPC Clients:

* [`grpc`](https://rubygems.org/gems/grpc) versions `*`

### Web Frameworks & APIs:

* [`rails`](https://rubygems.org/gems/rails) versions `*`
* [`rack`](https://rubygems.org/gems/rack) versions `*`
* [`sinatra`](https://rubygems.org/gems/sinatra) versions `*`
* [`grape`](https://rubygems.org/gems/grape) versions `*`
* [`gruf` (gRPC framework for Ruby)](https://rubygems.org/gems/gruf) versions `*`
* [`racecar` (Kafka consumer framework for Rails)](https://rubygems.org/gems/racecar) versions `*`

### Processing & Task Runners:

* [`rake`](https://rubygems.org/gems/rake) versions `>=13.0`
* [`concurrent-ruby` (Threading and scheduling utilities)](https://rubygems.org/gems/concurrent-ruby) versions `*`

### Query & Schema APIs:

* [`graphql`](https://rubygems.org/gems/graphql) versions `*`
* [`koala` (Facebook Graph API client)](https://rubygems.org/gems/koala) versions `*`

### Messaging & Background Jobs:

* [`sidekiq`](https://rubygems.org/gems/sidekiq) versions `*`
* [`resque`](https://rubygems.org/gems/resque) versions `*`
* [`delayed_job`](https://rubygems.org/gems/delayed_job) versions `*`
* [`que`](https://rubygems.org/gems/que) versions `*`
* [`bunny` (RabbitMQ)](https://rubygems.org/gems/bunny) versions `*`
* [`ruby-kafka`](https://rubygems.org/gems/ruby-kafka) versions `*`
* [`rdkafka`](https://rubygems.org/gems/rdkafka) versions `*`

### Databases & Caches:

* [`pg` (PostgreSQL)](https://rubygems.org/gems/pg) versions `*`
* [`mysql2`](https://rubygems.org/gems/mysql2) versions `*`
* [`trilogy` (MySQL client by GitHub)](https://rubygems.org/gems/trilogy) versions `*`
* [`mongo`](https://rubygems.org/gems/mongo) versions `*`
* [`redis`](https://rubygems.org/gems/redis) versions `*`
* [`dalli` (Memcached)](https://rubygems.org/gems/dalli) versions `*`
* [`lmdb` (Lightning Memory-Mapped Database)](https://rubygems.org/gems/lmdb) versions `*`

### Cloud SDKs:

* [`aws-sdk`](https://rubygems.org/gems/aws-sdk) versions `*`
* [`aws-sdk-lambda`](https://rubygems.org/gems/aws-sdk-lambda) versions `*`

### Serialization:

* [`active_model_serializers`](https://rubygems.org/gems/active_model_serializers) versions `>=0.10.0`


# Overview
Source: https://docs.odigos.io/oss/overview



Odigos is an [open-source](https://github.com/odigos-io/odigos) observability control plane. Odigos leverages the power of [OpenTelemetry](https://opentelemetry.io) and [eBPF](https://ebpf.io) to automatically instrument applications.

Odigos automatically deploys and scales collectors according to the traffic of applications. No need to waste time deploying and configuring collectors.

> [!TIP]
>
> The name Odigos originates from Greek, meaning "guide".

## Goals

* **Automatic instrumentation** - Odigos automatically instruments your applications and produces distributed traces and metrics **without any code changes**.
* **Collector management** - Odigos automatically deploys and scales collectors according to application traffic. No need to spend time deploying and configuring collectors.

Odigos achieves its goals by deploying a set of components that work together to collect, process, and export telemetry data.

## Who is Odigos for?

* Software Developers
* Platform Engineers
* IT Professionals
* Anyone comfortable with command-line interfaces and basic cloud infrastructure

> [!NOTE]
>
> Odigos is designed to be easy to use and deploy. It is a great tool for anyone who wants to get started with observability without the hassle of setting up and managing collectors.

## Adopt OpenTelemetry in minutes

Odigos currently supports all the popular managed and open source [destinations](./backends-overview). By producing data in the [OpenTelemetry](https://opentelemetry.io) format, Odigos can be used with any observability tool that supports OTLP and allow you to easily switch to another vendor in the future.

> **[Getting Started](./quickstart/introduction)**
>
> Use Odigos to generate distributed traces, metrics and logs with OpenTelemetry.

> **[Telemetry Types](./telemetry-types)**
>
> Telemetry types that Odigos supports and produce.

> **[Supported Languages](./instrumentations/overview)**
>
> Programming languages that Odigos supports and applies auto-instrumentation to.

> **[Supported Destinations](./adding-new-dest)**
>
> Destinations that Odigos supports and sends telemetry data to.


# Kubernetes RBAC Permissions
Source: https://docs.odigos.io/oss/permissions



This page lists the Kubernetes Roles and ClusterRoles used by Odigos and the Odigos Operator.

# Components

This section lists the RBAC policies used by the Odigos components.

## ClusterRoles

Below are the ClusterRoles used by Odigos components.

### cleanup-clusterrole

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| odigos.io | sources | \* | list delete |
| \* | pods | \* | list |
| \* | nodes | \* | list patch |

### odigos-gateway

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| odigos.io | instrumentationconfigs | \* | get list watch |
| \* | pods namespaces | \* | get list watch |
| apps | replicasets deployments statefulsets daemonsets | \* | get list watch |

### odigos-go-offsets-updater

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | list |

### odigos-instrumentor

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | nodes | \* | list watch get |
| \* | namespaces | \* | list watch get |
| \* | pods | \* | list watch get |
| batch | cronjobs | \* | list watch get |
| apps | daemonsets | \* | get list watch update patch |
| apps | deployments | \* | get list watch update patch |
| apps | statefulsets | \* | get list watch update patch |
| \* | pods/finalizers configmaps/finalizers | \* | update |
| apps | statefulsets/finalizers deployments/finalizers daemonsets/finalizers | \* | update |
| batch | cronjobs/finalizers jobs/finalizers | \* | update |
| argoproj.io | rollouts/finalizers | \* | update |
| apps.openshift.io | deploymentconfigs deploymentconfigs/finalizers | \* | get list watch update patch |
| argoproj.io | rollouts | \* | get list watch patch |
| operator.odigos.io | odigos/finalizers | \* | update |
| odigos.io | instrumentationconfigs/status | \* | get patch update |
| odigos.io | instrumentationconfigs | \* | create delete get list patch update watch |
| odigos.io | sources | \* | create delete get list patch update watch |
| odigos.io | sources/finalizers | \* | update |
| admissionregistration.k8s.io | mutatingwebhookconfigurations | \* | get list watch |
| admissionregistration.k8s.io | mutatingwebhookconfigurations | odigos-source-mutating-webhook-configuration odigos-pod-mutating-webhook-configuration | update |
| admissionregistration.k8s.io | validatingwebhookconfigurations | \* | get list watch |
| admissionregistration.k8s.io | validatingwebhookconfigurations | odigos-source-validating-webhook-configuration odigos-action-validating-webhook-configuration | update |
| apiregistration.k8s.io | apiservices | \* | get list watch update patch delete |

### odiglet

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | pods services | \* | get list watch |
| \* | pods/status | \* | get |
| \* | pods/finalizers | \* | update |
| \* | nodes | \* | get list watch patch update |
| odigos.io | instrumentationinstances | \* | create get list patch update watch delete |
| odigos.io | instrumentationinstances/status | \* | get patch update |
| odigos.io | instrumentationconfigs | \* | get list watch patch update |
| odigos.io | instrumentationconfigs/status | \* | get patch update |
| \* | nodes/stats | \* | get list |
| \* | pods namespaces | \* | get list watch |
| apps | replicasets deployments daemonsets statefulsets | \* | get list watch |
| discovery.k8s.io | endpointslices | \* | get list watch |

### odigos-ui

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | namespaces | \* | get list watch patch |
| apps | deployments statefulsets daemonsets | \* | get list watch update patch |
| batch | cronjobs | \* | get list watch update patch |
| apps.openshift.io | deploymentconfigs | \* | get list watch update patch |
| argoproj.io | rollouts | \* | get list watch update patch |
| apps | replicasets | \* | get list |
| \* | services | \* | get list |
| \* | pods | \* | get list watch delete |
| odigos.io | \* | \* | get list watch |
| odigos.io | instrumentationconfigs instrumentationinstances sources samplings | \* | update patch create delete |
| operator.odigos.io | odigos | \* | get list watch |
| actions.odigos.io | \* | \* | get list watch |

## Roles

Below are the Roles used by Odigos components. These Roles are only scoped to the Namespace in which Odigos is installed.

### cleanup-role

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | odigos-deployment odigos-configuration | get |
| \* | configmaps | \* | list |

### odigos-gateway

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | odigos-gateway | get list watch |

### odigos-go-offsets-updater

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | odigos-go-offsets | get update |
| \* | secrets | odigos-pro | get |

### odigos-instrumentor

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | get list watch create patch update delete |
| \* | services | \* | get list watch create patch update delete |
| \* | pods | \* | get list watch |
| apps | deployments | \* | create delete get list patch update watch |
| apps | deployments/status | \* | get |
| autoscaling | horizontalpodautoscalers | \* | create patch update delete |
| odigos.io | collectorsgroups | \* | get list create patch update watch delete |
| odigos.io | collectorsgroups/status | \* | get list watch patch update |
| odigos.io | destinations | \* | get list watch |
| odigos.io | destinations/status | \* | get patch update |
| odigos.io | instrumentationrules processors | \* | get list watch patch update delete create |
| odigos.io | processors/status | \* | get patch update |
| odigos.io | instrumentationrules/status | \* | get patch update |
| odigos.io | actions | \* | get list watch patch update delete create |
| odigos.io | samplings | \* | get list watch |
| odigos.io | actions/status | \* | get patch update |
| odigos.io | recommendations | \* | get list watch create patch update delete |
| odigos.io | recommendations/status | \* | get patch update |
| actions.odigos.io | \* | \* | get list watch update |
| actions.odigos.io | \*/status | \* | get patch update |
| odigos.io | samplings/status | \* | get patch update |
| \* | secrets | \* | get list watch |
| \* | secrets | instrumentor-webhooks-cert | update |
| \* | secrets | autoscaler-webhook-cert autoscaler-webhooks-cert | delete |
| apps | daemonsets | odiglet | get list watch |
| apps | deployments/finalizers | \* | update |
| odigos.io | collectorsgroups/finalizers | \* | get patch update |
| batch | cronjobs | odigos-go-offsets-updater | get list watch create update patch delete |
| batch | jobs | \* | create |

### odigos-leader-election-role

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | events | \* | create patch |
| coordination.k8s.io | leases | \* | get list watch create update patch delete |

### odiglet

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | list watch |
| \* | configmaps | odigos-data-collection effective-config | get list watch |

### odigos-ui

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps | \* | get list watch create update patch |
| \* | secrets | \* | get list watch create patch update delete |
| apps | replicasets | \* | get list |
| autoscaling | horizontalpodautoscalers | \* | get |
| odigos.io | \* | \* | get list watch |
| odigos.io | instrumentationrules destinations actions samplings | \* | create patch update delete |
| odigos.io | recommendations | \* | update patch |
| odigos.io | collectorsgroups | \* | get list watch |
| actions.odigos.io | \* | \* | get list watch create patch update delete |
| operator.odigos.io | odigos | \* | get list watch |
| apps | deployments/scale | odigos-instrumentor | get update patch |
| apps | daemonsets | odiglet | get update patch |
| \* | pods | \* | get list delete |
| \* | pods/log | \* | get |
| \* | pods/proxy | \* | get |

# Operator

This section lists the RBAC policies used by the Odigos Operator. Many of these permissions are necessary in order to create the RBAC policies for the components listed above.

## ClusterRoles

| APIGroups | Resources | Resource Names | Verbs |
| - | - | - | - |
| \* | configmaps endpoints secrets services | \* | create delete deletecollection get list patch update watch |
| \* | configmaps/finalizers pods/finalizers | \* | update |
| \* | events | \* | create get list patch watch |
| \* | namespaces | \* | get list patch watch |
| \* | namespaces/status nodes/spec nodes/stats replicationcontrollers replicationcontrollers/status resourcequotas | \* | get list watch |
| \* | nodes | \* | get list patch update watch |
| \* | pods | \* | delete get list watch |
| \* | pods/log pods/proxy pods/status | \* | get |
| \* | serviceaccounts | \* | create delete get list patch watch |
| actions.odigos.io | \* | \* | create delete deletecollection get list patch update watch |
| actions.odigos.io | \*/status | \* | get patch update |
| admissionregistration.k8s.io | mutatingwebhookconfigurations validatingwebhookconfigurations | \* | create delete get list patch update watch |
| apiextensions.k8s.io | customresourcedefinitions | \* | create delete deletecollection get list patch update watch |
| apiregistration.k8s.io | apiservices | \* | create get list patch update watch |
| apps | daemonsets deployments replicasets statefulsets | \* | create delete deletecollection get list patch update watch |
| apps | daemonsets/finalizers deployments/finalizers replicasets/finalizers statefulsets/finalizers | \* | update |
| apps | daemonsets/status deployments/status statefulsets/status | \* | get |
| apps | deployments/scale | \* | get patch update |
| apps.openshift.io | deploymentconfigs deploymentconfigs/finalizers | \* | get list patch update watch |
| argoproj.io | rollouts | \* | get list patch update watch |
| argoproj.io | rollouts/finalizers | \* | update |
| autoscaling | horizontalpodautoscalers | \* | create delete get list patch update watch |
| batch | cronjobs | \* | create delete get list patch update watch |
| batch | cronjobs/finalizers jobs/finalizers | \* | update |
| batch | jobs | \* | create delete get list watch |
| coordination.k8s.io | leases | \* | create delete get list patch update watch |
| discovery.k8s.io | endpointslices | \* | get list watch |
| extensions | daemonsets deployments replicasets | \* | get list watch |
| odigos.io | \* | \* | \* |
| odigos.io | collectorsgroups/finalizers sources/finalizers | \* | update |
| odigos.io | collectorsgroups/status destinations/status instrumentationconfigs/status instrumentationinstances/status processors/status | \* | get list patch update watch |
| odigos.io | instrumentationrules/status recommendations/status | \* | get patch update |
| odigos.io | recommendations sampling | \* | create delete get list patch update watch |
| operator.odigos.io | odigos | \* | create delete get list patch update watch |
| operator.odigos.io | odigos/finalizers | \* | update |
| operator.odigos.io | odigos/status | \* | get patch update |
| policy | podsecuritypolicies | privileged | use |
| rbac.authorization.k8s.io | clusterrolebindings clusterroles rolebindings roles | \* | create delete deletecollection get list patch update watch |
| security.openshift.io | securitycontextconstraints | \* | use |
| authentication.k8s.io | tokenreviews | \* | create |
| authorization.k8s.io | subjectaccessreviews | \* | create |


# Add Cluster Info
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/addclusterinfo

This action adds static resource attributes to spans, metrics data points and log records. It is useful to add cluster-wide attributes to all telemetry signals.

## Considerations

> [!WARNING]
>
> Before enabling **add cluster info**, please note the following:
>
> * This action is meant to be used for **static** attributes that are known in advanced. It is not suitable for dynamic attributes that need to be extracted from an API or calculated at runtime.
> * Some destinations may require specific configuration to index resource attributes and make them available for querying. Please refer to the documentation of the destination you are using for more information.
> * If the attribute already exists in the telemetry signal, the value will be overwritten by the value provided in the action.
> * An empty string is a valid value for the `attributeStringValue` field

## Configuration Options

The AddClusterInfo action is configured using the `odigos.io/v1alpha1.Action` CRD with the `addClusterInfo` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*addClusterInfo \*\*\*
>
> **addClusterInfo** `object` : Configuration for the AddClusterInfo action.
>
> * This field is *required* for this action type
>
> > \*\*clusterAttributes \*\*\*
> >
> > **clusterAttributes** `object[]` : An array of attributes to be added to the telemetry signals.
> >
> > * This field is *required*
> >
> > > \*\*attributeName \*\*\*
> > >
> > > **attributeName** `string` : The name of the attribute to be added to the telemetry signals.
> > >
> > > * This field is *required*
> >
> > > \*\*attributeStringValue \*\*\*
> > >
> > > **attributeStringValue** `string` : The value of the attribute to be added to the telemetry signals.
> > >
> > > * This field is *required*
>
> > **overwriteExistingValues**
> >
> > **overwriteExistingValues** `boolean` : Whether to overwrite existing attribute values.
> >
> > * This field is *optional*, and defaults to `false`

You can include any static value attribute that is meaningful to you, like `team.name` etc.

> [!TIP]
>
> We recommend:
>
> * `k8s.cluster.name`: The name of the k8s cluster which can be useful to distinguish between different clusters.
> * `deployment.environment`: The environment of the cluster (e.g. `production`, `staging`, `development`, etc).

## Basic Example

The following example demonstrates how to insert the `k8s.cluster.name` attribute to all telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml insert-cluster-name.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: insert-cluster-name
  namespace: odigos-system
spec:
  actionName: insert-cluster-name
  signals:
    - TRACES
    - METRICS
    - LOGS
  addClusterInfo:
    clusterAttributes:
      - attributeName: k8s.cluster.name
        attributeStringValue: my-cluster
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f insert-cluster-name.yaml
```


# DB Query Templatization
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/dbquerytemplatization

This action templatizes database query text by replacing number and string literals with placeholders to reduce cardinality.

## Considerations

> [!WARNING]
>
> Before enabling **DB query templatization**, please note the following:
>
> * The action rewrites `db.query.text` (or the deprecated `db.statement`) in place when present.
> * Number and string literals are replaced with `?`. Identifiers, bind/positional parameters, booleans, and `NULL` are left unchanged.
> * Spans without a `db.system.name` (or the deprecated `db.system`) attribute, or with a known non-SQL system (e.g. MongoDB, Redis), are skipped.

## Use Cases

**Cardinality control**

* Raw SQL with embedded literals (e.g. `WHERE id = 1` vs `WHERE id = 2`) creates high-cardinality query attributes and span names. Templatizing literals collapses those into a shared template such as `WHERE id = ?`.

**Cost and backend indexing**

* Lower cardinality on database spans improves aggregations, span metrics, and indexing in your observability backend, and can reduce storage volume.

**Sensitive values in queries**

* Replacing string and number literals removes many accidental secrets or PII that were inlined into SQL (passwords, emails, IDs) while keeping the query structure intact.

## Configuration Options

The DbQueryTemplatization action is configured using the `odigos.io/v1alpha1.Action` CRD with the `dbQueryTemplatization` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*dbQueryTemplatization \*\*\*
>
> **dbQueryTemplatization** `object` : Configuration for the DbQueryTemplatization action.
>
> * This field is *required* for this action type
>
> > **scopes**
> >
> > **scopes** `object` : Limits which sources this templatization config applies to.
> >
> > * This field is *optional*
> >
> > * If unset or empty, the config is applied to all sources.
> >
> > * If multiple source scopes are set, they must all match simultaneously (AND logic).
> >
> > > **sources**
> > >
> > > **sources** `object[]` : A list of workloads to apply this action to.
> > >
> > > * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, etc.).
> >
> > > **namespaces**
> > >
> > > **namespaces** `string[]` : Apply this action to all sources in the listed namespaces.
> >
> > > **languages**
> > >
> > > **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages.
>
> > **templatizeLiterals**
> >
> > **templatizeLiterals** `boolean` : When enabled, number and string literals in SQL queries are replaced with placeholders.
> >
> > * This field is *optional*, and defaults to `false`
> > * What is templatized:
> >   * number literals (e.g. `1`, `3.14`)
> >   * string literals (e.g. `'alice'`)
> > * What is **not** templatized:
> >   * identifiers (table names, column names, etc.)
> >   * positional parameters (e.g. `$1`)
> >   * bind parameters (e.g. `:name`)
> >   * boolean literals and `NULL`
> >   * keywords, operators, punctuation, whitespace, comments, etc.

## Basic Example

Given a span with:

```text theme={null}
db.query.text: SELECT * FROM users WHERE id = 1 AND name = 'alice'
```

the action rewrites it to:

```text theme={null}
db.query.text: SELECT * FROM users WHERE id = ? AND name = ?
```

**Step 1**

Create a YAML file with the following content:

```yaml db-query-templatization.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: db-query-templatization
  namespace: odigos-system
spec:
  actionName: DB Query Templatization
  signals:
    - TRACES
  dbQueryTemplatization:
    templatizeLiterals: true
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f db-query-templatization.yaml
```

## Scoped Example

To limit templatization to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources.

```yaml db-query-templatization-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: db-query-templatization-my-app
  namespace: odigos-system
spec:
  actionName: DB Query Templatization for my-app
  signals:
    - TRACES
  dbQueryTemplatization:
    scopes:
      sources:
        - name: my-app
          namespace: default
          kind: Deployment
    templatizeLiterals: true
```

> [!TIP]
>
> Pair this action with [Infer DB Attributes](./inferdbattributes) so database spans get both templatized query text and inferred `db.operation.name` / `db.collection.name` attributes.


# Delete Attribute
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/deleteattribute

This action will delete the specified attributes from all telemetry signals that are specified in the `signals` field, regardless of the source, or any other condition.

## Considerations

> [!WARNING]
>
> Before enabling **delete attribute**, please note the following:
>
> * This action will delete selected attributes from spans, metric data points and log records, but will not delete the entire span, metric data point or log record.
> * For matching attributes, the attribute value will be deleted entirely. This action is not suitable for partial deletion or masking of the attribute value. For example, if you collect an `http.request.body` attribute, this action can be used to delete the attribute but not a key from the JSON object.
> * While it is common for information to be recorded in attributes, it can also be recorded into names or other fields, for which this action will not be effective.

## Use Cases

**Security**

* By default, OpenTelemetry should not record PII (Personally Identifiable Information) or sensitive data such as passwords, api tokens, creditcard numbers etc. However, if the data is accidentally added to the telemetry signals, or if a manual instrumentation recorded it into an attribute, this action can be used to delete such data.

**Cost Reduction**

* Some vendors charge based on the amount of data ingested. For self hosted destinations, the cost is correlated to the use of cloud resources which grows with the based on the amount of data you process and store. By deleting unnecessary attributes, you can reduce the amount of data ingested and reduce costs.

**Usability**

* Remove duplications - sometimes, the same attribute is added to the telemetry signals by multiple sources. This action can be used to delete the duplicates and keep the data clean. For example, you might find the host name recorded both as `host.name` and `k8s.node.name` which are the same value.
* Verbosity - not all data is useful to all users. You might wish to delete certain attributes when they give no value to your organization or observability needs. Sometimes, recording too much data can obscure the important information and make it harder to find.

## Configuration Options

The DeleteAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `deleteAttribute` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*deleteAttribute \*\*\*
>
> **deleteAttribute** `object` : Configuration for the DeleteAttribute action.
>
> * This field is *required* for this action type
>
> > \*\*attributeNamesToDelete \*\*\*
> >
> > **attributeNamesToDelete** `string[]` : An array of strings representing the names of the attributes to be deleted from the telemetry signals.
> >
> > * This field is *required*

## Basic Example

The following example demonstrates how to delete the `host.name` attribute from all telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml delete-host-name.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: delete-host-name
  namespace: odigos-system
spec:
  actionName: "delete host.name"
  signals:
    - TRACES
    - METRICS
    - LOGS
  deleteAttribute:
    attributeNamesToDelete:
      - host.name
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f delete-host-name.yaml
```


# Extract Attribute
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/extractattribute

This action extracts values from span attributes (payloads, SQL statements, URL paths) into new top-level span attributes.

## Considerations

> [!WARNING]
>
> Before enabling **extract attribute**, please note the following:
>
> * This action only operates on **trace** signals.
> * Extraction reads from a fixed set of high-cardinality string attributes and writes the captured value to a **new** span attribute. The source attributes are scanned in order until a match is found, and only the **first** match is used per extraction rule.
> * The action will scan the following span attributes for matches:
>   * `db.statement`
>   * `db.query.text`
>   * `messaging.message.payload`
>   * `http.request.payload`
>   * `http.response.payload`
> * If the `targetAttributeName` attribute already exists on the span, it will **not** be overridden.
> * Each extraction must specify either `lookupKey` + `dataFormat`, **or** a custom `regex` — not both. When using `regex`, the pattern must contain exactly one capture group; the value of the first capture group is written to the attribute named `targetAttributeName`.
> * Each `targetAttributeName` must be unique within a single action.

## Use Cases

**Accessing deeply nested attributes within strings**

* Easily extract and access values of deeply nested attributes within payloads of JSONs, SQL statements or URLs.

**Indexing high-cardinality identifiers**

* Extract domain-specific identifiers (e.g. `order_id`, `user_id`, `study_id`) embedded inside payloads, SQL statements, or URL paths into top-level span attributes so they can be indexed, filtered, and aggregated by your backend.

**Correlating spans across services**

* Lift correlation IDs that are buried inside structured payloads (JSON envelopes, request/response bodies) onto the span itself so traces can be searched and joined by business identifiers.

**Cost-aware enrichment**

* Avoid sending the full payload to your backend by extracting only the meaningful fields, then dropping or masking the original payload attribute with another action.

## Configuration Options

The ExtractAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `extractAttribute` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*extractAttribute \*\*\*
>
> **extractAttribute** `object` : Configuration for the ExtractAttribute action.
>
> * This field is *required* for this action type
>
> > \*\*extractions \*\*\*
> >
> > **extractions** `object[]` : An ordered array of extraction rules to apply to each span. Must contain at least one entry.
> >
> > * This field is *required*
> >
> > > \*\*targetAttributeName \*\*\*
> > >
> > > **targetAttributeName** `string` : The name of the new span attribute the extracted value will be written to.
> > >
> > > * This field is *required*
> > > * Must be unique across all extractions in the same action.
> >
> > > **lookupKey**
> > >
> > > **lookupKey** `string` : The literal key to look up inside the scanned attributes (resolved via `dataFormat` into a regex).
> > >
> > > * This field is *required* when `dataFormat` is set, and must be omitted when `regex` is set.
> >
> > > **dataFormat**
> > >
> > > **dataFormat** `string` : A pre-set format that defines how `lookupKey` is matched inside the scanned attributes.
> > >
> > > * This field is *required* when `lookupKey` is set, and must be omitted when `regex` is set.
> > > * Supported values:
> > >   * `json` — matches `key: value` pairs in JSON envelopes (e.g. `{"order_id": "42"}` or `order_id: 42`).
> > >   * `sql` — matches `key = value` pairs in SQL statements (e.g. `WHERE order_id = '42'` or `order_id=42`).
> > >   * `resource_path` — matches `key/value` segments inside URL paths (e.g. `/projects/test-project`), stopping at the next `/`, whitespace, `?`, `&`, `#`, or quote.
> >
> > > **regex**
> > >
> > > **regex** `string` : A custom regular expression with a single capture group whose value is written to the attribute named `targetAttributeName`.
> > >
> > > * This field is *required* when `lookupKey` and `dataFormat` are not set.
> > > * The first capture group of the first match is used.

> [!TIP]
>
> Each extraction rule must use **exactly one** of:
>
> * `lookupKey` + `dataFormat` — for the built-in JSON/SQL/URL matchers, or
> * `regex` — for a custom pattern with a single capture group.

## Basic Example

The following example demonstrates the four extraction variations against a single span that has the following attributes:

* `messaging.message.payload`: `{"task_id": "task-7f3a9b2c", "status": "Completed"}`
* `http.request.payload`: `tenants/acme-corp/regions/us-east/workspaces/main/services/analytics/jobs/task-7f3a9b2c/runs/run-001`
* `db.statement`: `SELECT * FROM invoices WHERE invoice_id = '42' -- request_id req-xyz9`

After the action runs, the span will be enriched with the following extracted attributes:

| New attribute name | Extracted value |
| - | - |
| `extracted.json.task_id` | `task-7f3a9b2c` |
| `extracted.sql.invoice_id` | `42` |
| `extracted.url.tenant` | `acme-corp` |
| `extracted.url.region` | `us-east` |
| `extracted.url.job` | `task-7f3a9b2c` |
| `extracted.regex.request_id` | `xyz9` |

**Step 1**

Create a YAML file with the following content:

```yaml extract-attribute.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: extract-attribute
  namespace: odigos-system
spec:
  actionName: extract attribute
  signals:
    - TRACES
  extractAttribute:
    extractions:
      # JSON variation - matches a JSON envelope inside messaging.message.payload
      - lookupKey: task_id
        dataFormat: json
        targetAttributeName: extracted.json.task_id

      # SQL variation - matches `key = 'value'` syntax in SQL statements
      - lookupKey: invoice_id
        dataFormat: sql
        targetAttributeName: extracted.sql.invoice_id

      # URL variation - matches `key/value` segments inside http.request.payload
      - lookupKey: tenants
        dataFormat: resource_path
        targetAttributeName: extracted.url.tenant
      - lookupKey: regions
        dataFormat: resource_path
        targetAttributeName: extracted.url.region
      - lookupKey: jobs
        dataFormat: resource_path
        targetAttributeName: extracted.url.job

      # Custom regex variation - one capture group, value goes to the new attribute
      - regex: "req-([a-z0-9]+)"
        targetAttributeName: extracted.regex.request_id
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f extract-attribute.yaml
```


# Infer DB Attributes
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/inferdbattributes

This action parses database query text and adds attributes such as db.operation.name and db.collection.name.

## Considerations

> [!WARNING]
>
> Before enabling **infer DB attributes**, please note the following:
>
> * Currently, only trace signals are supported.
> * The action parses `db.query.text` when present, otherwise `db.statement`.
> * Existing `db.operation.name` / `db.collection.name` (or legacy `db.operation` / `db.sql.table`) values are **not** overwritten.
> * Attributes are added only when inference is unambiguous: a single SQL operation and a single table/collection.
> * When new operation (and optionally collection) attributes are added, the span name may be updated to `OPERATION collection` (e.g. `SELECT users`) if it does not already contain those values.
> * Spans without a `db.system` / `db.system.name` attribute, or with a known non-SQL system (e.g. MongoDB, Redis), are skipped.

## Use Cases

**Enrich incomplete database spans**

* Some instrumentations record only the raw query text. This action derives OpenTelemetry semantic attributes (`db.operation.name`, `db.collection.name`) so backends can filter and group by operation and table.

**Better span names and metrics**

* Normalized span names such as `SELECT users` make traces easier to read and improve aggregations when span metrics are enabled.

**Consistent attributes across languages**

* Apply the same inference rules cluster-wide, regardless of which language or library produced the span.

## Configuration Options

The InferDbAttributes action is configured using the `odigos.io/v1alpha1.Action` CRD with the `inferDbAttributes` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*inferDbAttributes \*\*\*
>
> **inferDbAttributes** `object` : Configuration for the InferDbAttributes action.
>
> * This field is *required* for this action type
>
> * No additional options are required; an empty object enables inference for matching sources.
>
> > **scopes**
> >
> > **scopes** `object` : Limits which sources this config applies to.
> >
> > * This field is *optional*
> >
> > * If unset or empty, the config is applied to all sources.
> >
> > * If multiple source scopes are set, they must all match simultaneously (AND logic).
> >
> > > **sources**
> > >
> > > **sources** `object[]` : A list of workloads to apply this action to.
> > >
> > > * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, or `DaemonSet`).
> >
> > > **namespaces**
> > >
> > > **namespaces** `string[]` : Apply this action to all sources in the listed namespaces.
> >
> > > **languages**
> > >
> > > **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages.

## Basic Example

Given a span with:

```text theme={null}
db.query.text: SELECT * FROM users WHERE id = 1
```

and without `db.operation.name` / `db.collection.name`, the action adds:

| Attribute | Value |
| - | - |
| `db.operation.name` | `SELECT` |
| `db.collection.name` | `users` |

and updates the span name to `SELECT users` if it hasn't been set to that already.

**Step 1**

Create a YAML file with the following content:

```yaml infer-db-attributes.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: infer-db-attributes
  namespace: odigos-system
spec:
  actionName: Infer DB Attributes
  signals:
    - TRACES
  inferDbAttributes: {}
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f infer-db-attributes.yaml
```

## Scoped Example

To limit inference to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources.

```yaml infer-db-attributes-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: infer-db-attributes-my-app
  namespace: odigos-system
spec:
  actionName: Infer DB Attributes for my-app
  signals:
    - TRACES
  inferDbAttributes:
    scopes:
      sources:
        - name: my-app
          namespace: default
          kind: Deployment
```

> [!TIP]
>
> Pair this action with [DB Query Templatization](./dbquerytemplatization) to both infer semantic attributes and replace literals in the query text.


# Kubernetes Attributes
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/k8sattributes

This action adds kubernetes related resource attributes to spans, metrics data points and log records.

## Overview

By adding Kubernetes-specific attributes to telemetry signals, you can gain better visibility into workloads running in the cluster.

Helps in correlating traces, logs and metrics with specific Kubernetes resources, such as pods, deployments, nodes and clusters.

Adding resource attributes based on custom kubernetes annotations/labels can enrich the data with user defined aggregations and metadata corresponding to the originating pod that produced the telemetry.

## Configuration Options

The K8sAttributes action is configured using the `odigos.io/v1alpha1.Action` CRD with the `k8sAttributes` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES, LOGS, METRICS`

> \*\*k8sAttributes \*\*\*
>
> **k8sAttributes** `object` : Configuration for the K8sAttributes action.
>
> * This field is *required* for this action type
>
> > **collectContainerAttributes**
> >
> > **collectContainerAttributes** `boolean` : Collect the following container related attributes:
> >
> > * `k8s.container.name`
> > * `container.id`
> > * `container.image.name`
> > * `container.image.tag`
> >
> > This field is *optional* and defaults to `false`
>
> > **collectReplicaSetAttributes**
> >
> > **collectReplicaSetAttributes** `boolean` : Collect the following replica set related attributes:
> >
> > * `k8s.replicaset.name`
> >   if CollectWorkloadUID is set to true, the following attribute will be collected as well:
> > * `k8s.replicaset.uid`
> >
> > This field is *optional* and defaults to `false`
>
> > **CollectWorkloadUID**
> >
> > **CollectWorkloadUID** `boolean` : Collect the workload UID:
> >
> > * `k8s.deployment.uid`
> > * `k8s.daemonset.uid`
> > * `k8s.statefulset.uid`
> >
> > Only one of these attributes can appear for a given span/log/metric - based on the workload it originated from.
> > This field is *optional* and defaults to `false`
>
> > **CollectClusterUID**
> >
> > **CollectClusterUID** `boolean` : Collect the `k8s.cluster.uid` attribute, which is set to the uid of the namespace "kube-system". This allows aggregating observability based on the cluster it originated from.
> >
> > This field is *optional* and defaults to `false`
>
> > **labelsAttributes**
> >
> > **labelsAttributes** `K8sLabelAttribute[]` : Collect resource attributes based on the specified labels.
> >
> > Each label attribute can specify:
> >
> > * `labelKey`: The Kubernetes label key to extract
> > * `attributeKey`: The OpenTelemetry attribute key to create
> > * `from` (deprecated): Single source - "pod" or "namespace"
> > * `fromSources`: Array of sources to extract from, with precedence (pod > namespace > node)
> >
> > When using `fromSources` with multiple sources, if the same label exists in multiple sources, the most specific source (e.g., pod) takes precedence over less specific sources (e.g., namespace).
> >
> > This field is *optional* and can remain empty.
>
> > **annotationsAttributes**
> >
> > **annotationsAttributes** `K8sAnnotationAttribute[]` : Collect resource attributes based on the specified annotations.
> >
> > Each annotation attribute can specify:
> >
> > * `annotationKey`: The Kubernetes annotation key to extract
> > * `attributeKey`: The OpenTelemetry attribute key to create
> > * `from` (deprecated): Single source - "pod" or "namespace"
> > * `fromSources`: Array of sources to extract from, with precedence (pod > namespace > node)
> >
> > When using `fromSources` with multiple sources, if the same annotation exists in multiple sources, the most specific source (e.g., pod) takes precedence.
> >
> > This field is *optional* and can remain empty.

## Basic Example

The following example demonstrates how to collect Kubernetes attributes, including labels and annotations, and attach them to telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml k8s-attributes.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: collect-k8s-attributes
  namespace: odigos-system
spec:
  actionName: "Collect Kubernetes Attributes"
  signals:
    - TRACES
    - METRICS
    - LOGS
  k8sAttributes:
    collectContainerAttributes: true
    collectWorkloadUID: true
    collectClusterUID: true
    labelsAttributes:
      - labelKey: "app.kubernetes.io/name"
        attributeKey: "app.kubernetes.name"
        from: "pod"
    annotationsAttributes:
      - annotationKey: "kubectl.kubernetes.io/restartedAt"
        attributeKey: "kubectl.kubernetes.restartedAt"
        from: "pod"
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f k8s-attributes.yaml
```

## Multi-Source Label Extraction Example

The following example demonstrates how to extract the same label from multiple sources with proper precedence. This is useful when you want pod-level labels to override namespace-level labels when both exist.

**Step 1**

Create a YAML file with the following content:

```yaml k8s-attributes-multi-source.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: collect-environment-label
  namespace: odigos-system
spec:
  actionName: "Collect Environment Label with Precedence"
  signals:
    - TRACES
    - METRICS
    - LOGS
  k8sAttributes:
    labelsAttributes:
      - labelKey: "environment"
        attributeKey: "k8s.environment"
        fromSources:
          - pod        # Highest precedence - pod labels override namespace labels
          - namespace  # Lower precedence - used as fallback
```

In this configuration:

* If a pod has an `environment` label, that value will be used
* If the pod doesn't have the label, but the namespace does, the namespace value will be used
* This allows namespace-wide defaults while permitting pod-level overrides

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f k8s-attributes-multi-source.yaml
```


# PII Masking
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/piimasking

This action masks PII from traces using predefined categories and optional custom format or regex rules.

## Considerations

> [!WARNING]
>
> Before enabling **pii masking**, please note the following:
>
> * Predefined PII categories replace matched values with a category-specific token (e.g. `***EMAIL***`, `***CREDIT_CARD***`).
> * Custom format and regex maskings replace only the matched capture group with `****`, leaving the rest of the value intact.
> * Currently, only trace signals are supported.
> * All span attribute values in every matching span will be examined and masked accordingly.

## Use Cases

**Security**

* By default, OpenTelemetry should not record PII (Personally Identifiable Information) or sensitive data such as passwords, api tokens, etc. However, if the data is accidentally added to the telemetry signals, or if a manual instrumentation recorded it into an attribute, this action can be used to mask such data.

**Legal and Compliance**

* Ensure compliance with legal and privacy.
  * Payment Card Industry (PCI) Data Security Standards prohibit logging certain things or storing them unencrypted.

**Domain-specific secrets**

* Mask application-specific fields (e.g. SSN in JSON payloads, passwords in SQL statements, or identifiers in URL paths) using `customFormatMaskings` or `customRegexMaskings` when they are not covered by a predefined PII category.

## Configuration Options

The PiiMasking action is configured using the `odigos.io/v1alpha1.Action` CRD with the `piiMasking` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`

> \*\*piiMasking \*\*\*
>
> **piiMasking** `object` : Configuration for the PiiMasking action.
>
> * This field is *required* for this action type
>
> > **scopes**
> >
> > **scopes** `object` : Limits which sources this masking config applies to.
> >
> > * This field is *optional*
> >
> > * If unset or empty, the config is applied to all sources.
> >
> > > **sources**
> > >
> > > **sources** `object[]` : A list of workloads to apply this action to.
> > >
> > > * Each entry requires `name`, `namespace`, and `kind` (`Deployment`, `StatefulSet`, or `DaemonSet`).
> >
> > > **namespaces**
> > >
> > > **namespaces** `string[]` : Apply this action to all sources in the listed namespaces.
> >
> > > **languages**
> > >
> > > **languages** `string[]` : Apply this action only to containers instrumented with the listed programming languages.
>
> > **piiCategories**
> >
> > **piiCategories** `string[]` : An array of strings representing the PII categories you want to mask.
> >
> > * This field is *optional*
> > * The available PII categories are:
> >   * `CREDIT_CARD` - mask Visa and MasterCard credit card numbers
> >
> >   * `EMAIL` - mask email addresses
> >
> >   * `JWT` - mask JSON Web Tokens
> >
> >   * `UUID` - mask UUIDs
> >
> > > [!NOTE]
> > >
> > > Can't find the PII category you need? Use `customFormatMaskings` or `customRegexMaskings`, or contact us and we will add it for you.
>
> > **customFormatMaskings**
> >
> > **customFormatMaskings** `object[]` : Format-based masking rules applied in order. Each rule looks up a key inside structured attribute values and masks the matched value.
> >
> > * This field is *optional*
> >
> > > \*\*lookupKey \*\*\*
> > >
> > > **lookupKey** `string` : The field or path segment whose value should be masked (e.g. a JSON key, SQL column name, or URL path segment).
> > >
> > > * This field is *required*
> >
> > > \*\*dataFormat \*\*\*
> > >
> > > **dataFormat** `string` : The format of the data to search in.
> > >
> > > * This field is *required*
> > > * Supported values:
> > >   * `json` — matches `key: value` pairs in JSON envelopes (e.g. `{"ssn": "123-45-6789"}` or `ssn: 123-45-6789`).
> > >   * `sql` — matches `key = value` pairs in SQL statements (e.g. `WHERE password = 'hunter2'`).
> > >   * `resource_path` — matches `key/value` segments inside URL paths (e.g. `/orders/abc-123`), stopping at the next `/`, whitespace, `?`, `&`, `#`, or quote.
>
> > **customRegexMaskings**
> >
> > **customRegexMaskings** `object[]` : Regex-based masking rules applied in order.
> >
> > * This field is *optional*
> >
> > > \*\*regex \*\*\*
> > >
> > > **regex** `string` : A custom regular expression with a single capture group whose value is masked with `****`.
> > >
> > > * This field is *required*
> > > * The pattern must contain at least one capture group; only the first capture group is replaced.

## Basic Example

The following example demonstrates how to configure the Pii Masking action to mask Visa and MasterCard credit card numbers from span attribute values using the Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml pii-masking.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: pii-masking
  namespace: odigos-system
spec:
  actionName: PII Masking
  signals:
    - TRACES
  piiMasking:
    piiCategories:
      - CREDIT_CARD
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f pii-masking.yaml
```

## Custom Masking Example

The following example combines a predefined category with custom format and regex rules. Given span attributes such as:

* `{"ssn": "123-45-6789", "name": "alice"}`
* `WHERE password = 'hunter2' AND status = 'ok'`
* `/api/v1/orders/abc-123/items`
* `auth api_key=super-secret-value next`

the action produces:

| Input fragment | Masked result |
| - | - |
| `user@example.com` | `***EMAIL***` |
| `"ssn": "123-45-6789"` | `"ssn": "****"` |
| `password = 'hunter2'` | `password = '****'` |
| `/orders/abc-123/` | `/orders/****/` |
| `api_key=super-secret-value` | `api_key=****` |

**Step 1**

Create a YAML file with the following content:

```yaml pii-masking-custom.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: pii-masking-custom
  namespace: odigos-system
spec:
  actionName: Custom PII Masking
  signals:
    - TRACES
  piiMasking:
    piiCategories:
      - EMAIL
    customFormatMaskings:
      - lookupKey: ssn
        dataFormat: json
      - lookupKey: password
        dataFormat: sql
      - lookupKey: orders
        dataFormat: resource_path
    customRegexMaskings:
      - regex: 'api[_-]?key=([^\s&]+)'
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f pii-masking-custom.yaml
```

## Scoped Example

To limit masking to specific workloads, add `scopes`. Omit `scopes` (or leave it empty) to apply the action to all sources.

```yaml pii-masking-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: pii-masking-my-app
  namespace: odigos-system
spec:
  actionName: PII Masking for my-app
  signals:
    - TRACES
  piiMasking:
    scopes:
      sources:
        - name: my-app
          namespace: default
          kind: Deployment
    piiCategories:
      - CREDIT_CARD
      - EMAIL
```


# Rename Attribute
Source: https://docs.odigos.io/oss/pipeline/actions/attributes/renameattribute

This action will rename the specified attributes from all telemetry signals that are specified in the `signals` field, regardless of the source, or any other condition.

## Configuration Options

The RenameAttribute action is configured using the `odigos.io/v1alpha1.Action` CRD with the `renameAttribute` configuration section.

> **actionName**
>
> **actionName** `string` : Allows you to attach a meaningful name to the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the action for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the action, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the action will operate on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*renameAttribute \*\*\*
>
> **renameAttribute** `object` : Configuration for the RenameAttribute action.
>
> * This field is *required* for this action type
>
> > \*\*renames \*\*\*
> >
> > **renames** `object` : A map of strings, key representing the name of the attribute to rename and values the desired name.
> >
> > * This field is *required*

## Basic Example

The following example demonstrates how to rename the deprecated `net.peer.address` attribute to stable `network.peer.address` in all telemetry signals using the new Action CRD.

**Step 1**

Create a YAML file with the following content:

```yaml rename-net-to-network.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Action
metadata:
  name: rename-net-to-network
  namespace: odigos-system
spec:
  actionName: Rename network attributes
  signals:
    - LOGS
    - METRICS
    - TRACES
  renameAttribute:
    renames:
      net.peer.address: network.peer.address
      net.local.address: network.local.address
      net.peer.ip: network.peer.address
      net.host.ip: network.local.address
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f rename-net-to-network.yaml
```


# Customized
Source: https://docs.odigos.io/oss/pipeline/actions/crd

You can define custom actions using OpenTelemetry Collector Processors in Odigos.

> [!NOTE]
>
> This option is for advanced users who are familiar with the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), its [configuration](https://opentelemetry.io/docs/collector/configuration/), and with [Kubernetes CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/).

You can browse the full list of processors that Odigos includes [here](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61).

To view the configuration for each processor, it is recommended to visit the README of this component in [OpenTelemetry Collector Contrib codebase](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor).

Adding a processor to your Odigos pipeline can be done by creating a `Processor` custom resource in your Kubernetes cluster under the `odigos-system` namespace:

## Configuration Options

> **processorName**
>
> **processorName** `string` : Allows you to attach a meaningful name to the processor for convenience.
>
> * This field is *optional*
> *

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the processor for convenience.
>
> * This field is *optional*
> *

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the processor, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> \*\*signals \*\*\*
>
> **signals** `string[]` : An array with the signals that the processor will act on.
>
> * This field is *required*
> * Supported values: `TRACES`, `METRICS`, `LOGS`

> \*\*collectorRoles \*\*\*
>
> **collectorRoles** `string[]` : An array with the collector roles that the processor will act on.
>
> * This field is *required*
> * Supported values: `CLUSTER_GATEWAY`, `NODE_COLLECTOR`

> \*\*type \*\*\*
>
> **type** `string` : The type of the processor.
>
> * This field is *required*
> * The name of the processor you want to use, as defined in the OpenTelemetry Collector (batch, attributes, etc).

> **orderHint**
>
> **orderHint** `number` : If your processors need to run in a specific order relatively to other processors, you can hint the order by setting an integer value here.
>
> * This field is *optional*
> * The lower the value, the earlier the processor will run in the collector pipeline. If the value is missing or 0, the processor will run in an arbitrary order in the collector pipeline.

> **processorConfig**
>
> **processorConfig** `object` : A field to pass configuration to the processor.
>
> * This field is *optional*
> * The structure of this field is specific to each processor, and you can find the configuration options for each processor in the [OpenTelemetry Collector Contrib codebase](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor).

## Basic Example

The following example demonstrates how to add a resource attribute `deployment.environment` with the value `production` to all spans in the cluster.

**Step 1**

Create a YAML file with the following content:

```yaml example-processor.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Processor
metadata:
  name: example-processor
  namespace: odigos-system
spec:
  type: resource
  processorConfig:
    attributes:
      - key: deployment.environment
        value: production
        action: insert
  signals:
    - TRACES
  collectorRoles:
    - CLUSTER_GATEWAY
```

**Step 2**

Apply the action to the cluster:

```bash theme={null}
kubectl apply -f example-processor.yaml
```


# Introduction
Source: https://docs.odigos.io/oss/pipeline/actions/introduction

Actions are a way to modify the OpenTelemetry data recorded by Odigos Sources, before it is exported to your destinations.

## What are Processors?

Under the hood, Odigos uses the [OpenTelemetry Collector Processors](https://opentelemetry.io/docs/collector/configuration/#processors) to implement actions.

Processors are a class of OpenTelemetry Collector component. They are used to transform, filter, or enrich your OpenTelemetry data before it is sent to your [destinations](../../backends-overview).

## Benefits

* Odigos Actions are language agnostic. They run in the OpenTelemetry collectors and can act on any data type (traces, metrics, logs) no matter the source or the programming language of your service.
* Processing the data on the k8s node or the cluster level reduces the load on your services and allows you to apply the same processing rules to all your services and signals.
* Applying actions in your cluster can reduce the volume of data sent to your backend, which can reduce your costs.
* OpenTelemetry maintains a large collection of processors, which are ready to use and can be configured to your needs.

Actions are a powerful tool to help you manage your observability data and ensure that you are sending the right data to your backend.

## Action Types

### Attribute Actions

Attribute Actions are a way to modify the OpenTelemetry data recorded by Odigos Sources, before it is exported to your destinations.

* [Add Cluster Info](../../pipeline/actions/attributes/addclusterinfo)
* [Delete Attribute](../../pipeline/actions/attributes/deleteattribute)
* [Rename Attribute](../../pipeline/actions/attributes/renameattribute)
* [Pii Masking](../../pipeline/actions/attributes/piimasking)
* [Kubernetes Attributes](../../pipeline/actions/attributes/k8sattributes)
* [Extract Attribute](../../pipeline/actions/attributes/extractattribute)
* [DB Query Templatization](../../pipeline/actions/attributes/dbquerytemplatization)
* [Infer DB Attributes](../../pipeline/actions/attributes/inferdbattributes)

### Sampling Actions

Sampling Actions decide which traces are kept and which are dropped before they reach your destinations. Rules are organized into three categories, evaluated in order—**Noisy Operation → Highly Relevant → Cost Reduction**—so you can drop noise early, guarantee retention for important traffic, and cap everything else for cost.

* [Sampling overview](../../pipeline/sampling/introduction)
* [Head and tail sampling](../../pipeline/sampling/head-and-tail-sampling)
* [Rules overview](../../pipeline/sampling/rules/overview)
* [Configuration reference](../../pipeline/sampling/configuration)
* [Examples](../../pipeline/sampling/examples)

### Custom Actions

Odigos also supports adding OpenTelemetry processors with [Kubernetes CRDs](../../pipeline/actions/crd) which you can apply manually, or through a GitOps workflow.


# Troubleshooting
Source: https://docs.odigos.io/oss/pipeline/actions/troubleshooting

If you are experiencing issues with your actions and processors, use the checklist below to troubleshoot the issue.

## View Collector Configurations

To view the current configurations of the collectors, you can check the ConfigMap objects in the `odigos-system` namespace.

You should see the processors configured under `data/conf/processors`, and actions configured under `data/conf/pipelines`.

**Node Collector:**

```bash theme={null}
kubectl get cm -n odigos-system -o yaml odigos-data-collection
```

**Gateway Collector:**

```bash theme={null}
kubectl get cm -n odigos-system -o yaml odigos-gateway
```

## View Instrumentor Logs

The `odigos-instrumentor` controller is the component that watches for changes in the action or processor CRD and generates the collector configuration.

If you don't see your processor in the collector configuration, you can check the logs of the instrumentor for errors or clues:

```sh theme={null}
kubectl logs -n odigos-system deployments.apps/odigos-instrumentor
```

## View Collector Logs

If you do see your processor in the collector configuration, and there's an issue, the collector should log an error and will not start. To check the status of the Odigos pods, run:

```sh theme={null}
kubectl get pods -n odigos-system
```

Look for pods with STATUS `CrashLoopBackOff` or `Error`.\

To see the error, you can examine the logs of that pod:

```sh theme={null}
kubectl logs -n odigos-system <pod-name>
```

You might find an error message like this:

```
* error decoding 'processors': unknown type: "invalid" for id: "invalid/example-processor" (valid values: [datadog experimental_metricsgeneration metricstransform resource servicegraph spanmetrics sumologic batch tail_sampling cumulativetodelta resourcedetection remotetap memory_limiter filter redaction routing span attributes deltatorate groupbyattrs groupbytrace k8sattributes probabilistic_sampler])
```

If this is the case, then please follow the instructions in the error message to correct the processor configuration.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Pipeline Benchmarks
Source: https://docs.odigos.io/oss/pipeline/benchmarks

Choose the right ResourceSizePreset for your cluster based on telemetry volume and scale.

The [ResourceSizePreset](./configuration#1-using-sizing-configuration) profiles (`size_s`, `size_m`, `size_l`, and `size_xl`) map collector resources and scaling headroom to telemetry volume. Use this page to assist you in picking a preset.

## Total span throughput

Span throughput is the metric we use for benchmarking and estimating the resources needed to operate Odigos.

To calculate the total span throughput (bytes/second), you can either:

* Estimate your monthly span volume in bytes, which can often be inferred from your existing observability tools. If you only know the number of spans, multiply it by the average span size in bytes to estimate the total volume. Then convert the monthly volume to bytes per second.
* View the span throughput in the Odigos UI under "collectors pipeline".

> [!NOTE]
>
> Collector pipeline UI requires enabling the [Odigos Own Metrics](./odigos-own-metrics) feature.

Note that while **total span throughput** is the primary metric for determining the appropriate Odigos preset size, additional factors can affect the maximum span throughput your Odigos deployment can handle:

* **Number of instrumented pods** - each instrumented pod consumes a small amount of additional static resources in the node collector.
* **Pipeline features** - actions and destinations consume additional collector CPU and memory resources.

## Cluster preset vs. total span throughput

### Maximal single collector capacity

#### Cluster gateway (1 pod) ([see maximum capacity here](#maximum-scaled-cluster))

The cluster gateway is a Deployment that receives telemetry from node collectors, processes it, and exports it to your destinations.

![Cluster gateway benchmark throughput by ResourceSizePreset](/images/shared/pipeline/cluster-light.png)

| Size | spans/s | span throughput (MB/second) |
| - | -: | -: |
| `size_s` | **3,000** | **5.8** |
| `size_m` | **18,000** | **34.5** |
| `size_l` | **23,000** | **44.1** |
| `size_xl` | **31,500** | **60.4** |

#### Node collector

The node collector is a DaemonSet (one pod per node) that receives telemetry from apps on that node and forwards it to the cluster gateway.

![Node collector benchmark throughput by ResourceSizePreset](/images/shared/pipeline/nc-community-light.png)

| Size | spans/s | span throughput (MB/second) |
| - | -: | -: |
| `size_s` | **23,500** | **45.1** |
| `size_m` | **35,000** | **67.1** |
| `size_l` | **49,500** | **94.9** |
| `size_xl` | **54,500** | **104.5** |

### Maximum scaled cluster gateway capacity

| Size | HPA Maximum Replicas | spans/s | span throughput (MB/second) |
| - | -: | -: | -: |
| `size_s` | **5** | **15,000** | **28.8** |
| `size_m` | **8** | **144,000** | **276.2** |
| `size_l` | **12** | **276,000** | **529.4** |
| `size_xl` | **15** | **472,500** | **906.3** |

\* Calculated with a span size of 1.9KB

## Troubleshooting

If you experience any of the following symptoms consistently:

* Memory limit hits or OOMKills
* CPU throttling under steady load
* Gateway HPA scaling to max replicas frequently
* Export backlogs or dropped spans

Consider selecting a larger preset or tune individual `collectorGateway` / `collectorNode` values on the [Configuration](./configuration) page.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Configuring the Pipeline
Source: https://docs.odigos.io/oss/pipeline/configuration

Configure the Odigos pipeline to optimize the performance and resource usage of the Odigos components.

## Configurable Components

* **Cluster Gateway Collector** - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, processes it, and exports it to the configured destinations.
* **Node Data Collection Collector** - a collector that runs as a k8s DaemonSet. It collects / scrape the OpenTelemetry data from the applications running on the nodes and forwards it to the Cluster Gateway Collector.

## Configuring the Pipeline

Odigos offers two main options for configuring the pipeline:

> [!WARNING]
>
> When configuring the pipeline, you must use **only one of the two methods** — either `profiles`, or directly editing the `odigos-configuration` ConfigMap.
>
> Using both methods will result in configurations overwriting each other, potentially leading to unexpected behavior.

#### 1. Using Sizing Configuration

To configure resource sizing at install time, set the `ResourceSizePreset` value in your Helm chart when installing Odigos—either in your `values.yaml` file or by using `--set ResourceSizePreset=<preset>` in the `helm install` command.

This approach streamlines the setup process and applies optimized defaults for common use cases.

The available sizing options—`size_s`, `size_m`, `size_l`, and `size_xl`—are predefined profiles designed to simplify pipeline configuration. Each profile specifies a defined set of parameters, as detailed below:

Don't know which size is the right one for your cluster? Read more in [Benchmarks](./benchmarks).

**Cluster Gateway Collector**:

| Size | Minimum Replicas | HPA Maximum Replicas | Request CPU (m) | Limit CPU (m) | Request Memory (Mi) | Limit Memory (Mi) |
| - | - | - | - | - | - | - |
| `size_s` | **1** | **5** | **300m** | **300m** | **300Mi** | **300Mi** |
| `size_m` | **2** | **8** | **1000m** | **1000m** | **600Mi** | **600Mi** |
| `size_l` | **3** | **12** | **1250m** | **1250m** | **850Mi** | **850Mi** |
| `size_xl` | **5** | **15** | **1500m** | **1500m** | **2000Mi** | **2000Mi** |

**Node Data Collection Collector**:

| Size | Request Memory (Mi) | Limit Memory (Mi) | Request CPU (m) | Limit CPU (m) |
| - | - | - | - | - |
| `size_s` | **150Mi** | **300Mi** | **150m** | **300m** |
| `size_m` | **250Mi** | **500Mi** | **250m** | **500m** |
| `size_l` | **500Mi** | **750Mi** | **500m** | **750m** |
| `size_xl` | **1024Mi** | **1024Mi** | **1000m** | **1000m** |

#### 2. Advanced Pipeline Configuration

For advanced control over the pipeline, you can override the default sizing and fine-tune resource and scaling parameters for the `collectorGateway` and `collectorNode`. The way you apply these advanced settings depends on how you installed Odigos:

* **If you installed Odigos using the CLI:**
  Edit the `odigos-configuration` ConfigMap directly in your cluster. This allows you to set advanced options for both `collectorGateway` and `collectorNode` (such as custom CPU/memory requests, limits, and replica counts).

* **If you installed Odigos using Helm:**
  Make your changes in the `values.yaml` file or by using the `--set` flag during installation or upgrade. **Do not** edit the `odigos-configuration` ConfigMap directly, as Helm manages this resource and will overwrite manual changes.

This method gives you full control over the pipeline configuration, letting you override any default or profile-based settings to match your specific requirements.

Below is an example of how to configure the `collectorGateway` and `collectorNode` parameters:

```yaml theme={null}
  # HYBRID CONFIGURATION APPROACH:
  # By default, all values below are automatically set based on the ResourceSizePreset values.yaml attribute describe above.
  # Uncomment and modify any value to override the automatic sizing for that specific parameter.
  #
  # IMPORTANT CONSTRAINTS:
  # 1. minReplicas must be less or equal to maxReplicas
  # 2. If you set limitMemoryMiB without requestMemoryMiB, request will equal limit
  # 3. If you set requestMemoryMiB without limitMemoryMiB, limit will equal request
  # 4. Same logic applies to CPU settings

collectorGateway:
  # the memory request for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource request
  # of the form "memory: <value>Mi".
  # default value is 600Mi
  # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value.
  # requestMemoryMiB: 600

  # the memory limit for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource limit
  # of the form "memory: <value>Mi".
  # default value is 600Mi
  # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value.
  # limitMemoryMiB: 600

  # the CPU request for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource request
  # of the form "cpu: <value>m".
  # default value is 1000m
  # If you set only requestCPUm, the limitCPUm will be set to the same value.
  # requestCPUm: 1000

  # the CPU limit for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource limit
  # of the form "cpu: <value>m".
  # default value is 1000m
  # If you set only limitCPUm, the requestCPUm will be set to the same value.
  # limitCPUm: 1000

  # The number of replicas for the cluster gateway collector deployment.
  # Also uses in MinReplicas the HPA config.
  # minReplicas: 2
  # The maxReplicas in the HPA config.
  # maxReplicas: 8

  # sets the "limit_mib" parameter in the memory limiter configuration for the collector gateway.
  # it is the hard limit after which a force garbage collection will be performed.
  # if not set, it will be 50Mi below the memory limit.
  # memoryLimiterLimitMiB: 550

  # sets the "spike_limit_mib" parameter in the memory limiter configuration for the collector gateway.
  # note that this is not the processor soft limit, but the diff in MiB between the hard limit and the soft limit.
  # if not specified, this value will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit).
  # memoryLimiterSpikeLimitMiB: 110

  # the GOMEMLIMIT environment variable value for the collector gateway deployment.
  # this is when go runtime will start garbage collection.
  # if not specified, it will be set to 80% of the hard limit of the memory limiter.
  # goMemLimitMiB: 440

collectorNode:
  # RequestMemoryMiB is the memory request for the node collector daemonset.
  # it will be embedded in the daemonset as a resource request of the form "memory: <value>Mi"
  # default value is 250Mi
  # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value.
  # requestMemoryMiB: 250

  # LimitMemoryMiB is the memory limit for the node collector daemonset.
  # it will be embedded in the daemonset as a resource limit of the form "memory: <value>Mi"
  # default value is 500Mi
  # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value.
  # limitMemoryMiB: 500

  # the CPU request for the node collector daemonset.
  # it will be embedded in the daemonset as a resource request
  # of the form "cpu: <value>m".
  # default value is 250m
  # If you set only requestCPUm, the limitCPUm will be set to the same value.
  # requestCPUm: 250

  # the CPU limit for the node collector daemonset.
  # it will be embedded in the daemonset as a resource limit
  # of the form "cpu: <value>m".
  # default value is 500m
  # If you set only limitCPUm, the requestCPUm will be set to the same value.
  # limitCPUm: 500

  # this parameter sets the "limit_mib" parameter in the memory limiter configuration for the node collector.
  # it is the hard limit after which a force garbage collection will be performed.
  # if not set, it will be 50Mi below the memory limit.
  # memoryLimiterLimitMiB: 450

  # this parameter sets the "spike_limit_mib" parameter in the memory limiter configuration for the node collector.
  # note that this is not the processor soft limit, but the diff in Mib between the hard limit and the soft limit.
  # if not set, this will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit).
  # memoryLimiterSpikeLimitMiB: 90

  # the GOMEMLIMIT environment variable value for the node collector daemonset.
  # this is when go runtime will start garbage collection.
  # if not specified, it will be set to 80% of the hard limit of the memory limiter.
  # goMemLimitMiB: 360
```

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Datastreams
Source: https://docs.odigos.io/oss/pipeline/datastreams

Datastreams let you split your observability data into multiple, independent pipelines - each routed to its own destination and optionally enhanced with custom actions like sampling, enrichment, or masking.

> [!WARNING]
>
> This feature is in beta. It may be subject to changes and improvements based on user feedback.

# Introduction

Odigos supports Datastreams, a powerful feature that allows you to split telemetry sources into multiple logical streams. Each stream can be independently routed to different destinations and have specific actions applied—such as [sampling](../pipeline/sampling/introduction), attribute enrichment, masking rules, and more.

# Motivation

There are many use cases for adopting Datastreams, including:

1. **Cost optimization** - Route lower-priority services (like internal or backoffice apps) to more cost-effective destinations, such as Parquet files stored in S3.

2. **Targeted processing** - Apply custom processing (e.g., [sampling](../pipeline/sampling/rules/overview), enrichment) to only a subset of services.

3. **Gradual migration** - Test a new destination on specific services before committing to a full migration.

4. **Organizational segmentation** - Split data based on development teams, product areas, or environments.

# Creating Datastreams

**UI**

After [installing Odigos CLI](../setup/installation), run the following command to start the Odigos UI:

```bash theme={null}
odigos ui
```

### In a new Odigos install

You'll be taken to the Setup page. This is where your journey with Odigos begins, and you'll be asked to provide a few initial details:

2. Provide a name for the Datastream
3. Select the telemetry [Sources](../pipeline/sources/introduction) to include
4. Choose the [destination(s)](../backends-overview) to send the data to
5. Click **Done**

### In an existing Odigos install

You can create a new Datastream from the Overview page. Click the `+ NEW` button at the top-left (under the current Datastream views) and follow the same steps as in the setup phase.

**Kubernetes Manifests**

This example demonstrates how to configure `example-source` to send telemetry to `jaeger-example` through the `example` Datastream.

1. Create or update a Source custom resource with the Datastream label, as described in the [linked documentation](../../pipeline/sources/create#workload-source).

*For example:*

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
  labels:
    odigos.io/data-stream-example: "true"
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
```

2. Create or Update a [destination](../../backends-overview) custom resource spec with the Datastream source selector.

*For example:*

```yaml theme={null}
  sourceSelector:
    dataStreams:
    - example
```

Now let's take [Jaeger](../../backends/jaeger) as complete example here:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: jaeger-example
  namespace: odigos-system
spec:
  data:
    JAEGER_URL: <Jaeger OTLP gRPC Endpoint>
  destinationName: jaeger
  signals:
  - TRACES
  type: jaeger
  sourceSelector:
    dataStreams:
    - example
```

> 💡 Tip: Even if you configure datastreams via manifests, you can still view and manage them through the Odigos UI for better visibility.

# Caveats and Considerations

1. **Trace integrity** - Distributed traces often span multiple services that participate in the same operation. If you assign these services to different datastreams, the trace will be broken across destinations. Each destination will only receive part of the trace, making it incomplete and harder to troubleshoot. To preserve trace integrity, keep all services involved in a single transaction within the same datastream.

2. **Action support** - At present, actions defined in Odigos are applied globally to all streams. Stream-specific actions (e.g., custom [sampling rules](../pipeline/sampling/rules/overview) per datastream) are not yet supported but will be in a future release.


# Odigos Own Metrics
Source: https://docs.odigos.io/oss/pipeline/odigos-own-metrics

Enable Odigos self-telemetry and the bundled VictoriaMetrics store for pipeline visibility in the UI.

## Overview

Odigos collects **own metrics** about its pipeline — collector throughput, export rates, eBPF instrumentation counters, and other internal telemetry. These metrics describe Odigos itself, not your application workloads.

By default, the bundled metrics store is **disabled** (`ownTelemetry.metricsStore.disabled: true`). When you enable it, Odigos deploys a VictoriaMetrics instance (`odigos-victoriametrics`) in the Odigos namespace and routes own metrics there so the UI can query them.

## What enabling the metrics store does

When `ownTelemetry.metricsStore.disabled` is set to `false`:

1. **VictoriaMetrics is deployed** — a single-replica `odigos-victoriametrics` Deployment with 1-day retention stores own metrics inside the cluster.
2. **Collectors report own telemetry** — node collectors and the cluster gateway scrape their OpenTelemetry internal `/metrics` endpoints and forward the data through the gateway to VictoriaMetrics.
3. **Odiglet instrumentation metrics are collected** — node collectors scrape eBPF instrumentation counters from odiglet and forward them through the gateway to the store.
4. **Odigos UI representation** — metrics are display under "collectors pipeline".

Separately from the metrics store, collectors always export traffic-size metrics directly to the UI server over OTLP. That path powers **per-source and per-destination throughput** (`throughputBytes`, `totalDataSent`) in the UI without requiring the metrics store.

## Enabling

The metrics store is disabled by default to keep the Odigos footprint minimal on small clusters.

**Helm yaml**

```yaml theme={null}
ownTelemetry:
  metricsStore:
    disabled: false
```

**Helm command**

```bash theme={null}
helm upgrade --install odigos odigos/odigos \
  --namespace odigos-system \
  --reuse-values \
  --set ownTelemetry.metricsStore.disabled=false
```

## Configuration options

### Export to external metrics destinations

You can also send own metrics to your existing metrics backends by enabling **Collect Odigos own metrics** on a metrics destination. When either the metrics store or at least one such destination is enabled, collectors start the own-metrics collection pipeline.

See [Odigos own metrics settings](/api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-OdigosOwnMetricsSettings) in the API reference for the full `OdigosOwnMetricsSettings` schema.

## Resource footprint

The bundled VictoriaMetrics instance requests roughly:

| Resource | Request | Limit |
| - | - | - |
| CPU | 100m | 500m |
| Memory | 512Mi | 2Gi |

Metrics are retained for **1 day** on ephemeral storage (`emptyDir`).

## Using own metrics for sizing

Own metrics help you understand how much telemetry Odigos is processing before you change [ResourceSizePreset](./configuration#1-using-sizing-configuration) or collector resources.

1. Enable the metrics store.
2. Instrument representative workloads and let traffic run for a few minutes.
3. Check **source throughput** and **destination throughput** in the Odigos UI.
4. Compare total span throughput (bytes/sec) with the [Benchmarks](./benchmarks) preset tables to pick `size_s` through `size_xl`.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Overview
Source: https://docs.odigos.io/oss/pipeline/overview



## High Level Architecture

![Odigos Architecture](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/odigos_architecture.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=d3eb43d4f678adfc133067a0fe684b32)

The Odigos Pipeline is composed of the following components:

* **Instrumentations** - code that records OpenTelemetry signals from your application. It can be code in your application native programming language or eBPF programs that record OpenTelemetry data from the linux kernel.
* **OpenTelemetry SDK** - a library that runs in each instrumented process. It receives the OpenTelemetry data from instrumentations and export it out of the process boundary.
* **Node Collector** - also referred to as `data-collection collector`. Each node in the cluster runs a single instance of this OpenTelemetry collector (DaemonSet). It receives the OpenTelemetry data from the OpenTelemetry SDKs, process it, and exports it to the next component in the pipeline.
* **Cluster Gateway Collector** - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, process it, and export it to the next component in the pipeline.

## Component types:

* **Kubernetes Operators** - reconcile the desired state of cluster resources and deploy / configure the necessary components.
* **DaemonSets** - perform node-level tasks, such as interacting with a process running on the node OS, accessing the Kubelet, and reading logs from the node filesystem.
* **Deployments** - run cluster-level components like the OpenTelemetry Collector and the Odigos UI.

## Component Segments

Odigos is divided into three main segments:

* Instrumentation
  * Inject OpenTelemetry agents to collect the instrumentation data from Odigos sources (instrumented applications)
  * Record and report runtime information from the process environment that is relevant to instrumentation
  * Export collected telemetry to the pipeline.

* Pipeline
  * Offload common or heavy tasks from the instrumented application runtime
  * Apply Odigos Actions on the telemetry data
  * Batch and send the telemetry data to the configured destinations

* Odigos Management
  * orchestrate the pipeline and the instrumentation based on the user's configuration and cluster state,
  * provide a UI for the odigos control plane.

## Pipeline

### Instrumentor (Pipeline Manager)

**Type**: The instrumentor is a Kubernetes operator\

**Role**: The instrumentor is responsible for managing the OpenTelemetry Collectors which implement the OpenTelemetry pipeline.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-instrumentor`

* Start and stop Collectors when needed (e.g. when sources or destinations are added or removed).
* Schedule the node collector to start only after the cluster collector is ready.
* Generate the configuration for the Collectors based on Odigos sources, destinations, and Actions.

### Cluster Collector (Gateway Collector)

**Type**: Cluster collector is a deployment of the OpenTelemetry collector\

**Role**: The cluster collector receives and processes all telemetry collected in the cluster, applies Actions, and exports data to the configured destinations.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-gateway`

* Batch telemetry data for better performance and export it to the configured destinations
* Modify the telemetry data based on the Odigos Actions.
* Apply [tail sampling](./sampling/head-and-tail-sampling) based on the Odigos [sampling rules](./sampling/rules/overview) to reduce the volume of data sent to the destinations.
* Collect metrics about the entities and report it to the UI for monitoring.
* Offload tasks from the agents, to reduce the overhead on the workload runtime environment
* Centralize any common processing that can be shared across multiple agents.

### Node Collector

**Type**: Node collector is a daemon set and will run one pod per node in the cluster\

**Role**: The Node Collector (in the context of Pipeline component) receives telemetry data from agents running on the node, enriches it with node-level attributes, and forward it efficiently and reliably to the cluster collector.\

**K8s Resource**: `kubectl get daemonsets -n odigos-system odigos-data-collection`

* Receive telemetry data from agents running on the node (Odiglet for eBPF based agents, and instrumentation processes for native OpenTelemetry agents).
* Add node-level attributes to the telemetry data (`k8s.node.name`, cloud attributes, etc).
* Batch and send telemetry data to the Cluster Collector, offloading any network overhead from the instrumented application runtime.

> [!NOTE]
>
> The Node Collector is also responsible for recording some k8s metrics, and collecting the generated logs from the node fs. Read more Under the [Instrumentation](#node-collector-2) section.

## Instrumentation

### Instrumentor (Auto Instrumentation Agent Manager)

**Type**: Instrumentor is a Kubernetes operator\

**Role**: The Instrumentor is responsible for managing auto instrumentation agents which collect telemetry data from Odigos sources.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-instrumentor`

* Create and manage `instrumentationconfig.odigos.io` objects
  * Trigger Odiglet to inspect runtime information of workloads.
  * Trigger deployment of the Odigos pipeline.
  * Merge Odigos instrumentation rules and config to a workload scope configuration for the agent.
* Inject the instrumentation device into the workload manifest resource as virtual device.
* Monitor the `source.odigos.io` CRD and cleanup any instrumentation related resources when a source is removed.
* Monitor the Odigos sources and destinations to trigger deployment of the Odigos pipeline.
* Reconcile Odigos configuration to run validations, set defaults, and apply Odigos Profiles (configuration presets) to the cluster.

### Odiglet

**Type**: Odiglet is a daemon set and will run one pod per node in the cluster\

**Role**: Odiglet sets up the node environment to run telemetry collection agents.\

**K8s Resource**: `kubectl get daemonsets -n odigos-system odiglet`

* Inspect container runtime details of instrumented sources (to determine programming language, relevant environment variables, runtime versions, etc).
* Implement the Kubernetes virtual instrumentation devices. In the presence of a supported device on the pod, Odiglet will mount the agent code from file system to the pod and add the relevant environment variables to initiate instrumentation.
* For eBPF-based instrumentation, Odiglet will inject the eBPF code into the pod, pick up telemetry from eBPF, and push it downstream to the Node Collector.
* Monitor for config changes to instrumentations and update the relevant agents accordingly at runtime.

### Node Collector

**Role**: Node collector (in the context of instrumentation) is responsible for scraping Kubernetes metrics, and collect logs from the node filesystem.

* When a metrics destination exists, the node collector will use the OpenTelemetry Collector ["kubeletstats receiver"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kubeletstatsreceiver) to scrape k8s related metrics from the Kubelet and export them as OpenTelemetry metrics to the cluster collector.
* When a logs destination exists, the node collector will use the OpenTelemetry Collector ["filelog receiver"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) to read the relevant pod logs from the node filesystem and export them as OpenTelemetry logs to the cluster collector.

## Odigos

### Odigos UI

**Type**: Odigos UI is a web application backed by a deployment in the cluster and accessible by the user's browser.\

**Role**: The Odigos UI is a web application that allows users to configure various aspects of the Odigos installation.\

**K8s Resource**: `kubectl get deployments -n odigos-system odigos-ui`

* Serve as web server for the Odigos UI
  * Serve static files to the frontend
  * Provide a GraphQL API for the frontend, which in turn interacts with Kubernetes API to manage the Odigos installation.
* Provide a user interface for monitoring and managing the Odigos installation, including the sources, destinations, actions, and instrumentation rules.
* Examine the current state of Odigos and show any errors.


# Code Attributes
Source: https://docs.odigos.io/oss/pipeline/rules/codeattributes

The "Code Attributes" rule can be used to add semantic Code attributes to spans. Code attributes can help trace back an operation in a trace to the code that generated it.

> [!NOTE]
>
> This Instrumentation Rule is currently only available with the Odigos **Enterprise** plan.\\
>
> [Contact us](https://odigos.io/) for more information.


# Custom Instrumentation
Source: https://docs.odigos.io/oss/pipeline/rules/custominstrumentation

The "Custom Instrumentation" rule can be used to instrument specific functions in PHP applications, or (with Odigos Enterprise) arbitrary Go/Java functions via eBPF.

## Considerations

In Odigos open source, Custom Instrumentation is supported for **PHP** applications using the community native OpenTelemetry distro.

Custom eBPF-based instrumentations for Go and Java require [Odigos Enterprise](https://odigos.io/pricing).

PHP probes are delivered as environment variables and require a pod restart to take effect.

## Configuration Options

> **PHP**
>
> > **className**
> >
> > **className** `string` - The fully qualified class name to instrument.
> > Leave empty for a global function. - This field is *optional*
>
> > **functionName**
> >
> > **functionName** `string` - The name of the function or method to
> > instrument. - This field is *required*

## Basic Example

The following example instruments the `processOrder` method on `App\Service\OrderService`.

**Step 1**

Create a YAML file named `custom-instrumentation.yaml` with the following content:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
    name: custom-instrumentation-rule
    namespace: odigos-system
spec:
    ruleName: "Custom Instrumentation for processOrder"
    customInstrumentations:
        php:
          - className: "App\\Service\\OrderService"
            functionName: "processOrder"
```

**Step 2**

Apply the rule to the cluster:

```shell theme={null}
kubectl apply -f custom-instrumentation.yaml
```

**Step 3**

Restart the PHP workload so the new `ODIGOS_PHP_AGENT_CUSTOM_INSTRUMENTATIONS` environment variable is picked up.


# Headers Collection
Source: https://docs.odigos.io/oss/pipeline/rules/headerscollection

The "Headers Collection" rule can be used to add span attributes by collecting HTTP Headers data.

> [!NOTE]
>
> This Instrumentation Rule is currently only available with the Odigos **Enterprise** plan.\\
>
> [Contact us](https://odigos.io/) for more information.


# Introduction
Source: https://docs.odigos.io/oss/pipeline/rules/introduction

Instrumentation Rules control how telemetry is recorded from your application. A rule can be applied to a set of Kubernetes workloads and instrumentation libraries.

## Rule Types:

* [Code Attributes](./codeattributes)
* [Headers Collection](./headerscollection)
* [Payload Collection](./payloadcollection)
* [Custom Instrumentations](./custominstrumentation)
* [Network Metrics](./networkmetrics)

## Configuration Options

> **ruleName**
>
> **ruleName** `string` : Allows you to attach a meaningful name to the rule for convenience and documentation.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **notes**
>
> **notes** `string` : Allows you to attach notes regarding the rule for convenience.
>
> * This field is *optional*
> * &#x20;Odigos does not use or assume any meaning from this field

> **disabled**
>
> **disabled** `boolean` : Allows you to temporarily disable the rule, but keep it saved for future use.
>
> * This field is *optional*, and defaults to `false`

> **workloads**
>
> **workloads** `[namespace, name, kind]` : Allows you to apply/limit the rule to specific workloads. If not specified, the rule will be applied to all workloads.
>
> * This field is *optional*, and defaults to `nil` (all workloads)
>
> * &#x20;Empty array will make the rule ineffective
>
> > **namespace**
> >
> > **namespace** `string` - The name of the Kubernetes namespace.
> >
> > * This field is *required*
>
> > **name**
> >
> > **name** `string` - The name of the Kubernetes workload.
> >
> > * This field is *required*
>
> > **kind**
> >
> > **kind** `string` - The kind of the Kubernetes workload.
> >
> > * This field is *required*
> > * Supported values: `Deployment`, `DaemonSet`, `StatefulSet`

> **instrumentationLibraries**
>
> **instrumentationLibraries** `[name, language, spanKind]` : Allows you to apply/limit the rule to specific instrumentation libraries. If not specified, the rule will be applied to all instrumentation libraries.
>
> * This field is *optional*, and defaults to `nil` (all libraries)
>
> * &#x20;Empty array will make the rule ineffective
>
> > **name**
> >
> > **name** `string` - The unique name of the instrumentation library.
> >
> > * This field is *required*
>
> > **language**
> >
> > **language** `string` - The programming language of the instrumentation library.
> >
> > * This field is *required*
> > * Alphabetical, lowercase, and no spaces
>
> > **spanKind**
> >
> > **spanKind** `string` - The span kind of the instrumentation library.
> >
> > * This field is *required*
> > * Supported values: `client`, `server`, `producer`, `consumer`, `internal`

## Apply to Sources

Odigos allows you to apply/limit rules to specific sources.

* To apply a rule to all sources, omit the `workloads` field.
* To apply a rule to specific sources, provide an array of workload objects (identified by its namespace, name, and kind).

> [!NOTE]
>
> Odigos allows you to define multiple rules, and will merge them together to create a single rule per source. Conflicting options will be resolved by choosing to record the attribute, if it's specified by one rule (at least).

## Apply to Instrumentation Libraries

Odigos allows you to apply/limit rules to specific instrumentation libraries.

* To apply a rule to all instrumentation libraries, omit the `instrumentationLibraries` field.
* To apply a rule to specific instrumentation libraries, provide an array of instrumentation libraries (identified by its unique name, language, and span kind for golang).

> [!NOTE]
>
> Any rule for a specific instrumentation library will take precedence over a rule that does not specify a library.

> [!TIP]
>
> This is an advanced feature. It is recommended to set the same rules for all instrumentation libraries, and only use this feature when you need fine-grained control.


# Network Metrics
Source: https://docs.odigos.io/oss/pipeline/rules/networkmetrics

Enable network flow and TCP stats metrics for instrumented workloads via InstrumentationRule

The **Network Metrics** rule enables network flow metrics and TCP stats metrics for matching workloads. These metrics are collected in odiglet and exported over OTLP alongside other workload telemetry.

Learn more about the underlying collectors and metric families in the [OBI instrumentation guide](/oss/instrumentations/obi).

## Considerations

> [!WARNING]
>
> Before enabling **network metrics**, please note the following:
>
> * Enable the cluster infrastructure in Helm by setting `metricsSources.networkMetrics.enabled` to `true` (it defaults to disabled).

## Configuration Options

> **networkMetrics**
>
> **networkMetrics** `object` : Enable network flow and TCP stats metrics for scoped workloads. Setting this field (even as an empty object `{}`) enables collection; omit it to leave metrics disabled.

> [!NOTE]
>
> Multiple rules merge with OR semantics: if any matching rule enables network metrics, they are enabled for that source/language.

## Cluster infrastructure for network flow metrics

Network flow metrics require odiglet to run with [`hostNetwork`](https://kubernetes.io/docs/reference/kubernetes-api/core/pod-v1/#:~:text=%60hostNetwork%60), which makes the pod share the host's network namespace instead of getting its own. Configure this once at the cluster level in Helm:

```yaml theme={null}
metricsSources:
  networkMetrics:
    enabled: true
```

The InstrumentationRule alone is not enough for network flow metrics without that Helm setting.

## Basic Example

> [!NOTE]
>
> Create `InstrumentationRule` resources in the Odigos installation namespace (typically `odigos-system`). Rules in other namespaces are not picked up by Odigos.

The following example enables network metrics cluster-wide for all workloads.

**1. Enable network metrics infrastructure**

Set `metricsSources.networkMetrics.enabled` to `true` in your Odigos Helm values and upgrade the release.

**2. Create the InstrumentationRule**

Create a YAML file with the following content:

```yaml network-metrics.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: network-metrics
  namespace: odigos-system
spec:
  ruleName: "Network metrics"
  networkMetrics: {}
```

**3. Apply the rule**

```shell theme={null}
kubectl apply -f network-metrics.yaml
```

Network metrics collection will run on any new pod created after the rule is applied. Existing pods will not be affected and require a restart to initiate metrics collection.

## Scoped Example

To limit the rule to a single workload, add `scopes`. Omit `scopes` (or leave it empty) to apply the rule cluster-wide.

```yaml network-metrics-scoped.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationRule
metadata:
  name: network-metrics-my-app
  namespace: odigos-system
spec:
  ruleName: "Network metrics for my-app"
  scopes:
    sources:
      - name: my-app
        namespace: default
        kind: Deployment
  networkMetrics: {}
```

Apply with:

```shell theme={null}
kubectl apply -f network-metrics-scoped.yaml
```

## Exported Metrics

When this rule is enabled, the following metrics are exported for matching workloads:

| Name (OpenTelemetry) | Name (Prometheus) | Type | Unit | Description |
| - | - | - | - | - |
| `odigos.network.flow.bytes` | `odigos_network_flow_bytes_total` | Counter | bytes | Bytes sent from a source network endpoint to a destination network endpoint |
| `odigos.network.flow.packets` | `odigos_network_flow_packets_total` | Counter | | Packets sent from a source network endpoint to a destination network endpoint |
| `odigos.network.inter.zone.bytes` | `odigos_network_inter_zone_bytes_total` | Counter | bytes | Bytes flowing between cloud availability zones in your cluster (Kubernetes only) |
| `odigos.stat.tcp.rtt` | `odigos_stat_tcp_rtt_seconds` | Histogram | seconds | TCP round-trip time (RTT) latency observed between network endpoints |
| `odigos.stat.tcp.failed.connections` | `odigos_stat_tcp_failed_connections_total` | Counter | 1 | Failed TCP connection attempts between endpoints, labeled by failure reason |

> [!NOTE]
>
> These metrics are collected using [OBI](https://opentelemetry.io/docs/zero-code/obi). See the [OBI instrumentation guide](/oss/instrumentations/obi) to learn more about the collectors, metric families, and available attributes.


# Payload Collection
Source: https://docs.odigos.io/oss/pipeline/rules/payloadcollection

The "Payload Collection" rule can be used to add span attributes by collecting payload data from various sources like HTTP requests, HTTP responses, database queries, and messaging systems.

> [!NOTE]
>
> This Instrumentation Rule is currently only available with the Odigos **Enterprise** plan.\\
>
> [Contact us](https://odigos.io/) for more information.


# Configuration
Source: https://docs.odigos.io/oss/pipeline/sampling/configuration

Configure sampling actions in Odigos Open Source.

You can configure sampling rules in the **UI** or as **YAML** (Kubernetes **CRDs**). The schema matches the same concepts as the UI, so GitOps workflows can express the same behavior as in the product.

Every rule has three parts: a **Source scope** that defines which workloads the rule applies to, an **Operation** that picks the specific operations within that scope, and a **Keep Percentage** that controls how much matching traffic is retained. The Source scope options are identical across all three categories; the Operation and Keep Percentage options differ by category. Each rule can also carry an optional **Name** and **Notes** for easier identification.

> [!TIP]
>
> This page is the **field reference**. For end-to-end walkthroughs—UI steps and the equivalent YAML side by side—see [Examples](./examples).

## Source scope

A rule can be scoped to one of:

* **Entire cluster** — the rule applies to every source in the cluster.
* **Specific Source** — one or more named sources (services).
* **Specific Namespace** — one or more Kubernetes namespaces.
* **Specific Programming Language** — one or more runtimes: **Python**, **Java**, **NodeJS**, **.Net**, **PHP**, **Ruby**, or **Go**.

You can select multiple values within a scope type—for example several namespaces, or several languages.

> [!WARNING]
>
> When you populate more than one scope type on the same rule, the selections are combined with **AND**, not **OR**. A rule scoped to "Specific Namespace = `prod`" plus "Specific Programming Language = Java" matches **only** Java workloads in the `prod` namespace; it does not match every workload in `prod` or every Java workload across the cluster.

## Operation

The Operation options depend on the rule's category.

**Noisy Operation**

* **All Operations** — every operation in scope.
* **HTTP Server** — match HTTP requests received by a server. Configure:
  * An **HTTP route** (exact path) or **HTTP route prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **HTTP Client** — match outbound HTTP calls made by the source. Configure:
  * A **server address** (the remote host being called).
  * A **templated path** (exact) or **templated path prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **gRPC Server** — match incoming gRPC calls received by a server. Configure:
  * A **service** — the fully-qualified gRPC service name (e.g. `acme.inventory.v1.InventoryService`). Set only the service to match every method on that service.
  * A **method** — the bare method name (e.g. `ListItems`). Combine with **service** for a single specific RPC.
* **gRPC Client** — match outbound gRPC calls made by the source. Configure:
  * A **service** — the fully-qualified gRPC service name (e.g. `acme.inventory.v1.InventoryService`). Set only the service to match every method on that service.
  * A **method** — the bare method name (e.g. `ListItems`). Combine with **service** for a single specific RPC.
  * A **server address** (the remote host being called).

**Highly Relevant**

* **All Operations** — every operation in scope.
* **HTTP Server** — match HTTP requests received by a server. Configure:
  * An **HTTP route** (exact path) or **HTTP route prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **Kafka Consumer** — match traces from a Kafka consumer. Configure the **Kafka topic**.
* **Kafka Producer** — match traces from a Kafka producer. Configure the **Kafka topic**.

HTTP Client is **not** available in this category.

**Cost Reduction**

* **All Operations** — every operation in scope.
* **HTTP Server** — match HTTP requests received by a server. Configure:
  * An **HTTP route** (exact path) or **HTTP route prefix**.
  * A **method**: **Any**, **GET**, **POST**, **PATCH**, **DELETE**, or **PUT**.
* **Kafka Consumer** — match traces from a Kafka consumer. Configure the **Kafka topic**.
* **Kafka Producer** — match traces from a Kafka producer. Configure the **Kafka topic**.

HTTP Client is **not** available in this category.

## Keep Percentage

Each rule defines what fraction of matching traffic is retained. The available options differ by category:

| Category | Options |
| - | - |
| Noisy Operation | **Drop All**, or **Keep at most** *N*% |
| Highly Relevant | **Keep All**, or **Keep at least** *N*% |
| Cost Reduction | **Keep at most** *N*% (set *N* = 0 to drop everything) |

**Keep at most** is a cap; **Keep at least** is a floor. See [Rule evaluation](./rules/overview#rule-evaluation) for how these percentages interact within a category and between Highly Relevant and Cost Reduction.

## Enabled

Every rule has an **Enabled** toggle that controls whether it participates in sampling decisions. Disabled rules stay in your configuration but are excluded from evaluation, which makes the toggle useful for several patterns:

* **Temporary suspension.** Turn a rule off during an incident, debugging session, or evaluation period when you need broader visibility—without losing the rule's configuration.
* **Disable vs. delete.** Disabling preserves the matcher, scope, and Keep Percentage so you can re-enable later without re-authoring. Use it when a rule may still be useful later but isn't needed right now.
* **"What-if" metrics.** Disabled rules continue to record matching metrics: how many traces *would have* matched, and how many would have been kept or dropped if the rule were active. Useful for sizing a candidate rule's impact before committing to it.

See [Disabling a rule](./examples#disabling-a-rule) for UI screenshots and the equivalent YAML.

## Name and Notes

Both fields are optional and exist purely to make rules easier to identify and maintain—they have **no effect** on evaluation or matching.

* **Name** — a short, free-text label for the rule. The name appears in the rule list in the UI and may also surface on metrics and span attributes, so use something descriptive that makes the rule recognizable in dashboards and queries.
* **Notes** — free-text describing **why** the rule exists. Use this to capture context that isn't obvious from the matcher itself: the incident that triggered the rule, the team that owns it, or the cost target it's serving. Especially useful when auditing rules or onboarding new operators.

> [!TIP]
>
> Treat Name and Notes like a commit message for your sampling policy. A good Name + Notes pair turns "what does this rule do?" into a one-line answer at a glance.


# Considerations
Source: https://docs.odigos.io/oss/pipeline/sampling/considerations

Operational and design considerations for sampling in Odigos Open Source.

Operational and design considerations to keep in mind when running sampling in production.

## Gateway load, OOM, and scaling

When the gateway accepts a trace for tail evaluation, it allocates memory to hold spans until the [aggregation window](./head-and-tail-sampling#aggregation-window) closes. If the gateway isn't sized correctly, bursts of traffic can cause **OOM** restarts. Symptoms include **dropped spans** or collector crashes that aren't caused by your sampling rules.

For tail sampling, prefer **scaling up** (bigger collector pods) over **scaling out** (more replicas). All spans of a single trace must land on the same pod for aggregation, so fewer, larger pods simplify routing and load balancing. Validate sizing against your own load—there's no one-size-fits-all.

## When any sampler enables tail mode

> [!WARNING]
>
> Any sampling rule that requires tail evaluation activates **tail mode** for the entire cluster, even rules that look inactive or minimal—for example a placeholder service name or a broad `/` route. Once tail mode is on, the gateway buffers each trace for the [aggregation window](./head-and-tail-sampling#aggregation-window), which increases memory use, export latency, and the risk of dropped spans. If you don't need tail behavior, either remove any unused or test sampling rules from the cluster, or disable tail sampling explicitly in Odigos configuration (Helm values, or the UI **Settings** page).


# Examples
Source: https://docs.odigos.io/oss/pipeline/sampling/examples

End-to-end walkthroughs of common sampling rules in Odigos Open Source, with UI steps and equivalent YAML.

End-to-end walkthroughs of common sampling rules. Each example shows the rule in the **UI** and as a **YAML** `Sampling` resource (the same CRD applied via `kubectl apply` or GitOps). Use these as starting points and adjust scope, operations, and percentages to match your environment.

> [!NOTE]
>
> YAML examples are accurate against the current `Sampling` CRD schema. The screenshots for each scenario reflect the live UI; minor styling may shift between releases.

For the field reference behind each example—every Source scope option, every Operation option, every Keep Percentage choice—see [Configuration](./configuration). For the rules engine itself, see [Rules](./rules/overview).

> [!NOTE]
>
> **YAML examples on this page** conform to the `Sampling` CRD (`odigos.io/v1alpha1`). For every field, type, validation rule, and default—including options not covered by the examples below—see the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling).

> [!TIP]
>
> **Scope rules to only the workloads they need to evaluate.** Cluster-wide rules are checked against every trace; tightly scoped rules (Specific Source, Namespace, or Language) are skipped entirely for unrelated workloads, which avoids wasted matching work in the gateway. Several examples below illustrate this pattern.

## 1. Drop a noisy custom endpoint

**Goal:** Drop traces for a high-volume polling endpoint (`/api/v1/poll`) that produces many traces but is not useful for debugging or SLOs. This is a typical **Noisy** rule for traffic that's neither a Kubernetes probe nor a metrics scrape, but is still pure noise.

**UI**

Create a Noisy rule with:

* **Name:** `Drop polling endpoint`
* **Notes:** `High-volume poll endpoint with no SLO/debug value.`
* **Source scope:** Specific Source = `odimall / Deployment / notification-service`
* **Operation:** HTTP Server, route prefix `/api/v1/poll`, method **GET**
* **Drop Percentage:** Drop All

![UI screenshot of a Noisy rule that drops GET traces for /api/v1/poll, scoped to the notification-service Deployment in the odimall namespace.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/drop-poll-endpoint-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=102aa70d7ece6368d65564256293c7f0)

**YAML**

```yaml drop-poll-endpoint.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: drop-poll-endpoint
  namespace: odigos-system
spec:
  name: Drop /api/v1/poll
  noisyOperations:
    - name: Drop polling endpoint
      notes: High-volume poll endpoint with no SLO/debug value.
      percentageAtMost: 0
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: notification-service
      operation:
        httpServer:
          routePrefix: /api/v1/poll
          method: GET
```

## 2. Keep slow requests at 100%

**Goal:** Always retain traces for `GET /api/products` whenever they take longer than 500 ms. This is a **Highly Relevant** duration rule—a complement to the built-in [Keep All Error Traces](./rules/highly-relevant#built-in-rule-keep-all-error-traces) rule, focused on latency rather than errors.

**UI**

Create a Highly Relevant rule with:

* **Name:** `Slow GET /api/products`
* **Notes:** `SLO-relevant path; retain anything over 500ms for investigation.`
* **Rule Type:** Duration
* **Source scope:** Specific Source = `odimall / Deployment / frontend`
* **Operation:** HTTP Server, route `/api/products`, method **GET**
* **Keep Percentage:** Keep All
* **Duration:** 500ms

![UI screenshot of a Highly Relevant rule that keeps 100% of slow GET /api/products traces, scoped to the frontend Deployment in the odimall namespace.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/keep-slow-products-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=a20b0f51a16027c4a32791d855a3ff0e)

**YAML**

```yaml keep-slow-products.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: keep-slow-products
  namespace: odigos-system
spec:
  name: Keep slow product reads
  highlyRelevantOperations:
    - name: Slow GET /api/products
      notes: SLO-relevant path; retain anything over 500ms for investigation.
      durationAtLeastMs: 500
      percentageAtLeast: 100
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: frontend
      operation:
        httpServer:
          route: /api/products
          method: GET
```

## 3. Apply a cluster-wide 5% baseline

**Goal:** Keep 5% of all cluster traffic as a statistical baseline, regardless of source or operation. This is the same shape as the built-in [Drop Traces Cluster-Wide](./rules/cost-reduction#built-in-rule-drop-traces-cluster-wide) rule and is the simplest way to cut volume after Noisy and Highly Relevant rules have run.

**UI**

Create a Cost Reduction rule with:

* **Name:** `5% baseline`
* **Notes:** `Statistical baseline for non-noisy, non-important traffic.`
* **Source scope:** Entire cluster
* **Operation:** All Operations
* **Drop Percentage:** 5

![UI screenshot of a Cost Reduction rule that caps cluster-wide retention at 5%.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/baseline-cluster-sample-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=50cc53df3883927c0a1702c856c02fe8)

**YAML**

```yaml baseline-cluster-sample.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: baseline-cluster-sample
  namespace: odigos-system
spec:
  name: Cluster baseline 5%
  costReductionRules:
    - name: 5% baseline
      notes: Statistical baseline for non-noisy, non-important traffic.
      percentageAtMost: 5
```

> [!TIP]
>
> Cost Reduction rules with **no** Operation block apply to every operation in the scope. Combine with Highly Relevant rules to make sure error and latency traces survive the cap—Highly Relevant always wins over Cost Reduction.

## 4. Scope a rule to namespace and language

**Goal:** Drop scrape traffic to `/internal/metrics` for **Java** workloads in the `prod` namespace only—leave other namespaces and other languages untouched. This example demonstrates the **AND** semantics across scope types: a single source scope item with both `workloadNamespace` and `workloadLanguage` set.

**UI**

Create a Noisy rule with:

* **Name:** `Java metrics scrape in prod`
* **Notes:** `Internal Prometheus scrape endpoint; the scrape itself isn't useful for tracing.`
* **Source scope:** Specific Namespace = `prod` **and** Specific Programming Language = **Java**
* **Operation:** HTTP Server, route `/internal/metrics`, method **GET**
* **Drop Percentage:** Drop All

![UI screenshot of a Noisy rule scoped with namespace prod AND language Java.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/drop-metrics-java-prod-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=275e5a1e330c052c4c75d27895afbe88)

**YAML**

```yaml drop-metrics-java-prod.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: drop-metrics-java-prod
  namespace: odigos-system
spec:
  name: Drop /internal/metrics (Java in prod)
  noisyOperations:
    - name: Java metrics scrape in prod
      notes: Internal Prometheus scrape endpoint; the scrape itself isn't useful for tracing.
      percentageAtMost: 0
      sourceScopes:
        - workloadNamespace: prod
          workloadLanguage: java
      operation:
        httpServer:
          route: /internal/metrics
          method: GET
```

> [!NOTE]
>
> Source scope combines in two layers:
>
> * **Different categories must all match.** This rule sets Specific Namespace = `prod` together with Specific Programming Language = `java`, so a workload has to be in `prod` **and** be a Java workload. In YAML that maps to a single `sourceScopes` item with both fields set on it.
> * **Multiple values inside one category are alternatives.** Setting the namespace category to `prod` and `staging` would match a Java workload in either one. In YAML that maps to multiple `sourceScopes` items, each carrying just the namespace field.

## 5. Combine Highly Relevant and Cost Reduction

**Goal:** A `payment-service` where **every error trace** must be kept (Highly Relevant) and **all other traffic** is capped at 1% (Cost Reduction). This is the canonical "keep what matters, sample the rest" pattern, expressed in a single `Sampling` object that bundles both categories together.

**UI**

Create two rules:

**1. Highly Relevant**

* **Name:** `Keep payment-service errors`
* **Notes:** `Errors from payment-service are always investigated; never sample them away.`
* **Rule Type:** Error
* **Source scope:** Specific Source = `odimall / Deployment / payment-service`
* **Operation:** All Operations
* **Keep Percentage:** Keep All

![UI screenshot of a Highly Relevant rule with Rule Type Error scoped to payment-service, Keep at least 100%.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/payment-service-policy-relevant-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=f4352d98a24c24594c9249eb60ceb281)

**2. Cost Reduction**

* **Name:** `payment-service 1% cap`
* **Notes:** `Cap healthy payment-service traffic to keep cost in check; HR errors above always win.`
* **Source scope:** Specific Source = `odimall / Deployment / payment-service`
* **Operation:** All Operations
* **Drop Percentage:** 1%

![UI screenshot of a Cost Reduction rule on payment-service with Keep at most 1%.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/payment-service-policy-cost-example.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=4948667e517c18b33fe0bf68c0522ad5)

**YAML**

```yaml payment-service-policy.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: payment-service-policy
  namespace: odigos-system
spec:
  name: payment-service policy
  notes: Always keep errors; cap healthy traffic at 1%.
  highlyRelevantOperations:
    - name: Keep payment-service errors
      notes: Errors from payment-service are always investigated; never sample them away.
      error: true
      percentageAtLeast: 100
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: payment-service
  costReductionRules:
    - name: payment-service 1% cap
      notes: Cap healthy payment-service traffic to keep cost in check; HR errors above always win.
      percentageAtMost: 1
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: payment-service
```

> [!NOTE]
>
> A trace from `payment-service` that contains any error span is kept at 100% by the Highly Relevant rule. A trace with no errors falls through to the Cost Reduction cap of 1%. Both rules are evaluated for every matching trace; Highly Relevant's keep percentage wins when both apply. See [Evaluation between Highly Relevant and Cost Reduction](./rules/overview#evaluation-between-highly-relevant-and-cost-reduction-categories).

## 6. Drop noisy gRPC health checks

**Goal:** Drop standard gRPC liveness/readiness checks (`grpc.health.v1.Health.Check`, `Watch`, ...) on the `inventory` Deployment. This is the gRPC counterpart of [Example 1](#1-drop-a-noisy-custom-endpoint), and shows how setting `service` alone scopes a rule to every method of a whole gRPC service.

```yaml drop-grpc-health-checks.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: drop-grpc-health-checks
  namespace: odigos-system
spec:
  name: Drop gRPC health checks
  noisyOperations:
    - name: Drop gRPC health checks
      notes: Standard liveness/readiness pings; not useful for tracing.
      percentageAtMost: 0
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: inventory
      operation:
        grpcServer:
          service: grpc.health.v1.Health
```

> [!TIP]
>
> Because `service` and `method` are independent, a single `service:` line filters every method on a service. Add `method: ListItems` to scope to one specific RPC instead, or combine with `grpcClient.serverAddress` to drop outgoing calls to a particular backend.

## YAML reference notes

A few details that aren't obvious from YAML alone but matter when authoring rules. The full schema—every field, type, and validation rule—is in the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling).

> **Source scope: language enum and workloadKind casing**
>
> The YAML enum for `workloadLanguage` uses lowercase identifiers: `java`, `python`, `go`, `dotnet`, `javascript`, `php`, `ruby`. The UI displays the friendly names (Java, Python, Go, .Net, NodeJS, PHP, Ruby).
>
> `workloadKind` must match the Kubernetes resource kind exactly (case-sensitive PascalCase): `Deployment`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob`. Lowercase or pluralized values will silently fail to match.

> **Defaults when fields are omitted**
>
> Empty or missing fields don't mean "match nothing"—each one has a default that often biases toward broader matching:
>
> * **`sourceScopes` empty or absent** — the rule matches **every** source.
> * **`operation` omitted** — the rule matches **every** operation in the scope.
> * **Noisy `percentageAtMost` omitted** — defaults to **0%** (drops everything).
> * **Highly Relevant `percentageAtLeast` omitted** — defaults to **100%** (keeps everything).
> * **Cost Reduction `percentageAtMost`** — **required**. There is no default; the resource will fail validation if the field is absent.

> **HTTP matchers: route, method, and templated paths**
>
> * On HTTP Server matchers, set **either** `route` (exact match) **or** `routePrefix` (prefix match)—not both. Both match the framework's **templated path** (e.g., `/users/{id}`, `/users/:id`, `/users/*`), not the resolved URL.
> * `method` is a free-form string in YAML. The UI restricts it to GET, POST, PATCH, DELETE, and PUT, but YAML accepts any method (e.g., HEAD, OPTIONS). Leave it empty to match any method.
> * HTTP Client matchers (Noisy only) use `templatedPath` / `templatedPathPrefix`—the framework's parameterized path (e.g., `/users/{id}`), **not** the resolved URL (e.g., `/users/42`). Match against the route as your HTTP library reports it.

> **gRPC matchers: service and method are independent fields**
>
> * `service` matches the fully-qualified gRPC service name (e.g., `acme.inventory.v1.InventoryService`, `grpc.health.v1.Health`); `method` matches the bare method name (e.g., `ListItems`, `Check`). Both fields are **exact match** and **case-sensitive**, mirroring proto naming.
> * The canonical "drop every call to this service" rule is one line: set `service` and leave `method` empty. To target a single RPC, set both. To match every gRPC call regardless of service, leave both empty.
> * Both OpenTelemetry semconv conventions are supported transparently: the older split form (`rpc.service` + bare `rpc.method`) and the newer fully-qualified form (`rpc.method` = `"Service/method"`, with `rpc.service` deprecated). For the latter the matcher splits `rpc.method` on the first `/` to derive the service/method, so the same rule matches spans from agents on either version.
> * The `rpc.*` attribute namespace is shared with other RPC frameworks (Apache Dubbo, Connect RPC, JSON-RPC, .NET WCF, Java RMI, ONC RPC). The matcher reads `rpc.system` / `rpc.system.name` and, when present, only matches spans whose system equals `"grpc"`; non-gRPC RPC spans never accidentally fire a gRPC rule. Spans that omit `rpc.system` entirely are still considered (permissive default for older instrumentations).
> * A span must carry at least `rpc.service` **or** `rpc.method` to be considered a gRPC span; rules on an attribute that can't be derived from the span will not match (e.g. a `service:` rule against a span whose only RPC attribute is the bare-method `rpc.method` or the `_OTHER` sentinel).
> * gRPC Client matchers (Noisy only) additionally support `serverAddress` for matching the remote host (the gRPC channel target). Leave it empty to match any address.

> **Kafka matchers: empty topic matches all topics**
>
> Leaving `kafkaTopic` empty on a `kafkaConsumer` or `kafkaProducer` matcher matches **every** topic for that direction. Set the topic explicitly to scope a Kafka rule to a specific stream.

> **Highly Relevant: combining `error` and `durationAtLeastMs`**
>
> Setting both `error: true` and `durationAtLeastMs` on a single `highlyRelevantOperations` item is an **AND**—the span must be an error **and** longer than the threshold to match. For "errors **or** slow," use two separate items in the `highlyRelevantOperations` array.

> **`name` and `disabled` surface in observability data**
>
> The rule `name` is written as a **metric attribute** and stored as a **span attribute** on participating spans, so use a value that's safe for dashboards and queries—avoid PII or characters your backend doesn't index well.
>
> `disabled: true` on a rule removes it from sampling decisions but **still emits metrics** for it. That's useful for tuning a rule and observing its would-be impact before re-enabling it, rather than deleting it outright.

## Applying YAML rules

You can apply any of the YAML examples above with `kubectl`, or commit them to your GitOps repository alongside the rest of your Odigos configuration:

```bash theme={null}
kubectl apply -f drop-poll-endpoint.yaml
```

Each `Sampling` resource is a self-contained group of rules. You can split rules across multiple `Sampling` objects—by team, environment, or use case—and they will be **joined together** to form the global sampling policy.

```bash theme={null}
kubectl get samplings.odigos.io -n odigos-system
```

### Disabling a rule

To take a rule out of sampling decisions without deleting it, you can either flip the toggle in the UI or set `disabled: true` in YAML. Disabled rules **still emit sampling metrics**, so you can observe what the rule would have done before re-enabling it—useful as a "soft delete" while tuning, rather than removing the rule outright. See [Enabled](./configuration#enabled) for when to reach for this versus deleting the rule.

**UI**

Each rule's detail view has a toggle in the top-right corner that flips its label between **Rule Enabled** and **Rule Disabled**. Slide it off to take the rule out of sampling decisions immediately; slide it back on to re-enable. The toggle is **per rule**—other rules in the same `Sampling` object continue to apply.

![UI screenshot of the Rule Enabled toggle in the on (green) position.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/ui-enable-rule.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=4e5f6ce5034b996a1d932fd2183ded88)

![UI screenshot of the Rule Disabled toggle in the off (red) position.](https://mintcdn.com/odigos/Ni37DdWA1VTwcWpK/images/sampling/ui-disable-rule.png?fit=max\&auto=format\&n=Ni37DdWA1VTwcWpK\&q=85\&s=8e0623bb7fee782f80f0473c69046805)

**YAML**

Set `disabled: true` on an individual entry in `noisyOperations`, `highlyRelevantOperations`, or `costReductionRules`. The other rules in the same `Sampling` object continue to apply. This is what the UI toggle writes.

```yaml disable-a-rule.yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Sampling
metadata:
  name: payment-service-policy
  namespace: odigos-system
spec:
  name: payment-service policy
  costReductionRules:
    - name: payment-service 1% cap
      notes: Cap healthy payment-service traffic to keep cost in check; HR errors above always win.
      disabled: true
      percentageAtMost: 1
      sourceScopes:
        - workloadNamespace: odimall
          workloadKind: Deployment
          workloadName: payment-service
```

> [!TIP]
>
> The examples on this page cover the most common shapes; the [`Sampling` API reference](../../../api-reference/odigos.io.v1alpha1#odigos-io-v1alpha1-Sampling) is the source of truth for every field, type, validation rule, and default on the CRD. Use it when you're extending an example or building a rule that doesn't fit one of the patterns above.


# FAQs
Source: https://docs.odigos.io/oss/pipeline/sampling/faqs

Frequently asked questions about sampling in Odigos Open Source.

Frequently asked questions about sampling in Odigos.

## Concepts and terminology

> **Does 'sampling' mean dropping traces or keeping them?**
>
> Both, depending on who you ask. Some teams describe sampling as **what gets dropped** (e.g. "drop 90%"), others as **what gets kept** (e.g. "keep 10%"). Odigos uses the **keep** framing throughout the UI and docs because it aligns with how rules are written (each rule has a **Keep Percentage**). The two phrasings describe the same outcome; we standardize on "keep" to avoid confusion. See [Introduction to sampling](./introduction#ambiguity-of-terminology).

> **Why is dropping health checks called 'sampling' if I'm dropping 100% of them?**
>
> "Sampling" is the name of the **pipeline mechanism** that makes a per-trace keep/drop decision—not a description of "always keep some, always drop some." Dropping all health probes is just sampling at 0%. The same engine that drops 100% of probe traffic also retains 25% of `/api/products` traffic; both are sampling decisions, just at different keep percentages.

> **Why does Odigos always keep or drop the entire trace?**
>
> Odigos applies a single keep/drop decision per trace, not per span. With **head sampling**, the decision is made at the root span and propagated to every child span (parent-based sampling). With **tail sampling**, the Odigos Gateway buffers all spans for a trace ID and makes one decision after the [aggregation window](./head-and-tail-sampling#aggregation-window). In both cases, splitting a trace—keeping some spans and dropping others—would break parent-child links, end-to-end timing, and any analysis that depends on the full request path, so a kept trace is always complete from root through children. See [Sampling level](./introduction#sampling-level).

## How rules are evaluated

> **If a trace matches rules in more than one category, which category wins?**
>
> Categories are evaluated in a fixed sequence: **Noisy → Highly Relevant → Cost Reduction**. A **Noisy** match short-circuits the rest—the trace is dropped (or sampled down) and the other categories are skipped. A **Highly Relevant** match does **not** short-circuit—Cost Reduction rules are still evaluated, but the Highly Relevant Keep Percentage wins. Cost Reduction caps only take effect on traces with no Highly Relevant match. See [Order of operations between categories](./rules/overview#order-of-operations-between-categories).

> **Within a category, do rules have a priority order?**
>
> No. Within a single category, rules have **no hierarchy**—they're all equal. When a trace matches multiple rules in the same category, the engine satisfies all matches at once: the **highest** Keep Percentage wins for **Highly Relevant** (floors, "keep at least"), and the **lowest** Keep Percentage wins for **Noisy Operation** and **Cost Reduction** (caps, "keep at most"). This is a deliberate design choice: ordered rule lists (like firewall rules) become unmaintainable when you have dozens of rules and have to mentally simulate the order to predict behavior.

> **Why does 'Keep at most 25%' not always mean exactly 25%?**
>
> Cost Reduction percentages are **caps**, not exact rates, because the engine considers **all** matching rules together. If two Cost Reduction rules match the same trace—one capped at 25% and another at 1%—the lower cap (1%) wins. This makes it safe to add new rules without worrying that they'll silently override existing ones. The same logic runs in reverse for Highly Relevant: percentages there are **floors** ("keep at least"), and the highest floor wins.

> **If I only create Highly Relevant rules and no Cost Reduction rules, what gets dropped?**
>
> Almost nothing—and that's usually not what you want. Highly Relevant rules mark important traces, but they don't reduce volume on their own; non-matching traces still flow through the pipeline. Worse, the gateway still buffers traces for tail evaluation, which **adds load without cutting volume**. To control volume, pair Highly Relevant rules with at least one Cost Reduction rule (commonly the built-in Drop Traces Cluster-Wide rule). See [Highly Relevant](./rules/highly-relevant#interaction-with-cost-reduction).

> **Can I combine multiple scope types (e.g. namespace + language) in a single rule?**
>
> Yes. Within a single scope type, multiple selections are combined with **OR** (e.g. namespaces `prod` OR `staging`). Across different scope types, selections are combined with **AND** (e.g. namespace `prod` AND language Java matches **only** Java workloads in `prod`—not every workload in `prod` and not every Java workload across the cluster). See [Source scope](./configuration#source-scope).

## Operational impact

> **Does adding any sampling rule increase load on the gateway?**
>
> It depends on the rule. **Noisy** rules use head sampling and run in the instrumentation agent where supported—they reduce gateway load. **Highly Relevant**, **Cost Reduction** and **Noisy** rules in agents with no head-sampling support, require tail evaluation, which means the gateway buffers each affected trace for the [aggregation window](./head-and-tail-sampling#aggregation-window). Even a single tail-mode rule activates buffering for all traffic, raising memory and CPU on the gateway.

> **Why do gateways sometimes crash instead of just dropping data?**
>
> Once the gateway accepts a trace for tail evaluation, it allocates memory to hold spans until the aggregation window closes. During traffic bursts, many traces can be accepted concurrently before backpressure kicks in—each one keeps allocating memory until the pod hits its limit and OOMs. This is why bursty workloads can produce dropped spans or collector restarts even when your sampling rules look correct. Sizing the gateway for peak burst capacity (not steady-state) is the most reliable fix. See [Gateway load, OOM, and scaling](./considerations#gateway-load-oom-and-scaling).

> **Should I scale up (bigger pods) or scale out (more replicas) for tail sampling?**
>
> Both arithmetically yield similar capacity, but **scaling up is usually preferable for tail sampling**. All spans of a single trace must land on the same gateway pod for aggregation, so each scale event re-shuffles the trace-ID hash routing. Fewer, larger pods scale less often and produce less routing churn. There's no one-size-fits-all answer—validate against your own load.

> **What happens to in-flight traces if a gateway crashes?**
>
> They are lost. Odigos does not currently use persistent queues or full backpressure between the data collector and the gateway, so a crash drops everything the gateway was buffering. **Graceful** shutdowns (rolling updates, scale-downs) flush their buffers before exit; only ungraceful crashes lose data.

> **My sampler looks empty or inactive—why is it still affecting my pipeline?**
>
> The presence of **any** sampling rule that requires tail evaluation activates tail mode for matching traffic—even rules that look minimal, such as a placeholder service name or a broad `/` route. Once tail mode is on, the gateway buffers traces for the aggregation window, which raises memory and latency to export. If you don't need tail behavior, either remove unused or test sampling rules from the cluster, or disable tail sampling explicitly in Odigos configuration (Helm values, or the UI **Settings** page). See [When any sampler enables tail mode](./considerations#when-any-sampler-enables-tail-mode).

## Tail sampling specifics

> **Why is the aggregation window 30 seconds, and can I change it?**
>
> There is no reliable signal for "this trace is finished," so the engine uses a fixed timeout instead. **30 seconds** is the Odigos default, matching the OpenTelemetry community default. You can change it through the Helm value `sampling.tailSampling.traceAggregationWaitDuration` (for example `'45s'` or `'15s'`); the same value applies to every trace in the cluster. Larger windows hold more memory and delay traces appearing in your backend; smaller windows risk evaluating partial traces and fragmenting a single trace across multiple sampling decisions. See [Aggregation window](./head-and-tail-sampling#aggregation-window).

> **Does the 30-second timer start at the first span or the last span of a trace?**
>
> The **first** span the gateway sees for a trace ID, following the OpenTelemetry tail-sampling standard. Practical consequence: requests longer than 30 seconds may be evaluated on a partial trace.

> **What is trace-based load balancing, and why does tail sampling need it?**
>
> Tail evaluation requires every span of a single trace to land on the same gateway pod—otherwise the gateway can't aggregate the trace before deciding. Odigos's data collectors use a **trace-based load balancing exporter** that hashes the trace ID and routes all spans for that trace to the same gateway. Plain round-robin (the default without tail sampling) would scatter spans across pods and break aggregation.

## Built-in rules

> **What exactly does the built-in Kubernetes Health Probes rule cover?**
>
> It drops traces for the **liveness**, **readiness**, and **startup** probe paths discovered on each workload's Kubernetes manifest. The rule's Keep Percentage defaults to **0%** (drop everything) and is configurable. It only matches **HTTPGet** probes—`exec` and `tcpSocket` probes, or probes routed through ingress rewrites or service-mesh wrappers, are not auto-handled and need an explicit Noisy rule. See [Built-in rule: Kubernetes Health Probes](./rules/noisy-operation#built-in-rule-kubernetes-health-probes).

> **Can I disable or change the built-in 'Keep All Error Traces' rule?**
>
> The built-in rule itself is fixed at 100% retention and is not configurable. If you need a different keep percentage for error traces—for example because your error volume is too high to retain all of it—author your own Highly Relevant rule on errors with the Keep Percentage you want. Custom rules are how you express any intent other than the built-in 100%. See [Built-in rule: Keep All Error Traces](./rules/highly-relevant#built-in-rule-keep-all-error-traces).

> **Why isn't there a built-in rule for high-latency traces?**
>
> Latency thresholds depend on the application: 200 ms is fast for one service and unacceptable for another, so there's no sensible default the product can ship. Errors are different—an error span is universally a signal worth keeping—which is why the error rule is built in but a duration rule is not. Define your own Highly Relevant duration rule with the threshold and Keep Percentage that fit your service.

> **Are there built-in operations beyond HTTP and Kafka (e.g. SQS, gRPC, database calls)?**
>
> Today's Operation options are **HTTP Server**, **HTTP Client** (Noisy only), and **Kafka Consumer / Kafka Producer** (Highly Relevant and Cost Reduction). Odigos will add support for additional protocols as users needs demand it.


# Head and tail sampling
Source: https://docs.odigos.io/oss/pipeline/sampling/head-and-tail-sampling

How head-based and tail-based trace sampling differ in Odigos and when each approach is appropriate.

## Types of sampling implementation

This page contrasts head sampling (decisions in the **agent** when a trace starts) and tail sampling (decisions at the **Odigos Gateway** after spans are brought together). For how Odigos treats whole traces and root-span decisions, see [Introduction to Sampling](./introduction).

### Head sampling (Agent)

Head sampling is the most basic and efficient form of sampling. The decision is made at the root span before child spans are processed; the trace in its entirety is either kept or dropped to match that decision.

* **Decision point** — At the root span as the trace starts, inside the supported instrumentation agents. The same keep or drop applies to the entire trace (parent-based).
* **Context** — Limited to root-level fields: service name, entry endpoint, and similar. Typical rule inputs are things like service, root span details, and specific routes (e.g. `foo/about`). You cannot base the decision on errors or latency that appear only on child spans, or at span end time (http response codes, etc)
* **Efficiency** — Spans in a dropped trace are skipped at span creation time, so almost no work is done on them: no agent processing, no serialization into eBPF maps, no handling at the Odiglet or data collection stage, and no network or I/O cost. Volume is cut before any downstream pipeline stage sees it.
* **Accuracy** — The choice is early: you do not yet know how the trace will finish, so you cannot guarantee keeping traces based on downstream failures or slowness. Percentage-based cuts behave statistically—rare one-offs may be missed, while frequent behavior still tends to show up in the retained sample over time.
* **Best for** — Noise you can recognize at the start (health checks, scrapes, probes, low-value traffic); percentage-based sampling when Odigos Gateway or destination capacity is tight; cost and volume control in environments where root-only policy is enough.

### Tail sampling (Odigos Gateway)

Tail sampling defers the keep/drop decision until spans for a trace are buffered and aggregated into a single unit the evaluator can read. The decision uses the full span set, not only the root span, so it can reflect the whole request path before anything is exported downstream. Timing, buffering, and delay are covered in [Aggregation window](#aggregation-window).

* **Decision point** — After spans for a trace are buffered and aggregated, in the Odigos Gateway.
* **Context** — Full trace: errors, duration, services and endpoints anywhere in the tree, including off the critical path. Rules can target behavior that only shows up on nested spans.
* **Efficiency** — CPU and memory requirements on the Odigos Gateway are increased compared with head sampling: buffering and per-trace evaluation need more resources. Most volume is still dropped before destinations, which can ease storage and downstream cost.
* **Accuracy** — Full-trace context lets retention match policy on errors, latency, or any span—but evaluation runs only after the [aggregation window](#aggregation-window), so traces show up later in UIs, and very long requests may be judged on a partial trace.
* **Best for** — Debugging errors and slow requests end-to-end; full-trace sampling goals; percentage-style or volume cuts after you have full-path context; policies that need services or routes off the entry span.

#### Aggregation window

The pipeline does not wait for a definitive “end of trace” event, because that would require an indefinite wait. Instead, spans are held up to a bounded window to bound memory and latency. The implementation uses a **default 30-second** interval from the first observed span for a trace ID, consistent with OpenTelemetry-style tail processing, then forwards the trace to the sampler with the spans received by then. Evaluation can therefore run on partial traces for very long requests.

> [!NOTE]
>
> The aggregation window is configurable through the Helm value `sampling.tailSampling.traceAggregationWaitDuration` (for example `'45s'` or `'2m'`). The same value applies to every trace in the cluster. Larger windows hold more memory and delay traces appearing in your backend; smaller windows risk evaluating partial traces and fragmenting a single trace across multiple sampling decisions.

> [!NOTE]
>
> Observability UIs typically show tail-sampled data delayed by the configured aggregation window (about 30 seconds by default). Export, queueing, storage, and indexing add further delay on top of that gap.

## Head vs. tail: quick comparison

| | Head | Tail (Odigos Gateway) |
| - | - | - |
| **Decision point** | Start of trace (root span) | After aggregation |
| **Context** | Root only (service, endpoint, …) | Full trace (errors, latency, any span) |
| **Efficiency** | High; low gateway load for the decision | Higher CPU and memory on the gateway |
| **Accuracy** | Early decision; percentage cuts are statistical | Full-trace policy; aggregation window tradeoffs |
| **Best for** | Noise, cost control, root-level rules | Errors, latency, nested routes, full-path rules |

## Choosing the Right Sampling Strategy: Head vs. Tail

Sampling is primarily a tool for cost and volume management. While both head and tail sampling aim to reduce the amount of data sent to your backend, they differ significantly in when the decision is made and what information is used to make it.

### Head-Based Sampling

Head-based sampling makes a decision at the very beginning of a trace—usually at the root span. This decision is then propagated to all downstream services (often called "parent-based sampling"), ensuring that either the entire trace is kept or the entire trace is dropped.

**When to use it:**

* **High-Volume Environments:** If your system generates an overwhelming number of traces, head sampling is the most efficient way to drop data early, saving CPU and memory on your application agents and reducing network traffic immediately.

* **Simple Logic:** Use it when you only need to sample based on known "head" attributes, such as the service name, the initial endpoint (e.g., /health vs. /api/orders), or a simple percentage (e.g., "keep 5% of all traffic").

* **Resource Constraints on Collectors:** Since decisions happen at the source, your collectors/gateways don't need to buffer or process data that is already marked to be dropped.

**The Trade-off:** \

It is a "blind" decision. Because the decision is made at the start, you cannot know if that specific trace will eventually encounter an error or experience high latency. You risk losing the exact "needle in the haystack" traces you might need for debugging.

> [!NOTE]
>
> OpenTelemetry formalizes this as **representativeness**: a smaller sample can mathematically represent the larger population, so percentage-based head sampling preserves *aggregate* signal even when individual traces are dropped. See [OpenTelemetry: Why sampling?](https://opentelemetry.io/docs/concepts/sampling/#why-sampling).

### Tail-Based Sampling

Tail-based sampling waits until the trace is complete (or a timeout is reached) before deciding whether to keep it. This is handled at the gateway collector level.

**When to use it:**

* **Ensuring Visibility of Errors:** If your goal is to "keep 100% of traces with errors," tail sampling is essential. You can inspect every span in the trace for an error status before making the final call.

* **Latency-Based Sampling:** Use this when you want to capture traces that exceed a specific duration (e.g., "keep any trace longer than 500ms").

* **Complex Business Rules:** When the importance of a trace depends on something that happens deep in the call stack—such as hitting a specific database or a legacy third-party API—tail sampling provides the necessary full context.

**The Trade-off:** \

It is resource-intensive. The gateway must buffer all spans for a trace (often for 30 seconds or more) to ensure it has the "full picture" before deciding. This increases memory usage at the gateway and introduces a delay before traces appear in your UI. Furthermore, it requires "trace-based load balancing" to ensure all spans for a single trace arrive at the same collector.


# Introduction to Sampling
Source: https://docs.odigos.io/oss/pipeline/sampling/introduction

Start here for trace sampling in Odigos: why sampling is used, how cost and visibility are balanced, and the core terminology behind the product.

## What is sampling

**Sampling** is how you choose which traces are **kept** in the pipeline and which are **dropped** before they reach backends and UIs. It trims volume so ingest, storage, and search stay manageable, while policies still aim to leave enough signal for debugging and reliability work.

In Odigos, sampling is **per trace**: the entire trace is either retained end to end or not exported, so parent–child structure and timing stay coherent when something is kept. How and when that choice is made—early in the agent at the time of creation (**head** sampling), or at the **Odigos Gateway** after spans are assembled (**tail** sampling)—is covered on [Head and tail sampling](./head-and-tail-sampling).

## Core motivations and definitions

### Primary driver: cost reduction

The main reason for sampling is **cost reduction**. Unsampled traces can flood collectors, backends, and analysis tools with volume that drives up ingest fees, storage, and compute. Sampling also helps prevent the observability stack from being overwhelmed by **redundant or low-value (“uninteresting”) data**, so capacity stays available for signals that support debugging and understanding production behavior.

### Reducing noise from uninteresting traces

Sampling is also commonly used as a **sanitation layer** to drop traces that add little debugging value. Typical examples include:

* **Health-check probes** and readiness/liveness endpoints
* **Metrics scraping** requests (for example, `/metrics` endpoints polled by Prometheus)
* **Infrastructure traffic** such as service-mesh sidecar chatter, internal control-plane calls, and routine background jobs

Removing this noise keeps the observability stack focused on application behavior and prevents redundant data from drowning out meaningful signals.

### Ambiguity of terminology

In many systems, “sampling” is described either as **what is dropped** or **what is kept**. Those views are equivalent mathematically, but they express different policy intent—retention versus reduction. Odigos is oriented around **prioritized signal**: defining and keeping **important** traces so that critical evidence remains available for investigation and problem-solving.

> [!NOTE]
>
> Phrasings such as “drop 90%” and “keep 10%” describe the same outcome. Odigos emphasizes **prioritization** and **clear intent** rather than opaque drop semantics alone.

## Introduction to sampling

### Why use sampling? Cost versus visibility

**Without sampling**

Visibility is maximized—every trace is available for analysis—but that comes with cost, infrastructure load, and noise. At high volume, the stack and ingest budget become the limiting factors.

**With sampling**

Not every trace is stored; the aim is still to preserve enough signal to detect errors, latency regressions, and dependency issues. The balance is deliberate: invest ingest and storage in traces that matter instead of exhaustive duplication of routine, successful traffic.

### The Odigos philosophy

Traditional framing often starts from dropping data to save money. Odigos shifts the emphasis to **prioritizing what matters**: configuration expresses what should be treated as important (errors, slow paths, specific services or attributes, and so on), and the pipeline retains those traces preferentially. Cost control follows from that prioritization rather than from arbitrary bulk deletion alone.

> [!TIP]
>
> When designing sampling policies, **define “important” first** (failures, SLO risks, critical services), then set fallbacks or rates for everything else. That mirrors how Odigos surfaces configuration and avoids tuning only by a global drop percentage.

### Sampling level

Sampling in Odigos is per trace: a single keep-or-drop decision is applied to every span. That decision is made at the root span for head sampling, or at the Odigos Gateway for tail sampling. Partial retention would break parent–child links, timing, and attributes that span the whole request, so a kept trace is always **complete** from the root span through its children.

### Key terminology

These terms describe how traces are classified in the sampling model:

> **Important traces**
>
> Traces that match configured rules for high value—for example, failures, high latency, or explicitly selected attributes. The system is designed to **keep** these preferentially so they remain available for investigation.

> **Regular traces**
>
> Typical traffic that does not match those priority rules. These may be **sampled at a lower rate** (or dropped according to configuration) while still allowing a statistical view of normal behavior where policies allow it.

> **Noisy traces**
>
> Low-value traffic such as **health checks, readiness/liveness probes, metrics scraping, and routine infrastructure calls**. These are typically **dropped on sight** as a sanitation step, since they add volume without aiding investigation.

> **Dropped traces**
>
> Traces that are **not retained** for export to destinations after sampling decisions—whether classified as noisy, deprioritized as regular, or excluded by rate limits. They are not available in the backend for that retention window; this is how volume and cost are reduced.

Together, this vocabulary ties configuration to outcomes: policies define what is important, drop what is noisy, treat everything else as regular unless additional rules apply, and dropped traces represent the tradeoff for sustainable cost and signal-to-noise ratio.


# Cost Reduction
Source: https://docs.odigos.io/oss/pipeline/sampling/rules/cost-reduction

Cap how much of the remaining traffic is exported to control trace volume and ingest cost.

**Cost Reduction** rules cap how much traffic is retained for export, after Noisy and Highly Relevant rules have run. They are the **third and final** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories) and are how you control **volume and ingest cost** for traffic that is neither obviously noisy nor specifically marked as important.

## How they're evaluated

* **Applied to traces that fall through earlier categories.** Traces that matched a Noisy drop never reach this stage; traces that matched a Highly Relevant rule are governed by that rule instead (see [Interaction with Highly Relevant](#interaction-with-highly-relevant)).
* **Percentage-based.** Each rule has a **Keep Percentage** that defines the fraction of matching traffic to retain.
* **Scoped.** A Cost Reduction rule can apply to the whole cluster, one or more namespaces, services, or programming languages, and can target all operations or a specific HTTP route or Kafka topic. See [Configuration](../configuration) for the full authoring options.

## Built-in rule: Drop Traces Cluster-Wide

Odigos ships a single built-in Cost Reduction rule, **Drop Traces Cluster-Wide**, which applies one **Keep Percentage** uniformly across every trace in the cluster. You configure the percentage; the engine then makes an independent, probabilistic keep/drop decision per trace.

At high trace volume, the **law of large numbers** applies and sample proportions—error rates, latency percentiles, per-service and per-route traffic shares—converge on the underlying population. The retained traces preserve the **shape** of your traffic, so common scenarios remain visible at their original relative frequencies.

> [!WARNING]
>
> Uniform sampling preserves **statistics**, not individual events. At low keep percentages, or for rare scenarios such as low-frequency error paths or occasional regressions, specific occurrences can be missed entirely. To **guarantee** retention for traces you care about, use a [Highly Relevant](./highly-relevant) rule—which always wins over Cost Reduction—rather than relying on this rule alone.

## Interaction with Highly Relevant

Highly Relevant **always wins** over Cost Reduction on traces that match both, regardless of which keep percentage is higher. That means a Cost Reduction cap of 10% on `/api/orders` does **not** suppress a Highly Relevant rule that keeps 100% of error traces on the same route. See [Evaluation between Highly Relevant and Cost Reduction](./overview#evaluation-between-highly-relevant-and-cost-reduction-categories) for worked examples.

## Typical use cases

* Keep 5% of all cluster traffic as a baseline statistical sample
* Keep 50% of `/api/products` traffic to control a high-volume endpoint
* Keep 1% of traces from a chatty internal service


# Highly Relevant
Source: https://docs.odigos.io/oss/pipeline/sampling/rules/highly-relevant

Mark and retain important traces using RED-style signals like errors and high duration, with a per-rule keep percentage.

**Highly Relevant** rules mark traces you want to treat as **important**, aligned with **RED**-style signals (request, **error**, **duration**). They are the **second** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories), evaluated only on traces that did **not** match any Noisy rule.

## How they're evaluated

* **Tail-aware.** Importance often depends on something that only appears mid-trace—an error on a child span, or a long downstream call—so the evaluator typically needs the assembled trace. See [Head vs. tail sampling](../head-and-tail-sampling).
* **One error span marks the entire trace.** When a Highly Relevant error rule applies, a single error span anywhere in the tree is enough to classify the whole trace as important.
* **100% retention by default.** Highly Relevant rules keep every matching trace by default—as the built-in error rule does—which works for most workloads. If a specific operation produces too much volume, lower its Keep Percentage on a custom rule to retain only a fractional sample, while still benefiting from Highly Relevant priority.

## Built-in rule: Keep All Error Traces

Odigos ships a single built-in Highly Relevant rule, **Keep All Error Traces**, which retains **100%** of traces that contain at least one span with **error** status. The built-in rule's keep percentage is fixed at **100%** and is **not configurable**.

Because the evaluator inspects every span in the trace, a single error anywhere in the call tree is enough to keep the whole trace—even when the root span itself succeeds.

> [!NOTE]
>
> If 100% retention of error traces is more than you need—for example because your error volume is high—author your own Highly Relevant rule on errors with the **Keep Percentage** you want. Custom rules are how you express any intent other than the built-in 100%.

## Interaction with Cost Reduction

When a trace matches rules in **both** Highly Relevant and Cost Reduction, the Highly Relevant Keep Percentage **always wins**, whether higher or lower than the Cost Reduction cap. See [Evaluation between Highly Relevant and Cost Reduction](./overview#evaluation-between-highly-relevant-and-cost-reduction-categories) for worked examples.

> [!WARNING]
>
> Highly Relevant rules without any Cost Reduction rules **do not aggressively reduce volume** on their own. Unimportant traffic still flows through the gateway and may be evaluated as part of full-trace policy. Pair Highly Relevant with Cost Reduction when you need both signal and volume control.

## Typical use cases

* Keep all (or most) traces with at least one error span
* Keep a higher percentage of traces over a duration threshold (for example > 500 ms)
* Always keep a sample from a critical service or route


# Noisy Operation
Source: https://docs.odigos.io/oss/pipeline/sampling/rules/noisy-operation

Drop or down-sample low-value traffic — health probes, scrapes, heartbeats — early at the root span with head sampling.

**Noisy Operation** rules drop or down-sample traffic that you treat as having little or no value—for example Kubernetes liveness and readiness probes, periodic health checks, or scraper traffic. They are the **first** stage in the [rule evaluation pipeline](./overview#order-of-operations-between-categories): a trace that matches a Noisy rule is **not** evaluated against Highly Relevant or Cost Reduction rules.

## How they're evaluated

* **Head sampling at the root span.** The keep/drop decision is made on the entry span, before child spans are processed.
* **No trace buffering.** Because the decision happens at the start of the trace, the gateway does not need to wait for an [aggregation window](../head-and-tail-sampling#aggregation-window).
* **Lowest overhead.** Head sampling is the **most efficient** sampling type, with the **least CPU and memory** impact on collectors and gateway.

> [!WARNING]
>
> Head-sampled Noisy rules require an instrumentation agent that supports head sampling. Today this includes **Node.js**, **Python**, **Java**, and **Go**. For other languages or runtimes, Noisy rules are **evaluated at tail-sampling time** instead—the trace is buffered for the full [aggregation window](../head-and-tail-sampling#aggregation-window) before the drop happens, so the efficiency advantages above (no buffering, lowest overhead) **do not apply**.

## Possible outcomes

A matching trace is either:

* **Dropped** — never exported and never evaluated against Highly Relevant or Cost Reduction rules.
* **Sampled down** — kept at a configured percentage. The kept fraction also skips later categories.

> [!NOTE]
>
> Dropped traces do not continue to other categories. If you need a noisy route to remain visible when something goes wrong (for example a failing health check), replace the Noisy rule with a **Highly Relevant** rule on errors and use a **Cost Reduction** rule to cap the successful traffic.

## Built-in rule: Kubernetes Health Probes

Odigos ships a single built-in Noisy rule, **Kubernetes Health Probes**, which drops traces for the **liveness**, **readiness**, and **startup** probe paths discovered on each workload's Kubernetes manifest.

The rule's **Keep Percentage** controls how much probe traffic is retained. Common configurations:

* **0% — drop everything (default).** Probes carry no signal about application behavior, so dropping them keeps the pipeline focused on real traffic and minimizes cost.
* **\~2% — sparse liveness signal.** For services with little or no real traffic, a small permanent value keeps a handful of probe traces flowing. The retained samples act as a "service is up" indicator on dashboards that would otherwise look empty, helping distinguish a *quiet* service from a *silently broken* one.
* **100% — debug probe failures.** Temporarily raise while investigating failing or flaky probes, then reset to your normal value once resolved.

> [!TIP]
>
> Only **HTTPGet** probes declared on the manifest are matched. Probes defined as `exec` or `tcpSocket`, or probes routed through different paths (custom configuration, ingress rewrites, service-mesh wrappers), are not auto-handled; add an explicit Noisy rule for those paths if needed.

## Typical use cases

* Kubernetes `liveness`, `readiness`, and `startup` probes
* Internal `/health` and `/healthz` endpoints
* Prometheus `/metrics` scrape traffic
* Background heartbeats and keep-alive pings
* High-volume, low-value endpoints — for example a static asset route, a polling endpoint, or an autocomplete API that produces many traces without contributing to debugging or SLOs


# Overview
Source: https://docs.odigos.io/oss/pipeline/sampling/rules/overview

How Odigos evaluates sampling rules: categories, order between Noisy, Highly Relevant, and Cost Reduction, conflicts inside a category, and head vs. tail sampling.

This page describes how **sampling rules** are evaluated in Odigos: the three **rule categories** (Noisy Operation, Highly Relevant, Cost Reduction), the **order between categories**, and how **conflicts inside a category** resolve. Each category has its own page; this overview focuses on how the categories interact.

## Rule categories

> **[Noisy Operation](./noisy-operation)**
>
> Traffic with little or no value (for example Kubernetes health probes). Decided early at the **root span** with **head sampling**.

> **[Highly Relevant](./highly-relevant)**
>
> **Important** traffic aligned with **RED**-style signals (errors, duration). Each rule's **Keep Percentage** controls how much is retained.

> **[Cost Reduction](./cost-reduction)**
>
> Caps applied to remaining traffic to control volume and cost (for example a percentage across the cluster or a specific route).

> [!NOTE]
>
> See [Head vs. tail sampling](../head-and-tail-sampling) for how head and tail sampling differ in Odigos (root-span decisions versus aggregation at the gateway).

If you only define **Highly Relevant** rules and **no Cost Reduction** rules, unimportant traffic is not aggressively reduced by those rules alone—policy may still require full-trace context to evaluate importance, which increases gateway work without cutting volume. Use **Highly Relevant** and **Cost Reduction** together when you need both signal and volume control.

## Order of operations between categories

Categories are **not** all evaluated for every trace in parallel. Odigos applies them in this fixed **sequence**:

> **Noisy Operation**
>
> If the trace **matches** a Noisy rule, it is **dropped or sampled down** as configured. The trace exits the pipeline at this stage; **Highly Relevant** and **Cost Reduction** are **not** evaluated for that trace.

> **Highly Relevant**
>
> Only traces that did **not** match any Noisy rule reach this stage. Rules here classify **important** traffic (for example errors and duration); **important** matches **can still be sampled** per each rule's **Keep Percentage**. Traces **continue** to **Cost Reduction** after this stage.

> **Cost Reduction**
>
> For traces that **match** rules in this **category** after the earlier stages, this stage applies caps to control volume and cost.

The pipeline runs the categories in this order. At each stage, every rule in that category is considered; how multiple matches resolve is covered in [Rule evaluation](#rule-evaluation).

## Rule evaluation

### Evaluation inside a category

Within a single category, rule order does not matter. When multiple rules match the same trace, the engine combines them so all of them are satisfied at once.

For Highly Relevant, percentages are floors — the highest keep percentage wins, so any retention guarantee is honored. For Noisy Operation and Cost Reduction, percentages are caps — the lowest keep percentage wins, so the strictest cap is honored. This makes it safe to add rules without fear of one silently overriding another.

### Evaluation between Highly Relevant and Cost Reduction categories

When a trace matches rules in **both** the **Highly Relevant** and **Cost Reduction** categories, the **Highly Relevant** keep percentage **always wins** — regardless of whether it is higher or lower than the **Cost Reduction** cap.

> **Highly Relevant rate higher than Cost Reduction**
>
> **Trace:** `POST /api/orders` that returned a `500` error.
>
> **Matching rules:**
>
> * **Highly Relevant** — keep **100%** of error traces.
> * **Cost Reduction** — keep at most **10%** of `/api/orders` traffic.
>
> **Outcome:** the trace is **kept at 100%**. Highly Relevant overrides the Cost Reduction cap.

> **Highly Relevant rate lower than Cost Reduction**
>
> **Trace:** `GET /api/products` that took **750 ms**.
>
> **Matching rules:**
>
> * **Highly Relevant** — keep **25%** of traces longer than **500 ms**.
> * **Cost Reduction** — keep at most **50%** of `/api/products` traffic.
>
> **Outcome:** the trace is **kept at 25%**. Highly Relevant still wins, even though its keep rate is lower than the Cost Reduction cap.

## Why full traces still matter

For traces that are evaluated on the **tail** path, the evaluator needs the assembled trace (within the aggregation window) so it can apply **Highly Relevant** conditions that depend on any span. That is why tail sampling and aggregation are part of the pipeline for those policies, even when the category order is Noisy → Highly Relevant → Cost Reduction.


# Service Graph
Source: https://docs.odigos.io/oss/pipeline/service-graph

Visualize how your services connect to each other and to uninstrumented dependencies like Redis or Kafka.

## Overview

The Service Graph is a live topology map that shows which services call each other and how those calls are performing.
It is built entirely from your distributed traces — no additional instrumentation or agents are required.

Odigos generates the service graph automatically when traces are enabled and at least one tracing destination is configured.

## How It Works

Odigos uses the OpenTelemetry [Service Graph Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector) inside the Cluster Gateway Collector. The pipeline looks like this:

```
Instrumented services
       │ (OTLP traces)
       ▼
 Node Collector (DaemonSet)
       │ (OTLP traces)
       ▼
 Cluster Gateway Collector
       ├──► servicegraph connector ──► Odigos UI (topology map)
```

1. The **servicegraph connector** receives every trace that passes through the gateway.
   It matches CLIENT spans with their corresponding SERVER spans to detect service-to-service calls.
2. For each matched pair it emits a metric labelled with the `client` and `server` service names plus any configured extra dimensions, which the Odigos UI uses to render the interactive topology map.

### Virtual Nodes (Uninstrumented Dependencies)

When a service calls an uninstrumented dependency (Redis, Kafka, a third-party API, …) there is no SERVER span.
The connector handles this with **virtual nodes**: it reads certain attributes from the CLIENT span to infer the name of the remote endpoint and creates a synthetic node in the graph.

The attributes checked — in priority order — are configured via `serviceGraphVirtualNodePeerAttributes`.
The connector picks the **first attribute that is present** on the CLIENT span.

> [!NOTE]
>
> Only **span-level attributes** are read when resolving virtual node identities (e.g. `server.address`, `db.system`, `peer.service`).

## Enabling and Disabling

The service graph is **enabled by default** when traces are active.
Disabling it removes the `servicegraph` connector from the gateway pipeline and stops emitting the related metrics.

> [!WARNING]
>
> Disabling the service graph also removes the load balancing behavior between the node collectors and the gateway, because the gateway no longer needs to aggregate spans from the same trace.

```yaml theme={null}
# values.yaml
collectorGateway:
  serviceGraphDisabled: true   # set to true to disable
```

## Configuration Options

All options live under `collectorGateway` in your `values.yaml` (Helm) or by manually editing the `odigos-configuration` ConfigMap in your cluster.

### Extra Dimensions

By default the graph distinguishes services by `service.name` only.
`serviceGraphExtraDimensions` adds more labels to the edge metrics, creating a separate node for each unique combination of values.

**When to use it:** multiple deployments share the same `service.name` (e.g. the same service deployed in `staging` and `production` namespaces).

```yaml theme={null}
collectorGateway:
  serviceGraphExtraDimensions:
    - k8s.namespace.name   # separate nodes per namespace
```

> [!NOTE]
>
> Both span-level and resource-level attributes are supported for extra dimensions — the connector reads from the full span resource context.
> Avoid high-cardinality attributes such as `k8s.pod.name`; each unique value creates a separate time series in Prometheus.

### Virtual Node Peer Attributes

`serviceGraphVirtualNodePeerAttributes` controls which span attribute is used to name uninstrumented dependencies.
The connector evaluates the list in order and uses the first attribute that has a non-empty value.

**When to use it:** multiple instances of the same technology (e.g. a Redis Cluster with several shards, or two separate Redis caches) are collapsing into a single node because they all resolve to the same value (e.g. `db.system=redis`).

```yaml theme={null}
collectorGateway:
  serviceGraphVirtualNodePeerAttributes:
    - peer.service      # explicit logical name — highest priority
    - server.address    # hostname of the specific instance (e.g. redis-cache.default.svc.cluster.local)
    - net.peer.name     # older convention, wide SDK support
    - db.name           # database index (rarely useful for Redis — most apps use index 0)
    - db.system         # technology name (e.g. "redis") — last resort, not instance-specific
```

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Add Sources
Source: https://docs.odigos.io/oss/pipeline/sources/create

Add a Source to enable telemetry collection for specific Kubernetes workloads, or a whole namespace.

## With Odigos UI

The easiest way to add a source to Odigos is to use the Odigos UI.\

After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI:

```shell theme={null}
odigos ui
```

And then, in the setup page or the sources page, select the workloads / namespaces for which you want telemetry collection to be enabled.

To include all the workloads in a namespace as sources, and automatically add future-new workloads, enable the `Future Apps` toggle in the UI.
Otherwise, you can enable the `Select All` toggle to include all the workloads in the namespace as sources, excluding future-new workloads.

## With Odigos CLI

See the [CLI documentation on Sources](../../cli/odigos_sources) for detailed command documentation for manipulating Sources.

## Kubernetes Manifests

If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can add a source to Odigos by creating a Source object.

> [!WARNING]
>
> Only one Source object may exist for a given workload, except in namespace instrumentation.

### Workload Source

The following Source object enables instrumentation for a Deployment named `frontend` in the `default` namespace and belong to Datastream `example`.

> **Note:** Sources that do not have a Datastream label will be automatically attached to the default Datastream.

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
  labels:
    odigos.io/data-stream-example: "true"
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
```

### Namespace Source

The following Source object enables instrumentation for all the workloads in the `default` namespace:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: namespace-source
  namespace: default
spec:
  workload:
    name: default
    namespace: default
    kind: Namespace
```

Namespace Sources must have the following fields:\


* `spec.workload.name == spec.workload.namespace`
* `spec.workload.kind: Namespace`

### Disable Workload Source in Namespace

You can have a namespace Source and a workload Source that both enable instrumentation for a workload. In this case, if one is deleted, the other will persist instrumentation for the workload.

To exclude specific workloads from namespace instrumentation, set `disableInstrumentation: true` on the workload Source. For example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: namespace-source
  namespace: default
spec:
  workload:
    name: default
    namespace: default
    kind: Namespace
---
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
spec:
  disableInstrumentation: true
  workload:
    name: frontend
    namespace: default
    kind: Deployment
```

In this example, all workloads in the `default` namespace will be instrumented except for the `frontend` Deployment. If `example-source` is deleted (or `disableInstrumentation` is set to `false`), the `frontend` Deployment will be instrumented.


# Remove Sources
Source: https://docs.odigos.io/oss/pipeline/sources/delete

Remove a Source to disable telemetry collection for specific Kubernetes workloads, or a whole namespace.

> [!NOTE]
>
> Sources are persistent and decoupled from the workloads they instrument.\\
>
> If a workload is deleted without deleting the Source, the Source will resume its effects if the workload is re-created later.

## With Odigos CLI

The Odigos CLI supports instrumenting and uninstrumenting workloads with the [`odigos sources enable`](../../cli/odigos_sources_enable)
and [`odigos sources disable`](../../cli/odigos_sources_disable) commands.

Each of these commands accepts a source type, name, and namespace.

For example, to remove a source from odigos, run:

```shell theme={null}
odigos sources disable deployment <deployment-name> -n <deployment-namespace>
```

This command will rollout existing pods to restart without any odigos additions.

It can be undone (re-instrumented) by running: "

```shell theme={null}
odigos sources enable deployment <deployment-name> -n <deployment-namespace>
```

## With Odigos UI

The easiest way to remove a source from Odigos is to use the Odigos UI.\

After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI:

```shell theme={null}
odigos ui
```

And then, in the sources page, unselect the workloads / namespace for which you want telemetry collection to be disabled.

## Kubernetes Manifests

If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can remove a source from Odigos by deleting the Source object, for example:

**Step 1**

Run:

```shell theme={null}
kubectl get sources -n default
```

Output:

```shell theme={null}
NAME           WORKLOAD     KIND         NAMESPACE
source-7sxsb   frontend     Deployment   default
source-b42rn   inventory    Deployment   default
source-b7zn9   coupon       Deployment   default
source-v6sxs   membership   Deployment   default
source-v9jjw   pricing      Deployment   default
```

**Step 2**

Run:

```shell theme={null}
kubectl delete source source-7sxsb -n default
```

Output:

```shell theme={null}
source.odigos.io "source-7sxsb" deleted
```


# Introduction
Source: https://docs.odigos.io/oss/pipeline/sources/introduction

Sources are a Custom Resource that define which Kubernetes workloads should be auto-instrumented by Odigos.

## How Sources work

Source custom resource objects are created as a reference to a workload or namespace ("entities").

Odigos reacts to changes in these custom resource objects to configure instrumentation for the referenced entity.

On this page, "Sources" refers to the custom resource object, while "workload", "namespace", or "entity" refers to the application
being instrumented.

See the [API Reference](../../api-reference/odigos.io.v1alpha1#source) for the full structure of `Source` objects and their fields.

## Supported Entities

Odigos supports the following Kubernetes entities:

* Kubernetes **Deployment**
* Kubernetes **DaemonSet**
* Kubernetes **StatefulSet**
* Kubernetes **Namespace**

## Types of Sources

A Source object can be either:

* A **Workload Source**, which controls instrumentation for a single workload (Deployment, DaemonSet, or StatefulSet)
* A **Namespace Source**, which controls instrumentation for every workload in a namespace.

### Workload Sources

Workload Sources specify a single workload, and the Source represents the instrumentation configuration for that workload.

Use Workload Sources to control instrumentation and configuration for individual Deployments, StatefulSets, or DaemonSets
with unique requirements.

Example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-workload
  namespace: default
spec:
  workload:
    kind: DaemonSet
    name: sample-app
    namespace: default
```

### Namespace Sources

Namespace Sources represent instrumentation settings for an entire namespace. Namespace Sources are created by setting
`spec.workload.kind: Namespace` along with `spec.workload.name` and `spec.workload.namespace` equal to the name of the
namespace to control.

Use Namespace Sources to control instrumentation for multiple Deployments, StatefulSets, and DaemonSets from a
single spot.

Example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-namespace
  namespace: default
spec:
  workload:
    kind: Namespace
    name: default
    namespace: default
```

## Using multiple Sources

Workload Sources are intended to be combined with Namespace Sources for flexible, fine-grained control. For example,
you may choose to create a Namespace Source to enable instrumentation for namespace `foo`, but want to exclude deployment
`bar`. In this case, you can create a Workload Source for deployment `bar` with `disableInstrumentation: true`:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: my-namespace
  namespace: foo
spec:
  workload:
    kind: Namespace
    name: foo
    namespace: foo
---
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: exclude-bar
  namespace: foo
spec:
  disableInstrumentation: true
  workload:
    kind: Deployment
    name: bar
    namespace: foo
```

The opposite is also possible: you could disable instrumentation for namespace `foo` with a Namespace Source but
choose to explicitly enable deployment `bar` via a Workload Source that sets `disableInstrumentation: false` (the default).

This is because Workload Sources take priority over Namespace Sources, and any settings in a Workload
Source will override those in a parent Namespace Source.

## Source Workflows

Sources are intended to control and configure instrumentation as objects that are completely decoupled from their
target workloads. When working with Sources, they should be treated as such.

The only requirements for a Source to identify a workload are:

* The Source must set `spec.workload.name`, `spec.workload.namespace`, and `spec.workload.kind` to match the workload (or namespace).
* The Source must exist in the same namespace as the target workload (or namespace).

Besides these requirements, you are encouraged to manage your Sources as their own objects.

For example, when creating Sources from YAML manifests, consider storing them together in their own file
such as `sources.yaml`, separate from your workload manifests. This separation of concerns allows you
to declaratively control your application's observability without tying it to your application manifests.

You should also define RBAC policies that restrict access to only what is needed to work with Source objects.
In other words, your observability team may not require the ability to view deployments in a namespace to manage
the instrumentation for those deployments.

For example, the following Role grants full editor permissions for Sources:

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: source-editor
rules:
- apiGroups: ["odigos.io"]
  resources: ["sources"]
  verbs: ["*"]
```

## Source Labels

Odigos uses mutating webhooks to automatically copy the values of `spec.workload` to matching labels. It is
encouraged to use these labels to manage Sources in a cluster:

* `odigos.io/workload-name`
* `odigos.io/workload-namespace`
* `odigos.io/workload-kind`

The value of these labels will be validated to always match the corresponding fields in `spec.workload`. They can be used to group and list Sources, for example:

```shell theme={null}
kubectl get sources -l odigos.io/workload-kind=Deployment
```

This will list all `Sources` for `Deployments`.

## Automatically created Sources

In some scenarios, Odigos may create Sources for you automatically.

* When instrumenting workloads in the UI, Odigos will create a Workload Source for each workload.
* When choosing **Include future workloads** in the UI, Odigos will create a Namespace Source for the namespace.
* When creating a Source grouping in the UI, Odigos will create a Workload Source for each Workload in the group.

In all of these cases, if a Source already exists, Odigos will either patch the existing Source or
return an error if it can't.


# Troubleshooting
Source: https://docs.odigos.io/oss/pipeline/sources/troubleshooting

If you selected a workload to auto-instrument and you don't see any data, use the checklist below to troubleshoot the issue.

> [!TIP]
>
> It is recommended to use the [`odigos describe`](../../cli/odigos_describe) command to get detailed information in one command.

## Check is Source Created?

A `Source` object is required for Odigos to instrument the workload or namespace, such as:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example
  namespace: foo
spec:
  workload:
    name: myapp
    namespace: foo
    kind: StatefulSet
```

If you expect to see data from a workload but don't, check if the `Source` object exists:

```shell get all in namespace theme={null}
kubectl get sources -n <namespace>
```

```shell get by deployment name theme={null}
kubectl get sources -n <namespace> -l odigos.io/workload-name=<deployment-name>
```

```shell get by deployment kind theme={null}
kubectl get sources -n <namespace> -l odigos.io/workload-kind=<deployment-kind>
```

## Check is Language Detected?

After a `Source` object is created, Odigos will attempt to detect the programming language of the containers in the workload pods, and write the detected language to an `InstrumentationConfig` object in the same namespace.

The resulting language is used to determine which instrumentation to apply.

If a `Source` object exists, verify that the `InstrumentationConfig` also exists:

```shell get all in namespace theme={null}
kubectl get instrumentationconfigs -n <namespace>
```

```shell get by crd name theme={null}
kubectl get instrumentationconfigs -n <namespace> <crd-name> -o yaml
```

If an `InstrumentationConfig` object exists, verify that the `status.runtimeDetailsByContainer` array contains the expected languages, for example:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: InstrumentationConfig
metadata:
  name: example
  namespace: foo
spec:
  serviceName: example
status:
  runtimeDetailsByContainer:
  - containerName: example
    language: go
    runtimeVersion: 1.21.2
```

## Check is Device Instrumented?

Once an `InstrumentationConfig` object is created, and the supported language is detected, Odigos will wait until a [destination](../../backends-overview) is added, and OpenTelemetry Collectors are deployed to the cluster (which will receive the telemetry data).

To inject the instrumentation agents into the pods, Odigos will patch the workload template containers with a resource based on the detected language.

If an `InstrumentationConfig` object exists, and the languages are detected, verify that the `Deployment` object contains the expected instrumentation devices:

```shell get all in namespace theme={null}
kubectl get deployments -n <namespace>
```

```shell get by deployment name theme={null}
kubectl get deployments -n <namespace> <deployment-name> -o yaml
```

It should look something like this:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: foo
spec:
  template:
    spec:
      containers:
      - resources:
          limits:
            instrumentation.odigos.io/go-ebpf-community: "1"
```

The `instrumentation.odigos.io/go-ebpf-community` resource-limit instructs Odigos on how to mount the relevant OpenTelemetry agents and environment variables into the container, and which processes should be instrumented with eBPF.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Configuration
Source: https://docs.odigos.io/oss/pipeline/sources/update



Beyond just enabling or disabling instrumentation, Source objects can be used to configure instrumentation settings as well.

All Source settings can be configured with the Odigos CLI (see [CLI documentation on updating Sources](../../cli/odigos_sources_update)) or Kubernetes-native tools like `kubectl`.

## service.name

The `service.name` attribute is a fundamental resource attribute in [OpenTelemetry](https://opentelemetry.io/docs/specs/semconv/resource/#service).\

It is required, and is reported in all telemetry data.\

It is used to identify the service that emitted a span, metric or log record.

Odigos will automatically populate this attribute based on the name of the workload. If there are multiple containers in the workload, the `service.name` will be set to the container name.

If you want to override the default value, you can also set the `service.name` attribute by adding the `otelServiceName` attribute to an exiting Source, or by creating a `Source` with that attribute:

```yaml theme={null}
apiVersion: odigos.io/v1alpha1
kind: Source
metadata:
  name: example-source
  namespace: default
spec:
  workload:
    name: frontend
    namespace: default
    kind: Deployment
  otelServiceName: my-custom-service-name
```

Namespace Sources do not support the `otelServiceName` attribute. If a namespace is instrumented, and a Source for a specific workload is not present, a specific source for that workload will need to be created including the desired `otelServiceName`.

### From Odigos UI

The easiest way to update a source from Odigos is to use the Odigos UI.\

After [installing Odigos CLI](../../setup/installation), run the following command to start the Odigos UI:

```shell theme={null}
odigos ui
```

And then, in the sources page, select the source you want to edit and fill in the value for the `service.name` field.
To undo the override, simply erase the value from the `service.name` field.

### Odigos CLI

To set the `service.name` using the Odigos CLI, use the `odigos sources update` command ([see documentation](../../cli/odigos_sources_update)).

If you know the name of the Source object to update, pass that to `odigos sources update` along with the `--otel-service` flag:

```shell theme={null}
odigos sources update <mysource> --otel-service=my-service-name -n <namespace>
```

If you don't know the name of the Source object to update, you can provide flags to `odigos sources update` to identify the Source by workload:

```shell theme={null}
odigos sources update --workload-kind=<Deployment/StatefulSet/DaemonSet> \
--workload-name=<my-workload> \
--workload-namespace=<namespace> \
--otel-service=my-service-name
```

### Kubernetes Manifest

If you prefer to use kubectl or GitOps to manage your Kubernetes workloads, you can can set the `service.name` attribute by updating the corresponding Source.

For example, to set the `service.name` attribute in telemetry for an existing source:

To get the relevant Source for your workload you can either use:

```shell theme={null}
kubectl get sources -n <source namespace> \
-l odigos.io/workload-kind=<Deployment/DaemonSet/StatefulSet/Namespace> \
-l odigos.io/workload-name=<Workload name> \
-l odigos.io/workload-namespace=<source namespace>
```

or list the sources in the relevant namespace:

```shell theme={null}
kubectl get sources -n <namespace>
```

After we have the desired Source we can patch it:

```shell theme={null}
kubectl patch source/<source-name> -n <namespace> --type=merge -p '{"spec":{"otelServiceName":"<serviceName>"}}'
```

If you don't know the name of a Source object (such as when a Source
is automatically created by instrumentation done via the UI), you can
combine the use of Source labels with the `kubectl patch` command. For
example:

```shell theme={null}
kubectl patch source \
$(kubectl get source -l 'odigos.io/workload-name=<workloadName>,odigos.io/workload-kind=<Deployment/StatefulSet/DaemonSet>' --no-headers -o custom-columns=":metadata.name") \
--type=merge \
-p '{"spec":{"otelServiceName":"<serviceName>"}}'
```


# Building a pipeline
Source: https://docs.odigos.io/oss/quickstart/building-a-pipeline



## Choosing target applications

You should now see the following page:

![Select target applications](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_choose_apps.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=f6fd9fc8e033b3a0507112d0e2490708)

Select all the applications in the `default` namespace and click `Next`.

> [!NOTE]
>
> Selecting **Apply for future apps** will automatically generate distributed traces for any application that will be deployed on that namespace in the future.

## Adding Jaeger as a destination

In the next step, you will be asked to select a destination for your traces.
Scroll down to the self hosted destinations and click on **Jaeger**.

![Select Jaeger](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_choose_dest.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=4cd44e8b085d4b669293f075b593d0d0)

#### Connection details

You will now be asked to provide the connection details for your Jaeger instance:

* **Destination name**: A name for your destination
* **Host**: The hostname of your Jaeger instance.

Enter any name you want for the destination, and enter `jaeger.tracing:4317` as the host.

![Jaeger connection details](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_jaeger_connection.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=2a0e1189e01181665972c87f2b1c6a1d)

**That's it!** You can now click `Done` and finish the wizard.

Odigos will now instrument your selected applications and deploy the nessesary OpenTelemetry collectors.
The following page will show an overview of your observability pipeline:

![Observability pipeline](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/ui_overview.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=c1db71afde509a147b602457ef55a23c)


# Generating Data
Source: https://docs.odigos.io/oss/quickstart/generation-data



## Wait for collectors to be ready

Before generating data, make sure that the collectors and applications are ready. To check this run the following command:

```bash theme={null}
kubectl get pods -n default
```

```bash theme={null}
kubectl get pods -n odigos-system
```

![Wait for collectors to be ready](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/collectors_ready.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=4724ddc8c75204c4640bd3652d433256)

## Access frontend application and generate data

In order to access the frontend application, run the following command:

```bash theme={null}
kubectl port-forward svc/frontend 8080:8080
```

Then, open the following URL in your browser: [http://localhost:8080](http://localhost:8080) and purchase some items.

![Frontend application](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/demo_frontend_app.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=70e9781773d14615c3c929ac108b2200)


# Installation
Source: https://docs.odigos.io/oss/quickstart/installation



## Installing Odigos CLI

Odigos CLI is the easiest way to install Odigos on your Kubernetes cluster. It
is a single binary that can be installed on any Linux, MacOS or Windows machine.

**Homebrew**

If you are on MacOS, you can install Odigos CLI using Homebrew:

```bash theme={null}
brew install odigos-io/odigos-cli/odigos
```

**GitHub Releases**

1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux).

2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system.

3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH".

* For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt.

* For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`.

4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH.

**Windows**

1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows).

2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files.

3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how:

* **Option 1**:
  Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder.

* **Option 2**:
  Add the folder containing the extracted binary to your PATH manually:
  1. Right-click the **Start** button and select **System**.
  2. Click **Advanced system settings** on the left.
  3. In the **System Properties** window, click **Environment Variables** at the bottom.
  4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**.
  5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file.
  6. Click **OK** to close all windows.

4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located.

## Installing Odigos on Kubernetes

To install Odigos on your Kubernetes cluster, run the following command:

```bash theme={null}
odigos install
```

## Accessing Odigos UI

After Odigos is installed, open the Odigos UI by running the following
command:

```bash theme={null}
odigos ui
```

By default, Odigos UI is available at [http://localhost:3000](http://localhost:3000).


# Introduction
Source: https://docs.odigos.io/oss/quickstart/introduction



In this tutorial we are going to use Odigos for generating distributed traces for a microservices-based application written in Go, Java, Python, .NET and Node.js.

We are going to deploy the application in a Kubernetes cluster and use [Jaeger](https://www.jaegertracing.io/) as the backend for storing and visualizing the traces.

> [!NOTE]
>
> This guide is designed for:
>
> * IT professionals with experience in system administration or DevOps
> * Software engineers familiar with containerization and Kubernetes concepts
> * Anyone comfortable with command-line interfaces and basic cloud infrastructure
>
> If you're new to these concepts or prefer a more detailed, step-by-step approach, we recommend starting with our [Beginner's Guide to Odigos Installation](../quickstart/installation-guide-for-beginners).

## Preparing the environment

### Creating a Kubernetes cluster

This first step is to create a Kubernetes cluster.
We recommend using [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/docs/) for trying Odigos out in a local environment.

> [!WARNING]
>
> **Mac users**: please avoid using Docker Desktop built-in Kubernetes cluster as it [does not](https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation) support bind propagation.

Create a new local Kubernetes cluster, by running the following command:

**Kind**

```bash theme={null}
kind create cluster
```

**Minikube**

```bash theme={null}
minikube start
```

### Deploying the target application

We are going to install a [demo application](https://github.com/odigos-io/simple-demo) that consists of 6 microservices written in Java, Go, Python, Node.js, .NET and PHP.

![Architecture of demo application](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/demo_architecture.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=079eb8601b77c8fdc5e777a34f686f25)

Deploy the application using the following command:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/odigos-io/simple-demo/main/kubernetes/deployment.yaml
```

### Deploying Jaeger

Jaeger is a popular open source distributed tracing backend. We are going to use it for storing and visualizing the traces generated by Odigos.

Install Jaeger using the following command:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/odigos-io/simple-demo/main/kubernetes/jaeger.yaml
```

### Waiting for the applications to start

Before moving to the next step, make sure that the application pods and Jaeger pods are running, this may take a few minutes.

```bash theme={null}
kubectl wait --for=condition=available --timeout=300s deployment --all --all-namespaces
```

![Deployments Ready](https://mintcdn.com/odigos/_RekT-psRDkRKhxE/images/pods_ready.png?fit=max\&auto=format\&n=_RekT-psRDkRKhxE\&q=85\&s=a417114517cace40ac4344592702d2b3)


# Next Steps
Source: https://docs.odigos.io/oss/quickstart/next-steps



You just finished the Odigos quickstart guide! 🎉

## Not just distributed tracing

**Odigos supports metrics and logs too!** You can send metrics and logs from your
application to Odigos, and then visualize them in your favorite backend.

In order to send metrics and logs, just select a destination that supports them.

> **[Many more observability backends!](../backends-overview)**
>
> Odigos supports all the major observability backends, both open-source and commercial.
> Select the relevant backend for your use case below to connect it to Odigos.


# Viewing traces
Source: https://docs.odigos.io/oss/quickstart/viewing-traces



Now that we have some traces, let's view them!
Access the Jaeger UI by running the following command:

```bash theme={null}
kubectl port-forward -n tracing svc/jaeger 16686:16686
```

Then, open [http://localhost:16686](http://localhost:16686) in your browser and explore the traces.

![Jaeger UI](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/jaeger_ui.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=623e9ab3fa2cf69effa287c1cdad19da)


# Install Odigos Images from a Custom Docker Registry
Source: https://docs.odigos.io/oss/setup/docker-registry



## Tagging and Pushing Images

### Step 1: Set Environment Variables

First, export the `VERSION` and `CUSTOM-DOCKER-REGISTRY` environment variables in your shell:

```bash theme={null}
export VERSION=<YOUR-DESIRED-VERSION>
export CUSTOM_DOCKER_REGISTRY=<YOUR-CUSTOM-DOCKER-REGISTRY>
export PLATFORM=<YOUR-PLATFORM-ARCHITECTURE>
```

Replace `<YOUR-DESIRED-VERSION>` with the version of the Odigos images you want to use, `<YOUR-CUSTOM-DOCKER-REGISTRY>` with the URL of your custom Docker registry, and `<YOUR-PLATFORM-ARCHITECTURE>` with the platform architecture of the environment where you are deploying Odigos (linux/amd64, linux/arm64).

***

### Step 2: Pull the Images

Start by pulling the required images from the default community registry (`docker.io/keyval`) onto your local machine or CI environment:

```bash theme={null}
docker pull docker.io/keyval/odigos-instrumentor:$VERSION
docker pull docker.io/keyval/odigos-ui:$VERSION
docker pull docker.io/keyval/odigos-odiglet:$VERSION
docker pull docker.io/keyval/odigos-collector:$VERSION
docker pull docker.io/keyval/odigos-victoria-metrics:$VERSION
docker pull docker.io/keyval/odigos-cache:$VERSION
```

If you are using the [k8s-init-container](../instrumentations/configuration/mount-method#3-InitContainer) mount method
make sure the odigos-agents image is also pulled, as it's required by the injected InitContainer.
On Kubernetes 1.35+, that init container also needs the registry pull secret in each instrumented namespace. See [Init container image pull secrets](../instrumentations/configuration/mount-method#init-container-image-pull-secrets).
**This is not the default configuration**, if you're unsure what this means, you can safely ignore this step and continue with the standard setup.

```bash theme={null}
docker pull docker.io/keyval/odigos-agents:$VERSION
```

If you are using [image mirrored Go offsets updates](../instrumentations/golang/ebpf#about-go-offsets), also mirror the offsets image:

```
docker pull docker.io/keyval/cli-offsets:latest
```

Keeping this image regularly up-to-date is necessary to pull the latest offsets into your cluster.

> [!WARNING]
>
> Odigos component images are published in multi-arch for ARM64 and AMD64 architectures. When pulling these images, be sure to specify the correct architecture *for the environment they will be deployed*.

```bash theme={null}
docker pull --platform $PLATFORM docker.io/keyval/odigos-instrumentor:$VERSION
docker pull --platform $PLATFORM docker.io/keyval/odigos-ui:$VERSION
docker pull --platform $PLATFORM docker.io/keyval/odigos-odiglet:$VERSION
docker pull --platform $PLATFORM docker.io/keyval/odigos-collector:$VERSION
docker pull --platform $PLATFORM docker.io/keyval/odigos-victoria-metrics:$VERSION
docker pull --platform $PLATFORM docker.io/keyval/odigos-cache:$VERSION
```

### Step 3: Tag the Images

Next, Tag each image with your custom Docker registry prefix:

```bash theme={null}
docker tag docker.io/keyval/odigos-instrumentor:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-instrumentor:$VERSION
docker tag docker.io/keyval/odigos-ui:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION
docker tag docker.io/keyval/odigos-odiglet:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-odiglet:$VERSION
docker tag docker.io/keyval/odigos-collector:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-collector:$VERSION
docker tag docker.io/keyval/odigos-victoria-metrics:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION
docker tag docker.io/keyval/odigos-cache:$VERSION $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION
```

> **Note:** Community images use the registry prefix `docker.io/keyval` (for example `docker.io/keyval/odigos-instrumentor`). Image names themselves are `odigos-<component>` without an embedded `keyval/` path segment. If you previously mirrored images under an older layout, re-tag them to match this naming when pushing to your custom registry.

### Step 4: Push the Images

Now, push the tagged images to your custom Docker registry:

```bash theme={null}
docker push $CUSTOM_DOCKER_REGISTRY/odigos-instrumentor:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-ui:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-odiglet:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-collector:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-victoria-metrics:$VERSION
docker push $CUSTOM_DOCKER_REGISTRY/odigos-cache:$VERSION
```

### Step 5: Configure Access for Private Registries

If your Docker registry is private, configure your Kubernetes cluster to pull images from it. Refer to the official Kubernetes documentation for guidance: [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).

Pass the secret name as `imagePullSecrets` at install time so Odigos components can pull mirrored images. For `k8s-init-container` on Kubernetes 1.35+, also copy that secret into each instrumented namespace. See [Init container image pull secrets](../instrumentations/configuration/mount-method#init-container-image-pull-secrets).

> **Note: To ensure your Odigos deployments and daemonsets can pull images from a private registry, patch them to add the image pull secret.**
>
> **Use the following commands:**
>
> ```
> export ODIGOS_NAMESPACE=<NAMESPACE-WHERE-ODIGOS-IS-INSTALLED>
> export IMAGE_PULL_SECRET=<NAME-OF-IMAGE-PULL-SECRET>
> kubectl patch deployment odigos-ui -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-instrumentor -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-gateway -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-victoriametrics -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch deployment odigos-cache -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch daemonset odiglet -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> kubectl patch daemonset odigos-data-collection -n $ODIGOS_NAMESPACE \
> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/imagePullSecrets", "value": [{"name": '$IMAGE_PULL_SECRET'}]}]'
> ```

### Step 6: Install Odigos Using Custom Images

Finally, install Odigos using the images from your custom Docker registry:

**Odigos CLI**

```bash theme={null}
odigos install --image-prefix=$CUSTOM_DOCKER_REGISTRY
```

**Helm Chart**

```bash theme={null}
helm repo update
helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace --set imagePrefix=$CUSTOM_DOCKER_REGISTRY
kubectl label namespace odigos-system odigos.io/system-object="true"
```


# Installation
Source: https://docs.odigos.io/oss/setup/installation



> [!TIP]
>
> Installation issues? Need help? Message us on [Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)

Checkout the [installation options](../setup/installation-options) to customize your installation.

There are 3 ways to install odigos in your kubernetes cluster:

* [Odigos CLI](#odigos-cli)
* [Helm Chart](#helm-chart)
* [OpenShift Operator](#openshift-operator)

All install methods will deploy and configure resources in the active Kubernetes cluster as per the current context.
Before proceeding with the installation, ensure that you are targeting the correct cluster with `kubectl config current-context`.

## Odigos CLI

### Install the CLI

**Homebrew**

If you are on MacOS, you can install Odigos CLI using Homebrew:

```bash theme={null}
brew install odigos-io/odigos-cli/odigos
```

**GitHub Releases**

1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux).

2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system.

3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH".

* For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt.

* For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`.

4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH.

**Windows**

1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows).

2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files.

3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how:

   * **Option 1**:
     Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder.

   * **Option 2**:
     Add the folder containing the extracted binary to your PATH manually:
     1. Right-click the **Start** button and select **System**.
     2. Click **Advanced system settings** on the left.
     3. In the **System Properties** window, click **Environment Variables** at the bottom.
     4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**.
     5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file.
     6. Click **OK** to close all windows.

4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located.

### Install Odigos

```shell theme={null}
odigos install
```

This will install odigos in the `odigos-system` namespace in your kubernetes cluster.

foo bar

At this point, odigos is installed in your cluster. You can now add sources and destinations to instrument your cluster with OpenTelemetry.

## Helm Chart

### Add Helm Repository

```shell theme={null}
helm repo add odigos https://odigos-io.github.io/odigos/
```

### Install Odigos

```shell theme={null}
helm repo update
helm upgrade --install odigos odigos/odigos --namespace odigos-system --create-namespace
```

Once the Odigos pods are online, simply run the following command to launch the Odigos UI.

```shell theme={null}
odigos ui
```

## OpenShift Operator

For Operator installation steps, see the docs on: [OpenShift Installation](../setup/installation-options#openshift-installation)


# Installation Options
Source: https://docs.odigos.io/oss/setup/installation-options



You can configure odigos installation using the following options:

## Ignore Namespaces

Don't show specific uninteresting namespaces in the Odigos UI.

Default values: `kube-system`, `local-path-storage`, `istio-system`, `linkerd`, `kube-node-lease`

Add additional namespaces to ignore with **cli**:

```shell theme={null}
odigos install --ignore-namespace <namespace1> --ignore-namespace <namespace2>
```

List all namespaces to be ignored with **helm chart**:

```shell theme={null}
helm install odigos odigos/odigos --set ignoreNamespace=<namespace1>,<namespace2>
```

Any Source objects for the namespace and workloads in the namespace will still be honored.

On GKE, Google may warn that a custom admission controller can affect `kube-system`. The Odigos **pod mutating webhook** is cluster-wide so it can inject agents into application namespaces. It uses `failurePolicy: Ignore`, so webhook errors do not block pod creates in `kube-system`. Ignored namespaces (including `kube-system` by default) are not shown for instrumentation and are not instrumented unless you create a Source for them.

## Ignore Containers

This option is useful for excluding specific sidecar containers from being instrumentated by odigos.

Default values: `istio-proxy`

**Odigos CLI**

```shell theme={null}
odigos install --ignore-container <container1> --ignore-container <container2>
```

**Helm Chart**

```shell theme={null}
helm install odigos odigos/odigos --set ignoreContainer=<container1>,<container2>
```

## Custom Container Runtime Socket Path

Odigos uses the container runtime socket to query for application environment variables info, and distinguish between those injected by the runtime (via Dockerfile for example) and those defined in Kubernetes manifests.

To perform this analysis, Odiglet requires access to the container runtime’s Unix socket (e.g., `containerd.sock`). However, in certain environments—such as Rancher—the socket may not be located in a default path.

To address this, Odigos provides a configuration option to mount a custom container runtime socket path into the Odiglet. This path can be specified either during installation or afterwards.

### Usage

**Odigos CLI**

1. During and Post installation using set flag for the install command:

```shell theme={null}
odigos install --set odiglet.customContainerRuntimeSocketPath=<unix socket path>
```

**Helm Chart**

1. During installation:

```shell theme={null}
helm install odigos odigos/odigos --set odiglet.customContainerRuntimeSocketPath=<unix socket path>
```

## OpenShift Installation

Installing on OpenShift requires additional settings that can be
configured based on your installation mode:

**Odigos CLI**

Run the Odigos CLI with the `--openshift` flag:

```shell theme={null}
odigos install --openshift
```

**Helm Chart**

Set `openshift.enabled=true` in `Values.yaml`:

```yaml theme={null}
openshift
  enabled: true
```

**Operator**

In the OpenShift console, navigate to **Operators** > **OperatorHub** and search for **Odigos**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-2.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=0b2238fee5cdadcd3be83db9fd304204)

Click **Install**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-3.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=6c0311adaf5191c43c4054eaddc42d91)

Choose a namespace to install the Operator in (this will be the same namespace that Odigos is installed in) and click **Install**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-4.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=98c89e169348fc88301148d351d4751e)

When installation is complete, click **View Operator**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-5.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=72e90cbaa7bb630f1efcbe1aaf5a7187)

Click **Create Instance** to create an `Odigos` custom resource:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-6.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=c2bec9274033f6e850c336d50651fe41)

If you have an Enterprise Token, fill in the **On Prem Token** field with your Enterprise Token.  Otherwise leave it blank to use Odigos OpenSource.

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-7.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=569391cd0146df727538b74b1b6ba6f7)

Scroll down and click **Create**:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-8.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=db6b4869e4a22358525fd4f29fed57a7)

Confirm that the Custom Resource has reached state `OdigosInstalled`:

![](https://mintcdn.com/odigos/q83sjmvkbDHigqAk/images/operator/operator-9.png?fit=max\&auto=format\&n=q83sjmvkbDHigqAk\&q=85\&s=f35484a82b81ace0914ba3536a301aa2)

OpenShift installation does the following:

* Uses [Red Hat certified component container images](https://odigos.io/blog/openshift-certified)
* Configures required SELinux permissions for pods to access auto-instrumentation agents on RHEL nodes
* Configures RBAC permissions for Odigos components required by default in OpenShift clusters

By default, the `--openshift` flag will pull Odigos component images
from `registry.connect.redhat.com/odigos`. These images are based on
[Red Hat's Universal Base Image
9](https://catalog.redhat.com/software/base-images) and denoted with
the `-rhel-certified` suffix. These images are also hosted on the Odigos
DockerHub registry, which you can deploy by setting `--openshift` with
`--image-prefix docker.io/keyval`.

### Exposing the Odigos UI via OpenShift Routes

To access the Odigos UI in OpenShift, use either the standard `odigos
ui` command or create a `Route` that exposes the `ui` service on your
cluster's external address. For example:

```
oc expose service ui -n odigos-system
```

Then get the external address for the Route with `oc get routes ui -n
odigos-system`.

**Note:** If you're deploying Odigos on OpenShift, you may need to set the route's TLS termination to Edge to load it in your browser.
you will need to modify the YAML of the oc route with `spec.tls.termination: edge`

Check the OpenShift documentation or run `oc expose -h` for more
information on configuring OpenShift routes according to your
cluster's requirements.

## GKE Installation

Installing on Google Kubernetes Engine (GKE) requires additional settings
that can be configured based on your cluster type:

**Odigos CLI**

For GKE Standard clusters:

```shell theme={null}
odigos install --set gke.enabled=true
```

For GKE Autopilot clusters:

```shell theme={null}
odigos install --set gke.enabled=true --set gke.autopilot=true
```

**Helm Chart**

For GKE Standard clusters, set `gke.enabled=true` in `values.yaml`:

```yaml theme={null}
gke:
  enabled: true
```

Or pass it as a flag during installation:

```shell theme={null}
helm install odigos odigos/odigos --set gke.enabled=true
```

For GKE Autopilot clusters, also set `gke.autopilot=true`:

```yaml theme={null}
gke:
  enabled: true
  autopilot: true
```

Or pass both flags during installation:

```shell theme={null}
helm install odigos odigos/odigos --set gke.enabled=true --set gke.autopilot=true
```

GKE installation with `gke.enabled=true` does the following:

* Configures a `ResourceQuota` for `system-node-critical` pods, which is required for odiglet on GKE

### GKE Autopilot

GKE Autopilot restricts privileged workloads by default. Odiglet requires
elevated permissions for eBPF-based instrumentation, so it must be allowlisted
before it can run on Autopilot clusters.

Allowlist synchronization requires GKE version `1.32.2-gke.1652000` or later.

When `gke.autopilot=true`, the Helm chart:

* Creates an `AllowlistSynchronizer` that syncs the Odigos odiglet allowlist from the [Google-managed partner repository](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads)
* Skips node label setup in the odiglet init phase, where Autopilot nodes cannot be modified. Autopilot is auto-detected by the `gk3-` node name prefix; set `gke.autopilot=true` to force this behavior

#### Node label limitations

On standard clusters, odiglet sets `odigos.io/odiglet-oss-installed=true` (or
`odigos.io/odiglet-enterprise-installed=true` for enterprise) on each node after
preparing it for instrumentation. Autopilot does not allow modifying node labels,
so odiglet skips this step.

Without the label:

* **Instrumented pod scheduling with hostPath or CSI mount methods** — For
  `k8s-host-path` and `k8s-csi-driver` mount methods, the instrumentor webhook
  adds required node affinity matching the odiglet installed label so pods only
  schedule on nodes where odiglet has prepared `/var/odigos`. Without the label
  on Autopilot nodes, newly instrumented pods may remain Pending and fail to
  schedule. The default `k8s-virtual-device` mount method does not rely on this
  label and is recommended on Autopilot.
* **Karpenter integration** — Odiglet normally removes the `odigos.io/needs-init`
  startup taint when it sets the node label. This step is also skipped on
  Autopilot, so the Karpenter integration workflow is not supported.

#### Manually create the AllowlistSynchronizer

If you do not set `gke.autopilot=true`, create the `AllowlistSynchronizer` yourself
before odiglet pods are scheduled. See the [GKE documentation for running Autopilot
partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads)
for more details.

Apply the following manifest to install all Odigos allowlists from the partner
repository:

```yaml theme={null}
apiVersion: auto.gke.io/v1
kind: AllowlistSynchronizer
metadata:
  name: odigos-allowlist-synchronizer
spec:
  allowlistPaths:
    - "Odigos/Odigos/*"
```

```shell theme={null}
kubectl apply -f allowlist-synchronizer.yaml
```

After the allowlist is installed, verify that it is ready before odiglet pods start:

```shell theme={null}
kubectl wait --for=condition=Ready allowlistsynchronizer/odigos-allowlist-synchronizer --timeout=60s
kubectl get workloadallowlist
```

For more information on privileged workload allowlisting in Autopilot, see the
[GKE documentation on running Autopilot partner workloads](https://cloud.google.com/kubernetes-engine/docs/how-to/run-autopilot-partner-workloads).


# Configure Odigos with Argo CD
Source: https://docs.odigos.io/oss/setup/odigos-with-argocd



This guide shows you how to deploy Odigos in your Kubernetes cluster using Argo CD with the official Odigos Helm chart. This allows for GitOps-based management and observability instrumentation across your applications.

## Prerequisites

Before you begin, make sure you have:

* A running Kubernetes cluster
* Argo CD installed and configured in the cluster
* Helm CLI installed (optional but useful for inspection and verification)

## Step 1: Add the Odigos Helm Repository

Add the Odigos Helm repository to your Helm configuration:

```bash theme={null}
helm repo add odigos https://odigos-io.github.io/odigos --force-update
helm repo update
```

## Step 2: Identify the Helm Chart Version

Check available versions of the Odigos chart to determine the desired `targetRevision`:

```bash theme={null}
helm search repo odigos --versions
```

## Step 3: Define the Argo CD Application Manifest

Create an Argo CD `Application` custom resource to deploy Odigos using Helm. Save the following YAML as `odigos-application.yaml`:

```yaml theme={null}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: odigos
  namespace: <ARGOCD_NAMESPACE>  # <- Replace with your Argo CD namespace
  finalizers:
  - resources-finalizer.argocd.argoproj.io/foreground  # <- # Cascade-delete managed resources when the Application is deleted (otherwise only the Application CR is removed).

spec:
  project: default
  source:
    repoURL: https://odigos-io.github.io/odigos
    chart: odigos
    targetRevision:  # REQUIRED: set the version from 'helm search repo odigos --versions'
    helm:
      releaseName: odigos
      values: |
        # OPTIONAL: add your custom Helm values here if needed.
        # Argo CD expects this field to be a YAML-formatted string.
        # To use chart defaults, delete this field entirely or set it to an empty string ("").
  destination:
    server: https://kubernetes.default.svc
    namespace: <ODIGOS_NAMESPACE>  # <- Replace with the desired namespace where you want Odigos to be installed
  # Chart regenerates odigos-deployment-id on every Argo render; ignore it to avoid OutOfSync flaps.
  ignoreDifferences:
    - kind: ConfigMap
      name: odigos-deployment
      jsonPointers:
        - /data/odigos-deployment-id
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      # Without this, sync still overwrites ignored fields from the rendered manifest.
      - RespectIgnoreDifferences=true
```

## Step 4: Apply the Application

Use `kubectl` to create the application in Argo CD:

```bash theme={null}
kubectl apply -f odigos-application.yaml
```

This will instruct Argo CD to deploy Odigos using the Helm chart.

## Step 5: Monitor the Deployment

You can monitor the status of the deployment through the Argo CD UI:

* Navigate to your Argo CD web interface.
* Find the application named `odigos`.
* Verify that it reaches a **Healthy** and **Synced** state.

Once complete, Odigos will be installed in your cluster and ready to instrument workloads.


# Configure Odigos with Karpenter
Source: https://docs.odigos.io/oss/setup/odigos-with-karpenter



## What is Karpenter?

[Karpenter](https://karpenter.sh/) is an open-source cluster autoscaler built for Kubernetes. It dynamically launches just the right compute resources to handle your cluster’s workloads and efficiently scales them based on real-time demand.

Unlike traditional autoscalers, Karpenter can provision nodes in seconds and makes scheduling decisions at the pod level, enabling highly dynamic infrastructure scaling.

***

## Mount Method Compatibility

Odigos support for Karpenter depends on the configured [mount method](../instrumentations/configuration/mount-method):

| Mount method | Karpenter support | What you need |
| - | - | - |
| [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer) | Works out of the box | No additional Karpenter configuration |
| [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) | Supported with Odigos Karpenter integration | Enable `karpenter.enabled` and configure NodePool `startupTaints` (see [HostPath and CSI](#hostpath-and-csi-startup-taints)) |
| [`k8s-virtual-device`](../instrumentations/configuration/mount-method#1-VirtualDevice) (default) | Supported only when your Karpenter installation can apply [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) capacity | Enable the NodeOverlay feature (when required) and apply a NodeOverlay CR (see [Virtual Device](#virtual-device-nodeoverlay)) |

Follow the section for your mount method below.

***

## Init Container

If you use `k8s-init-container`, no Karpenter-specific configuration is required. You can stop here.

***

## Virtual Device (NodeOverlay)

With `k8s-virtual-device`, instrumented pods request the extended resource `instrumentation.odigos.io/generic`. That capacity is advertised by the Odigos device plugin on nodes where `odiglet` is already running.

Karpenter simulates new nodes from instance-type information. Because that simulation does not include the Odigos device resource by default, Karpenter can fail provisioning with errors such as:

```text theme={null}
no instance type has enough resources ... instrumentation.odigos.io/generic: "1"
```

### Provider and version caveats

Support depends on your **Karpenter version, cloud provider, and how the controller is packaged**:

* [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) is an alpha Karpenter feature (available since Karpenter v1.7.x). Exact enablement steps and API readiness vary by release.
* On many installs (including official AWS Karpenter Helm charts), NodeOverlay is **disabled by default** and must be turned on with a feature gate, for example:

```bash theme={null}
helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace <KARPENTER_NAMESPACE> \
  --reuse-values \
  --set settings.featureGates.nodeOverlay=true
```

Equivalently, set `--feature-gates NodeOverlay=true` / `FEATURE_GATES=...,NodeOverlay=true` on the Karpenter controller.

* Not every Karpenter distribution applies NodeOverlays during provisioning, even if the CRD and feature gate exist. Validate in your environment that creating the overlay below allows Karpenter to launch nodes for pods that request `instrumentation.odigos.io/generic`.

If NodeOverlay is unavailable or ineffective in your Karpenter install, use [`k8s-init-container`](../instrumentations/configuration/mount-method#3-InitContainer), or [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) / [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) with the [startup taint integration](#hostpath-and-csi-startup-taints).

### Apply a NodeOverlay

After NodeOverlay support is enabled in your Karpenter controller, apply a [NodeOverlay](https://karpenter.sh/docs/concepts/nodeoverlays/) that adds the Odigos device capacity for the NodePool that should run instrumented workloads.

Replace `default` with your NodePool name if needed:

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: odigos-instrumentation
spec:
  requirements:
    - key: karpenter.sh/nodepool
      operator: In
      values: ["default"]
  capacity:
    instrumentation.odigos.io/generic: "1024"
EOF
```

Confirm the overlay is ready:

```bash theme={null}
kubectl get nodeoverlay odigos-instrumentation
```

Karpenter should then be able to provision nodes for pods that request `instrumentation.odigos.io/generic`. After the node registers, `odiglet` advertises the real device capacity so pods can schedule.

***

## HostPath and CSI (Startup Taints)

For the [`k8s-host-path`](../instrumentations/configuration/mount-method#2-HostPath) and [`k8s-csi-driver`](../instrumentations/configuration/mount-method#4-CSI-Driver) mount methods, Odigos adds a **node affinity** rule to instrumented pods so they only run on nodes where `odiglet` is installed.

This works fine in static clusters — but in Karpenter-managed clusters, this causes a problem:

* `odiglet` is installed by a DaemonSet **after** the node is created.
* The label `odigos.io/odiglet-installed=true` is only added to the node at runtime.
* **Karpenter does not know in advance** that this label will appear.
* So when the scheduler tries to place a pod with this affinity, Karpenter concludes: "no nodes exist (or can be created) that match this label" — and **no node is provisioned**.

Once you enable the **Karpenter integration option** (explained below), Odigos will no longer apply node affinity to new instrumented pods.
Instead, it relies on a **startup taint mechanism**:

* Nodes launched by Karpenter should be initialized with a special taint (`odigos.io/needs-init=NoSchedule`).
  \[!NOTE] The exact method for configuring this taint in Karpenter will be explained in the following section.
* The `odiglet` removes this taint after it prepares the node for instrumentation.
* This allows instrumented pods to schedule only **after** the node is ready — achieving the same safety as affinity, without blocking provisioning.

### 1. Configure the Karpenter NodePool CRD

You must configure your Karpenter [NodePool](https://karpenter.sh/docs/concepts/nodepools/) to include a startup taint. This ensures Odigos can prepare each node before instrumented pods are scheduled onto it.

Add the following under `spec.template.spec.startupTaints`:

```yaml theme={null}
spec:
  template:
    spec:
      startupTaints:
      - key: odigos.io/needs-init
        effect: NoSchedule
```

To safely append this taint without overwriting any existing `startupTaints`, use the following command:

```bash theme={null}
export NODEPOOL_NAME=<NODEPOOL_NAME>
kubectl patch nodepool $NODEPOOL_NAME --type merge --patch "$(kubectl get nodepool $NODEPOOL_NAME -o json | jq -c '.spec.template.spec.startupTaints = (.spec.template.spec.startupTaints // []) + [{"key":"odigos.io/needs-init","effect":"NoSchedule"}] | {spec: {template: {spec: .spec.template.spec}}}')"
```

Replace `<NODEPOOL_NAME>` with the actual name of your Karpenter NodePool.

### 2. Add the Startup Taint to Existing Nodes

To manually add the startup taint to all current nodes:

```bash theme={null}
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | xargs -I {} kubectl taint node {} odigos.io/needs-init=:NoSchedule --overwrite
```

This ensures existing nodes behave consistently until they're replaced by Karpenter-managed ones.

### 3. Enable the Karpenter Integration in Odigos

Once the tainting setup is complete, inform Odigos to disable node affinity and rely on the taint mechanism.

**Odigos CLI**

Run the following using the Odigos CLI:

```bash theme={null}
odigos set config karpenter-enabled=true
```

**Helm Chart**

If installing or upgrading using helm:

```bash theme={null}
helm upgrade --install odigos odigos/odigos \
  --namespace odigos-system \
  --set karpenter.enabled=true
```

### 4. Restart the Odiglet DaemonSet

To ensure that taint-removal logic is triggered on all nodes, perform a rolling restart of the `odiglet` DaemonSet:

```bash theme={null}
kubectl rollout restart daemonset/odiglet -n odigos-system
```


# System Requirements
Source: https://docs.odigos.io/oss/setup/system-requirements

To ensure optimal performance and compatibility when deploying Odigos, your environment must meet the following system requirements:

## Kubernetes Version

Odigos requires Kubernetes version 1.21.12 or higher.

## Kubernetes Cluster Types

* [Kind](https://kind.sigs.k8s.io)
* [Minikube](https://minikube.sigs.k8s.io/docs/start)
* [K3s](https://k3s.io)
* [Red Hat OpenShift](https://catalog.redhat.com/software/container-stacks/detail/675201b3ade18c062e2efc0b) (see [how to install on OpenShift](../setup/installation-options#openshift-installation))
* [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)
* [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) (see [how to install on GKE](../setup/installation-options#gke-installation))
* [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service)

## Resource Requirements

To achieve its functionality Odigos uses a set of Deployments and DaemonSets.

### Cluster

The following components run as Deployment and request the following resources **(per pod)**:

| Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit |
| - | - | - | - | - |
| odigos-autoscaler | 64Mi | 512Mi | 10m | 500m |
| odigos-instrumentor (2 replicas) | 64Mi | 512Mi | 10m | 500m |
| odigos-ui | 64Mi | 512Mi | 10m | 500m |

In addition, Odigos deploys OpenTelemetry Collector Deployment,
with auto scaling (multiple replicas) and configurable resources requests and limits.
The default values (unless overridden) are:

| Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit |
| - | - | - | - | - |
| odigos-gateway | 600Mi | 600Mi | 1000m | 1000m |

Read more [here](../pipeline/configuration).

### Per Node

A DaemonSet running on every node in the cluster, consuming resources as outlined below:

| Component Name | Memory Request | Memory Limit | CPU Request | CPU Limit |
| - | - | - | - | - |
| odiglet | 500Mi | 500Mi | 500m | 500m |
| data-collection | 250Mi | 500Mi | 250m | 500m |
| deviceplugin | 200Mi | 300Mi | 40m | 100m |

You can override these values by providing a custom configuration (read more [here](../pipeline/configuration)).

## Network Requirements

> [!WARNING]
>
> Ensure that any network-layer components are properly configured to allow the required connectivity between Odigos components. Improper configuration may impact the correct operation of Odigos.

To ensure proper operation of Odigos components and instrumented applications, the following network connectivity must be allowed inside the Kubernetes cluster:

### Core Network Connectivity

| Source | Destination Service | Ports | Purpose |
| - | - | - | - |
| Instrumented Application Pods | `odiglet-local` | `4320` | Retrieve instrumentation configuration from Odiglet |
| Instrumented Application Pods | `odigos-data-collection-local-traffic` | `4317` (gRPC), `4318` (HTTP) | Export application telemetry data (OTLP protocol) |
| Odiglet Pods | `odigos-data-collection-local-traffic` | `4318` | Export eBPF-collected telemetry data |

### Network Layer Considerations

In some environments, the following network components may affect traffic between these services:

* Service meshes (e.g. Istio, Linkerd, Cilium, Consul)
* HTTP proxies or egress gateways
* Transparent proxies (corporate or cluster-level)
* Network security appliances (WAFs, L7 firewalls, DPI systems)
* Kubernetes NetworkPolicies


# Uninstall
Source: https://docs.odigos.io/oss/setup/uninstall



## With CLI

```shell theme={null}
odigos uninstall
```

## Uninstall Existing Odigos Installation with Helm

If you installed Odigos using Helm, you can uninstall it by running the following commands:

```shell theme={null}
helm delete odigos -n odigos-system
kubectl delete ns odigos-system
```


# Upgrade
Source: https://docs.odigos.io/oss/setup/upgrade



## Finding the Current Version

```shell theme={null}
odigos version
```

Will print the version of the CLI tool and the version of odigos that is currently installed in the active cluster.

## Upgrading Existing Odigos Installation with CLI

First, upgrade the CLI tool to latest version:

**Homebrew**

If you are on MacOS, you can install Odigos CLI using Homebrew:

```bash theme={null}
brew install odigos-io/odigos-cli/odigos
```

**GitHub Releases**

1. Download the Latest Version: Head over to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page and click on the latest version available. You’ll see a list of files—choose the one that matches your operating system (Windows, macOS, or Linux).

2. Extract the Downloaded File: Once the file is downloaded, it will likely be in a compressed format like `.zip` or `.tar.gz`. To use the program, you’ll first need to extract (or unzip) it. Right-click on the file and select "Extract Here" or use a similar option, depending on your operating system.

3. Move the File to a Folder in Your `PATH`. You’ll need to move this file to a folder where your computer looks for programs, known as your "PATH".

* For Windows: Move the extracted file to a folder like `C:\Windows\System32` or any folder listed in your system’s PATH. You can find out where by typing echo `%PATH%` in Command Prompt.

* For macOS/Linux: Move the file to a folder like `/usr/local/bin/` or `/usr/bin/`, both of which are common folders included in the PATH. You can check your PATH by opening the terminal and typing `echo $PATH`.

4. Verify It’s Working: Once the file is in the correct folder, open a terminal or command prompt and type the name of the program (e.g., odigos). If it runs, you’ve successfully installed it! If not, double-check that you moved the file to a folder included in your PATH.

**Windows**

1. Download the Latest Version: Go to the [Odigos GitHub Releases](https://github.com/odigos-io/odigos/releases) page. Look for the latest version and download the file that matches your system (typically, this will be a `.zip` file for Windows).

2. Extract the Downloaded File: Once the download is complete, locate the `.zip` file in your Downloads folder. Right-click on it and choose "Extract All." Select a location on your computer where you'd like to save the extracted files.

3. Move the Binary to a Folder in Your PATH: Windows has specific folders where it looks for programs to run, called the "PATH". To make Odigos work from anywhere on your system, you’ll need to move the program (binary) into one of these folders. Here’s how:

   * **Option 1**:
     Move the file to a folder already in the PATH, such as `C:\Windows\System32`. Simply drag the extracted file (often named something like `odigos.exe`) into that folder.

   * **Option 2**:
     Add the folder containing the extracted binary to your PATH manually:
     1. Right-click the **Start** button and select **System**.
     2. Click **Advanced system settings** on the left.
     3. In the **System Properties** window, click **Environment Variables** at the bottom.
     4. In the **System Variables** section, scroll down and find the variable named `Path`. Select it and click **Edit**.
     5. In the **Edit Environment Variables** window, click **New**, and then add the folder where you saved the extracted file.
     6. Click **OK** to close all windows.

4. Verify Installation: Open the **Command Prompt** by searching for `cmd` in the Start menu. Type the name of the program, like `odigos.exe`, and press Enter. If the program runs, then you’ve successfully installed it. If it doesn’t, double-check that you either moved the file to a folder in your PATH or correctly added the folder where the file is located.

And upgrade Odigos installation in the active kubernetes cluster with CLI:

```shell theme={null}
odigos upgrade
```

## Upgrade Existing Odigos Installation with Helm

**Upgrade to Latest Version**

```shell theme={null}
helm repo update
helm upgrade odigos odigos/odigos --namespace odigos-system
```

**Upgrade to Specific Version**

```shell theme={null}
helm repo update
helm search repo odigos --versions ## list all available versions
helm upgrade --install odigos odigos/odigos \
--namespace odigos-test \
--create-namespace \
--version <VERSION>
```


# Verifying Odigos Images
Source: https://docs.odigos.io/oss/setup/verifying-images



All Odigos container images are signed at build time with [Sigstore cosign](https://docs.sigstore.dev/)
(keyless), and each carries a CycloneDX SBOM attestation. Signatures are stored in the
registry next to the image and recorded in the public
[Rekor transparency log](https://search.sigstore.dev/), so you can verify that any image
you pull was published by the Odigos release pipeline — no keys involved.

## Verify an image

Verification needs [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) v3+.
Always verify by digest:

> [!NOTE]
>
> Verify against the canonical registry, `us-central1-docker.pkg.dev/odigos-cloud/components`.
> `registry.odigos.io` is a pull alias: images resolve through it, but signature
> discovery for retroactively signed versions does not currently surface there.

```bash theme={null}
IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-odiglet
DIGEST=$(docker buildx imagetools inspect "${IMG}:<version>" --format '{{.Manifest.Digest}}')

cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \
  "${IMG}@${DIGEST}"
```

The identity accepts two workflows on purpose: new releases are signed by the
release pipeline, while versions published before signing went live were signed
retroactively by the audited backfill workflow. A single-identity pattern would
fail on one era or the other.

To verify and inspect the attached SBOM (same identity rules):

```bash theme={null}
cosign verify-attestation --type cyclonedx \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$' \
  "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components'
```

The SBOM attestation is attached to the **multi-arch index digest** — always pass
the digest from `imagetools inspect` as above, not a per-platform child digest
(children carry signatures but no attestation).

> **Full worked example: verifying odigos-instrumentor v1.35.0**
>
> Run as-is — no placeholders to fill in:
>
> ```bash theme={null}
> IMG=us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor
> TAG=v1.35.0
> DIGEST=$(docker buildx imagetools inspect "${IMG}:${TAG}" --format '{{.Manifest.Digest}}')
> IDENTITY='^https://github\.com/odigos-io/(odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$'
>
> cosign verify \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp "$IDENTITY" \
>   "${IMG}@${DIGEST}"
>
> cosign verify-attestation --type cyclonedx \
>   --certificate-oidc-issuer https://token.actions.githubusercontent.com \
>   --certificate-identity-regexp "$IDENTITY" \
>   "${IMG}@${DIGEST}" | jq -r .payload | base64 -d | jq '.predicate.components | length'
> ```
>
> Expected output (trimmed):
>
> ```text theme={null}
> Verification for us-central1-docker.pkg.dev/odigos-cloud/components/odigos-instrumentor@sha256:f46fa614… --
> The following checks were performed on each of these signatures:
>   - The cosign claims were validated
>   - Existence of the claims in the transparency log was verified offline
>   - The code-signing certificate was verified using trusted certificate authority certificates
> [{"critical":{"identity":{"docker-reference":"…/odigos-instrumentor"},
>   "image":{"docker-manifest-digest":"sha256:f46fa614…"},"type":"cosign container image signature"}, …}]
>
> 1022
> ```
>
> The 1022 components are the SBOM contents — inspect them with
> `jq '.predicate.components'` instead of `length`, e.g. the first entries here are
> `base-files 13.8+deb13u6` and `github.com/argoproj/argo-rollouts v1.9.1`. Because
> v1.35.0 predates release-time signing, the certificate for this version names the
> audited backfill workflow (`ci-core/.github/workflows/backfill-sign.yml`) — that is
> the expected identity for retroactively signed history, and the reason the identity
> regexp accepts both workflows.

## Signing identities

Each artifact is signed by its own release workflow — never verify with a broad
pattern like `.*odigos-io.*`. The canonical list of identities and the version each
artifact is signed from lives in
[`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md).

| Artifact | Release identity (regexp) |
| - | - |
| Component images (`odigos-*`) | `^https://github\.com/odigos-io/odigos/\.github/workflows/publish-modules\.yml@refs/tags/v.*$` |
| CLI image + `odigos-victoria-metrics` + `odigos-cache` | `^https://github\.com/odigos-io/odigos/\.github/workflows/release\.yml@refs/heads/main$` |
| Enterprise images (`odigos-enterprise-*`) | `^https://github\.com/odigos-io/odigos-enterprise/\.github/workflows/release-images\.yml@refs/(tags/.*\|heads/.*)$` |
| `odigos-cli-offsets` | `^https://github\.com/odigos-io/enterprise-go-instrumentation/\.github/workflows/offsets-push\.yml@refs/heads/.*$` |
| `odigos-vmagent-instrumentations` | `^https://github\.com/odigos-io/vm-agent/\.github/workflows/publish-agents\.yml@refs/(heads/.*\|tags/agents/.*)$` |

To also cover retroactively signed history, combine the release identity with the
backfill identity in one pattern, as the commands above do:
`^https://github\.com/odigos-io/(<release workflow path>@<ref>|ci-core/\.github/workflows/backfill-sign\.yml@refs/heads/main)$`

Two more certificate details are expected and legitimate:

* **Backfill**: historical releases were signed retroactively by
  `odigos-io/ci-core/.github/workflows/backfill-sign.yml` — the certificate honestly
  records the backfill identity and date, and it is auditable in Rekor.
* **Stable `x.y.0` tags** verify against the release-candidate tag identity: the stable
  tag is a retag of the exact RC digest, so the certificate names the RC that built it.

## Attestable images

Every image below is signed and carries a CycloneDX SBOM attestation, from the
version listed in [`SIGNING.md`](https://github.com/odigos-io/ci-core/blob/main/SIGNING.md):

* **Odigos platform**: `odigos-instrumentor`,
  `odigos-collector`, `odigos-odiglet`, `odigos-ui`, `odigos-operator`, `odigos-agents`,
  `odigos-cli`, `odigos-victoria-metrics`, `odigos-cache`
* **Odigos Enterprise**: `odigos-enterprise-odiglet`, `odigos-enterprise-instrumentor`,
  `odigos-enterprise-ui`, `odigos-enterprise-collector`, `odigos-enterprise-central-proxy`,
  `odigos-enterprise-central-backend`, `odigos-enterprise-central-ui`, `odigos-enterprise-agents`,
  `odigos-enterprise-connector-runtime`, `odigos-enterprise-connector-aws`,
  `odigos-enterprise-connector-azure`, `odigos-enterprise-connector-gcp` — `odiglet`
  and `agents` also publish `-extended` variants (separate digests, same identity)
* **Supporting images**: `odigos-cli-offsets`, and
  `odigos-vmagent-instrumentations` (signed for publishes after 2026-08-12)

## CLI binary verification

Every [GitHub release](https://github.com/odigos-io/odigos/releases) ships a
`checksums.txt` covering the CLI archives, keyless-signed by the release pipeline.
The Sigstore bundle (`checksums.txt.sigstore.json`) is published as a release asset
next to it. Verify the checksum file, then check your downloaded archive against it:

```bash theme={null}
VERSION=<version>   # the release you downloaded, e.g. v1.40.0
BASE=https://github.com/odigos-io/odigos/releases/download/${VERSION}

curl -sfLO "${BASE}/checksums.txt"
curl -sfLO "${BASE}/checksums.txt.sigstore.json"

cosign verify-blob \
  --bundle checksums.txt.sigstore.json \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/odigos-io/odigos/\.github/workflows/release\.yml@refs/heads/main$' \
  checksums.txt

curl -sfLO "${BASE}/cli_${VERSION#v}_linux_amd64.tar.gz"   # your platform's archive
sha256sum -c checksums.txt --ignore-missing
```

`--ignore-missing` limits the check to the archives you actually downloaded
(on macOS use `shasum -a 256 -c checksums.txt --ignore-missing`).

## Air-gapped verification

The Rekor proof travels inside the signature bundle, so verification works offline:
supply `--trusted-root` with a Sigstore trusted-root file instead of relying on
`--offline` (removed in cosign 3.x).

## Linux package signing keys

Every `.rpm` and `.deb` we publish is signed with the current key below. The keyring at
`https://raw.githubusercontent.com/odigos-io/odigos/main/docs/keys/odigos-package-signing-keys.asc` always contains every key that signed a published package: point `gpgkey=` at it once
and never change it, rotation only appends.

| Fingerprint | Issued | Status | Notes |
| - | - | - | - |
| `6E80516AF71835E5FD12E3D8272251E89CC74CFF` | 2026-10-18 | current | initial package signing key (RUN-1042) |


# Telemetry Types
Source: https://docs.odigos.io/oss/telemetry-types



Odigos supports the producing and correlation of the following telemetry types:

* Traces
* Metrics
* Logs

For a telemetry type to be collected, a [destination](./backends-overview) that accepts it must be
configured. In addition, users can enable or disable the collection of a
telemetry type on a per-application basis. If none of the applications are
configured to collect a telemetry type, the telemetry type will not be
collected.

## Traces

Distributed traces collected by Odigos will automatically include spans for
popular open source projects such as HTTP clients and servers, gRPC clients and
servers, Database clients, and many more.

In addition, users can enrich their distributed traces with manually created
spans by using the relevant
[OpenTelemetry APIs](https://opentelemetry.io/docs/instrumentation/).

The collection of traces is achieved by combining two open source technologies:

* OpenTelemetry for languages with JIT compilation such as Python, Java,.NET and
  Javascript.
* eBPF for compiled languages such as Go.

Being based on popular open-source standards allows Odigos to automatically
support a huge number of libraries and frameworks. Below is a list of supported
libraries for every language:

> **[Go](https://github.com/odigos-io/opentelemetry-go-instrumentation/tree/master/pkg/instrumentors/bpf)**
>
> opentelemetry-go-instrumentation/ pkg/ instrumentors/ bpf/

> **[Javascript](https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node)**
>
> opentelemetry-js-contrib/ plugins/ node/

> **[Python](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation)**
>
> opentelemetry-python-contrib/ instrumentation/

> **[Java](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation)**
>
> opentelemetry-java-instrumentation/ instrumentation/

> **[PHP](https://github.com/open-telemetry/opentelemetry-php-contrib/tree/main/src/Instrumentation)**
>
> opentelemetry-php-contrib/ src/ Instrumentation/

> **[Ruby](https://github.com/open-telemetry/opentelemetry-ruby-contrib/tree/main/instrumentation)**
>
> opentelemetry-ruby-contrib/ instrumentation/

> **[.NET](https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/tree/main/docs/config.md#instrumented-traces-libraries-and-frameworks)**
>
> opentelemetry-dotnet-instrumentation/ docs/ config.md

## Metrics

There are three kinds of metrics that Odigos supports:

* Metrics related to the running of the application (number of HTTP requests,
  latency, DB connections, etc.)
* Metrics related to the language runtime (GC, threads, heap, etc.)
* Metrics related to the host environment (CPU, memory, disk, etc.)

### Applicative Metrics

Application-related metrics are derived from the distributed traces described
above. Many observability vendors automatically compute application metrics
according to the distributed traces. For example, the number of requests served
by the application is computed by counting the number of spans with the
`http.server` label. In case a destination that does not automatically compute
application metrics (like Prometheus + Tempo) is configured, Odigos will compute
them on its own by invoking the
[spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md)
connector.

### Runtime Metrics

Those metrics describe the behavior of the language runtime, such as garbage
collection, threads, heap, etc. Runtime metrics are collected by the same
mechanism that collects distributed traces.

### Host Metrics

Host-related metrics are collected via the
[host metrics receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver)

## Logs

Currently, Odigos will ship logs written by the application to the stdout or
stderr of the process. Correlation to other telemetry types will be done
according to timestamp and resource.

We are in the process of shipping logs using the same mechanism that collects
distributed traces, this would allow users to correlate logs to distributed
traces and metrics in a more accurate way.


# Usage report
Source: https://docs.odigos.io/oss/usage-reports



Odigos sends non-sensitive, non-personally identifiable information to our system.
This information helps Odigos maintainers to understand how the open-source community runs Odigos and
improve the project.

### Disabling the report

#### Using helm chart

When installing Odigos using helm chart, you can disable the report by setting `telemetry.enabled` to `false`.

#### Using CLI

When installing Odigos using CLI, you can disable the report by setting `--telemetry` to `false`.

### What is being reported

Odigos collects the following anonymous information:

* Are traces/metrics/logs enabled
* Selected destinations type (e.g. Jaeger, Prometheus, etc.)
* Number of instrumented applications by programming language
* Number of unrecognized applications
* Number of nodes


# ECS Configuration Reference
Source: https://docs.odigos.io/vmagent/ecs/configuration

Every environment variable of the Odigos ECS Agent and ECS Proxy, how platform names are derived, and where configuration is persisted.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

On ECS the **task definition is the configuration surface**. On a fresh
container instance it is the only input the agent has, so the Central endpoint
and any other defaults travel inside it — nothing depends on service discovery
or a pre-provisioned file.

Everything beyond bootstrap — [sources](/vmagent/setup/configuration/add-sources),
[destinations](/vmagent/setup/configuration/add-destinations),
[actions](/vmagent/setup/configuration/actions/overview), and
[instrumentation rules](/vmagent/setup/configuration/instrumentation-rules/overview) —
is managed from Odigos Central or with
[`odictl`](/vmagent/setup/configuration/odictl), exactly as on any VM Agent.

## Agent environment variables

| Variable | Required | Effect |
| - | - | - |
| `ODIGOS_DEFAULT_CONFIG` | Recommended | A full config document (YAML **or** JSON, same shape as `config.yaml`) that seeds the agent on a fresh node. This is where you set the Central endpoint: `{"central":{"enabled":true,"endpoint":"central.example.com:443"}}`. In Agent + Proxy mode, point it at the proxy instead. |
| `ODIGOS_ONPREM_TOKEN` | Yes, unless `license.jwt` is pre-provisioned | Written to `/etc/odigos-ecs-agent/license.jwt` at startup. Deliver it from AWS Secrets Manager via the task definition's `secrets` block, not as a plain `environment` value. |
| `ODIGOS_CENTRAL_ENDPOINT` | No | Shorthand for just the endpoint (`host:port` or URL) when you don't need a full document; equivalent to setting `central.enabled` + `central.endpoint`. `ODIGOS_DEFAULT_CONFIG` wins if both are set. |
| `ODIGOS_ECS_BEHIND_PROXY` | Agent + Proxy mode | `true` tells the agent it connects to an `odigos-ecs-proxy` rather than to Central. Set it in [Agent + Proxy](/vmagent/ecs/installation-with-proxy) mode; omit it in [Agent only](/vmagent/ecs/installation-agent-only) mode. An endpoint alone cannot distinguish the two, and the reported platform name differs per mode. |
| `ODIGOS_AGENT_NAME` | No | Platform name reported upstream, overriding the per-mode default below. |
| `ODIGOS_ECS_CLUSTER_NAME` | No | The ECS cluster name. Normally unnecessary — the agent reads the cluster from the ECS task metadata endpoint. Set it when metadata is unavailable, or to report a different name. |

> [!NOTE]
>
> `:443` in an endpoint implies `wss`. The connection is **outbound-only**, so a
> public Central endpoint or one in a different AWS account works with no inbound
> rules on the cluster.

### Platform naming per mode

The agent's name depends on the topology so each mode displays naturally in the Central UI:

| Mode | Platform name | Why |
| - | - | - |
| **Agent only** (no proxy) | the **ECS cluster name**, auto-detected | Each node registers with Central directly; naming the platform after the cluster is what makes a single-node cluster appear as the cluster rather than as an EC2 hostname. |
| **Agent + Proxy** | the **node hostname** | The proxy is what reports the cluster name upstream; it identifies its agents per node, and its coverage check matches agent names against the instances' private DNS. |

`ODIGOS_AGENT_NAME` overrides both. In Agent-only mode, if the cluster cannot be
determined (task metadata unavailable), the agent falls back to the hostname and
logs a warning.

Both modes register with platform type **`aws.ecs`**, so a cluster presents
consistently in Central either way.

### Configuration precedence and persistence

`ODIGOS_DEFAULT_CONFIG` is a **default**, not an override. Precedence, lowest to
highest:

**1. Built-in ECS distribution defaults**

The baseline shipped in the image.

**2. Discrete env vars**

`ODIGOS_CENTRAL_ENDPOINT`, `ODIGOS_AGENT_NAME`, and so on.

**3. ODIGOS\_DEFAULT\_CONFIG**

The inline document from the task definition.

**4. The persisted config.yaml**

Whatever Central or `odictl` has written. **This wins on every boot after the first.**

The config tree persists on the container **instance** at
`/etc/odigos-ecs-agent` (a host bind mount), so the agent keeps its deployment
identity and configuration across task restarts and image upgrades.

> [!WARNING]
>
> Because the persisted `config.yaml` takes precedence, **editing
> `ODIGOS_DEFAULT_CONFIG` does not change a value that has already been
> persisted**. To change a seeded value after first boot, update it through
> Central or `odictl` — or clear the persisted config on the instance so the new
> default is applied again.

## Proxy environment variables

| Variable | Required | Effect |
| - | - | - |
| `ODIGOS_CLUSTER_NAME` | Yes | The ECS cluster this proxy represents. Becomes Central's platform name and derives the proxy ID (`ecs-cluster/<name>`). |
| `ODIGOS_CENTRAL_ENDPOINT` | No | Central `host:port`. Empty runs the proxy downstream-only (agents still aggregate against it). |
| `ODIGOS_CENTRAL_SECURE_WS` | No | `true` forces `wss` upstream (also implied by `:443`). |
| `ODIGOS_PLATFORM_TYPE` | No | Platform type reported upstream. Defaults to `aws.ecs`, matching the per-node agent. |
| `ODIGOS_PROXY_ID` | No | Overrides the derived Central connection id. |
| `ODIGOS_PROXY_LISTEN` | No | Agent listen address. Default `:4321`. |
| `ODIGOS_PROXY_JOURNAL` | No | Config-journal **file** path. Default `/var/lib/odigos-ecs-proxy/journal.jsonl`. Point it at a file on an [EFS mount](/vmagent/ecs/installation-with-proxy#persist-the-config-journal-on-efs) if the journal must survive task replacement. |
| `ODIGOS_COVERAGE_CHECK` | No | `true` enables the expected-vs-connected check: the proxy lists the cluster's container instances through the ECS/EC2 APIs and reports ACTIVE instances that have no connected agent. Requires `ecs:ListContainerInstances`, `ecs:DescribeContainerInstances`, and `ec2:DescribeInstances` on the task role. |
| `ODIGOS_COVERAGE_INTERVAL` | No | Coverage check period (Go duration). Default `60s`. |

The proxy is unprivileged — no host PID, no Docker socket, no host mounts.

### What the proxy does with requests

| Request | Routing |
| - | - |
| Config queries (`sources`, `destinations`, `actions`, `instrumentationRules`, `config`, …) | Any one healthy agent — the config is identical on every node. |
| Live-state queries (`workloadReports`, counters) | Fan out to all agents; lists merged. |
| Mixed queries (config + live fields in one operation) | Fan out; live lists concatenate, config fields keep a single agent's value so they are never multiplied by node count. |
| Config mutations (`createSource`, `updateDestination`, …) | Broadcast to every agent **and journaled**. |
| Live mutations (`restartSource`, …) | Broadcast, not journaled. |
| Profiling jobs, diagnose bundles | One agent. |

The journal replays config mutations to agents that connect later, so a node
that joins the cluster — or a task that migrates to a fresh instance — converges
to the cluster configuration automatically. Each agent's replay position is
tracked by its stable deployment id, so a reconnecting agent replays only what it
missed.

### Membership and lifecycle events

* Agent connects and disconnects are logged and forwarded to Central as
  `ecs-proxy.agent-lifecycle` events; departed agents keep a tombstone (id, node,
  last seen).
* Coverage changes are logged and forwarded as `ecs-proxy.coverage` events.
* `GET /healthz` on port 4321 returns the whole view: connected agents, departed
  tombstones, and the last coverage result. See
  [checking the healthz endpoint](/vmagent/ecs/troubleshooting#check-the-proxy-healthz-endpoint).

## What gets instrumented

| | Behavior on ECS |
| - | - |
| ECS-managed containers | **Instrumented.** Discovered through the instance's Docker daemon. |
| Host processes and `systemd` services on the container instance | **Ignored.** Unlike the [VM Agent](/vmagent/setup/installation) and [Docker deployment](/vmagent/setup/docker), the ECS distribution only classifies ECS tasks. |
| Task restarts | **Never.** All instrumentation is direct attach; ECS owns the container lifecycle, so the agent does not restart, recreate, or modify tasks. |
| Java, Go, C++ | Their dynamic eBPF SDKs (Java via dynamic attach). |
| Python, Node.js, .NET, PHP, Ruby, others | Covered by [OBI](/oss/instrumentations/obi), which also attaches with no restart. Restart-based environment injection is intentionally not used on ECS. |

The agent's own binaries, the `amazon-ecs-agent`, the SSM agent, and `awsvpc`
pause containers are excluded from instrumentation automatically.

## How the collector runs

As in the [Docker deployment](/vmagent/setup/docker#how-the-collector-runs),
there is no `systemd` inside the container:

* `odigos-otelcol` runs as a **supervised child process** of the agent, so its
  lifecycle follows the agent task.
* Agent and collector **share the task's stdout**, captured by the `awslogs`
  driver into the `/odigos/ecs-agent` log group.
* With host networking, the collector and OpAMP server listen on the instance's
  network namespace, reachable by instrumented tasks on localhost or the instance
  IP.
* Collector logs are JSON-formatted on ECS; the `awslogs` driver ships them to
  CloudWatch Logs as the primary log stream.
* [systemd memory limits](/vmagent/setup/configuration/services-mem-limits-configuration)
  do not apply — cap the agent with the task definition's `cpu`, `memory`, and
  `memoryReservation` instead.

## Instrumentation file sync

The image bakes the Java instrumentation files into a staging directory. On
startup the agent rsyncs them onto the container instance's `/var/odigos` (the
task definition bind mount), and seeds them into a workload's container namespace
when it attaches. Go, C++, and the OBI fallback are compiled into the agent
binary and need no on-disk files.

This is why `/var/odigos` must be a writable host mount. The full list of
elevated task settings is under
[required permissions](/vmagent/ecs/requirements#required-permissions).


# Install on ECS: Single Node (Agent only)
Source: https://docs.odigos.io/vmagent/ecs/installation-agent-only

Deploy the Odigos ECS Agent as a daemon service. Every container instance runs one agent that connects directly to Odigos Central — no extra AWS resources.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

This is the default and simplest ECS deployment: **one agent task per EC2
container instance**, each connecting **directly** to Odigos Central. No proxy,
no load balancer, no service discovery.

It works identically whether the cluster has 1 node or 50 — the only difference
is that **each node appears in Central as its own platform**. If you want the
whole cluster to appear as a single platform with an aggregated workload list,
use [Install: Agent + Proxy](/vmagent/ecs/installation-with-proxy) instead.

> [!NOTE]
>
> Complete the [System Requirements](/vmagent/ecs/requirements) first — in
> particular the **EC2 launch type** and the **kernel 5.10+ with BTF**
> requirement (use the Amazon Linux 2023 ECS-optimized AMI).

## Step 1: Set your session variables

Run every command in this guide in the **same shell**.

```bash theme={null}
export AWS_REGION=<region>
export CLUSTER=<your-ecs-cluster-name>
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
export REG=${ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
export TAG=<agent-image-tag>
export CENTRAL_ENDPOINT='<central-host>:<port>'   # e.g. central.example.com:443 (:443 implies wss)

echo "ACCOUNT=$ACCOUNT REG=$REG"   # neither may be empty before continuing
```

## Step 2: Store the license token in Secrets Manager

```bash theme={null}
export SECRET_ARN=$(aws secretsmanager create-secret --name odigos-onprem-token \
  --secret-string "<YOUR_ODIGOS_ONPREM_TOKEN>" --query ARN --output text 2>/dev/null \
  || aws secretsmanager describe-secret --secret-id odigos-onprem-token --query ARN --output text)

echo "SECRET_ARN=$SECRET_ARN"   # must be non-empty
```

That create-or-fetch form is idempotent — safe to re-run if the secret already
exists.

> **Alternative: pass the token as a plain environment variable**
>
> If IAM or an AWS Organizations SCP blocks `secretsmanager:GetSecretValue`, you
> can put the token directly in the task definition instead: drop the `secrets`
> block and add `ODIGOS_ONPREM_TOKEN` to `environment`.
>
> > [!WARNING]
> >
> > The token then becomes readable to anyone who can call
> > `ecs:DescribeTaskDefinition`. Acceptable for a throwaway test cluster, not
> > for production.

## Step 3: Create the IAM roles

```bash theme={null}
TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ecs-tasks.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

# Execution role: image pull, logs, and the license secret.
aws iam create-role --role-name odigos-ecs-exec-role --assume-role-policy-document "$TRUST"
aws iam attach-role-policy --role-name odigos-ecs-exec-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam put-role-policy --role-name odigos-ecs-exec-role --policy-name odigos-extras \
  --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"logs:CreateLogGroup\",\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":\"secretsmanager:GetSecretValue\",\"Resource\":\"arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT}:secret:odigos-onprem-token-*\"}]}"

# Task role: needed for odictl through ECS Exec.
aws iam create-role --role-name odigos-ecs-task-role --assume-role-policy-document "$TRUST"
aws iam put-role-policy --role-name odigos-ecs-task-role --policy-name ssm-exec \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssmmessages:CreateControlChannel","ssmmessages:CreateDataChannel","ssmmessages:OpenControlChannel","ssmmessages:OpenDataChannel"],"Resource":"*"}]}'
```

> [!NOTE]
>
> Note the trailing `-*` on the secret ARN in the policy: Secrets Manager
> appends a random 6-character suffix to every secret, and that suffix changes
> if the secret is deleted and recreated. The wildcard means you never have to
> touch this policy again.
>
> `EntityAlreadyExists` on `create-role` means the role exists from a previous
> attempt — safe to continue with the `attach-role-policy` / `put-role-policy`
> commands.

## Step 4: Register the task definition

Save the task definition below as `odigos-ecs-agent.json`, then substitute your
values.

> **odigos-ecs-agent.json**
>
> ```json theme={null}
> {
>   "family": "odigos-ecs-agent",
>   "requiresCompatibilities": ["EC2"],
>   "networkMode": "host",
>   "pidMode": "host",
>   "executionRoleArn": "<EXEC_ROLE_ARN>",
>   "taskRoleArn": "<TASK_ROLE_ARN>",
>   "volumes": [
>     { "name": "docker-sock", "host": { "sourcePath": "/var/run/docker.sock" } },
>     { "name": "docker-lib", "host": { "sourcePath": "/var/lib/docker" } },
>     { "name": "sys-kernel-debug", "host": { "sourcePath": "/sys/kernel/debug" } },
>     { "name": "odigos-config", "host": { "sourcePath": "/etc/odigos-ecs-agent" } },
>     { "name": "odigos-instrumentations", "host": { "sourcePath": "/var/odigos" } }
>   ],
>   "containerDefinitions": [
>     {
>       "name": "odigos-ecs-agent",
>       "image": "<AGENT_IMAGE_URI>",
>       "essential": true,
>       "privileged": true,
>       "cpu": 512,
>       "memoryReservation": 512,
>       "memory": 2048,
>       "stopTimeout": 120,
>       "mountPoints": [
>         { "sourceVolume": "docker-sock", "containerPath": "/var/run/docker.sock", "readOnly": true },
>         { "sourceVolume": "docker-lib", "containerPath": "/var/lib/docker", "readOnly": true },
>         { "sourceVolume": "sys-kernel-debug", "containerPath": "/sys/kernel/debug", "readOnly": false },
>         { "sourceVolume": "odigos-config", "containerPath": "/etc/odigos-ecs-agent", "readOnly": false },
>         { "sourceVolume": "odigos-instrumentations", "containerPath": "/var/odigos", "readOnly": false }
>       ],
>       "environment": [
>         {
>           "name": "ODIGOS_DEFAULT_CONFIG",
>           "value": "{\"central\":{\"enabled\":true,\"endpoint\":\"<CENTRAL_ENDPOINT>\"}}"
>         }
>       ],
>       "secrets": [
>         { "name": "ODIGOS_ONPREM_TOKEN", "valueFrom": "<TOKEN_SECRET_ARN>" }
>       ],
>       "ulimits": [
>         { "name": "memlock", "softLimit": -1, "hardLimit": -1 },
>         { "name": "nofile", "softLimit": 65536, "hardLimit": 65536 }
>       ],
>       "linuxParameters": { "initProcessEnabled": true },
>       "logConfiguration": {
>         "logDriver": "awslogs",
>         "options": {
>           "awslogs-group": "/odigos/ecs-agent",
>           "awslogs-create-group": "true",
>           "awslogs-region": "<REGION>",
>           "awslogs-stream-prefix": "odigos-ecs-agent"
>         }
>       }
>     }
>   ]
> }
> ```
>
> All settings are required for specific capabilities. For details, see
> [required permissions](/vmagent/ecs/requirements#required-permissions).

Substitute the placeholders:

```bash theme={null}
sed -e "s|<AGENT_IMAGE_URI>|${REG}/odigos-ecs-agent:${TAG}|" \
    -e "s|<CENTRAL_ENDPOINT>|${CENTRAL_ENDPOINT}|" \
    -e "s|<TOKEN_SECRET_ARN>|${SECRET_ARN}|" \
    -e "s|<EXEC_ROLE_ARN>|arn:aws:iam::${ACCOUNT}:role/odigos-ecs-exec-role|" \
    -e "s|<TASK_ROLE_ARN>|arn:aws:iam::${ACCOUNT}:role/odigos-ecs-task-role|" \
    -e "s|<REGION>|${AWS_REGION}|" \
    odigos-ecs-agent.json > /tmp/agent-td.json
```

> [!WARNING]
>
> **Verify before registering.** A leftover `<placeholder>` produces a task that
> starts fine but can never connect — and the failure only shows up in the logs
> minutes later.
>
> ```bash theme={null}
> grep -c '<' /tmp/agent-td.json   # want 0
> jq -r '.containerDefinitions[0].environment[] | select(.name=="ODIGOS_DEFAULT_CONFIG").value' /tmp/agent-td.json | jq .
> # → {"central": {"enabled": true, "endpoint": "central.example.com:443"}}
> #   confirm this is your REAL endpoint
> ```

Register it:

```bash theme={null}
aws ecs register-task-definition --cli-input-json file:///tmp/agent-td.json >/dev/null && echo "task definition registered"
```

## Step 5: Create the daemon service

The `DAEMON` scheduling strategy runs exactly one agent task per container
instance, including on instances that join the cluster later.

```bash theme={null}
aws ecs create-service \
  --cluster "$CLUSTER" \
  --service-name odigos-ecs-agent \
  --task-definition odigos-ecs-agent \
  --scheduling-strategy DAEMON \
  --launch-type EC2 \
  --enable-execute-command
```

> [!NOTE]
>
> `create-service` is a **one-time** operation — re-running it fails with
> `Creation of service was not idempotent`. To roll out a new image or an env-var
> change later, register a new task definition revision and run
> [`update-service`](/vmagent/ecs/maintenance#upgrade) instead.
>
> `--enable-execute-command` is what lets you later run `odictl` inside the
> agent container.

## Step 6: Verify

**1. The daemon is placed on every instance**

```bash theme={null}
aws ecs describe-services --cluster "$CLUSTER" --services odigos-ecs-agent \
  --query 'services[0].{running:runningCount,desired:desiredCount,events:events[:3].message}'
```

Wait until `running == desired`. Any placement failures appear in `events` —
see [Troubleshooting](/vmagent/ecs/troubleshooting).

**2. The instance kernel is supported**

```bash theme={null}
TASK=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name odigos-ecs-agent \
  --desired-status RUNNING --query 'taskArns[0]' --output text)

aws ecs execute-command --cluster "$CLUSTER" --task "$TASK" \
  --container odigos-ecs-agent --interactive --command "uname -r"
```

Expect `6.1.x` (or any 5.10+). `4.14.x` means the instance is on the Amazon
Linux 2 stock AMI and **nothing will be instrumented** — see
[System Requirements](/vmagent/ecs/requirements#container-instance-kernel-and-ami).

**3. The agent is healthy and connected**

```bash theme={null}
aws logs tail /odigos/ecs-agent --since 5m --follow
```

You want to see the agent connect to Central, and you want **no**
`runtime-detector stopped` errors.

**4. The platform appears in Odigos Central**

Open the Central UI. You should see **one platform per container instance**,
each named after the ECS cluster.

Your ECS tasks written in a supported language appear as discovered
workloads, ready to be turned into
[sources](/vmagent/setup/configuration/add-sources).

## Step 7: Build your pipeline

The ECS Agent is configured exactly like any other VM Agent — from Odigos
Central, or with `odictl` inside the agent container:

```bash theme={null}
aws ecs execute-command --cluster "$CLUSTER" --task "$TASK" \
  --container odigos-ecs-agent --interactive --command odictl
```

> **[Add Sources](/vmagent/setup/configuration/add-sources)**
>
> Choose which ECS tasks to instrument.

> **[Add Destinations](/vmagent/setup/configuration/add-destinations)**
>
> Send the telemetry to your observability backend.

> **[Actions](/vmagent/setup/configuration/actions/overview)**
>
> Transform, filter, and enrich telemetry.

> **[Configuration Reference](/vmagent/ecs/configuration)**
>
> ECS-specific environment variables and permissions.

> [!NOTE]
>
> In this mode `odictl` targets **single node's** agent — there is no cluster-wide
> `odictl`. Configuration made through Central applies to the platform you
> selected, i.e. that node. For cluster-wide configuration, use
> [Agent + Proxy](/vmagent/ecs/installation-with-proxy).

## Moving to Agent + Proxy later

Nothing is thrown away. Deploy the proxy, re-point the agents, and roll the
daemon — no reinstall. Follow
[Switch to Multi Node](/vmagent/ecs/installation-switch-to-multi-node).


# Switch from Single Node to Multi Node
Source: https://docs.odigos.io/vmagent/ecs/installation-switch-to-multi-node

Move an existing Agent-only ECS deployment to Agent + Proxy so the cluster appears in Odigos Central as one platform — without reinstalling the agents.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

No reinstall. The agents already running from
[Install: Single Node](/vmagent/ecs/installation-agent-only) speak the same
protocol to the proxy that they speak to Central. You add the proxy, re-point
the agents at it, and roll the daemon.

After the switch, Central shows **one** platform named after the ECS cluster,
replacing the per-node platforms.

> [!NOTE]
>
> You need the **`odigos-ecs-proxy`** image in addition to the agent image you
> already run. Mirror it the same way as in
> [System Requirements](/vmagent/ecs/requirements#license-and-images).

## Switch

**1. Deploy the proxy**

Follow [Install: Multi Node](/vmagent/ecs/installation-with-proxy) steps
3–5 (stable address, proxy task definition, proxy service). Prefer the
internal NLB; a pinned instance is test/dev only.

Do **not** create a second agent service — keep the daemon you already have.

**2. Re-point the agents**

In the agent task definition, change `ODIGOS_DEFAULT_CONFIG`'s endpoint to
the proxy's `host:4321` **and** add `ODIGOS_ECS_BEHIND_PROXY=true`. Register
the new revision.

The endpoint alone cannot tell a proxy from Central. Without
`ODIGOS_ECS_BEHIND_PROXY=true`, every agent still registers under the
cluster name and the coverage check reports all instances as uncovered. See
[platform naming](/vmagent/ecs/configuration#platform-naming-per-mode).

**3. Roll the daemon**

```bash theme={null}
aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-agent \
  --task-definition odigos-ecs-agent --force-new-deployment
```

**4. Verify**

Use [Install: Multi Node, Step 8](/vmagent/ecs/installation-with-proxy#step-8-verify):
agents connected to the proxy, `/healthz` lists every node, and Central
shows one platform for the cluster.

Values already persisted in each node's `config.yaml`
[take precedence](/vmagent/ecs/configuration#configuration-precedence-and-persistence)
over `ODIGOS_DEFAULT_CONFIG`. After the first boot behind the proxy, change the
endpoint through Central or `odictl` if a node still points at Central.

## Switch back to single node

The same flow in reverse:

1. Point `ODIGOS_DEFAULT_CONFIG`'s endpoint back at Central.
2. Remove `ODIGOS_ECS_BEHIND_PROXY` (or set it to `false`).
3. Register a new revision and `update-service --force-new-deployment`.
4. Delete the proxy service.

Each node then appears in Central as its own platform again.

## Next steps

> **[Install: Multi Node](/vmagent/ecs/installation-with-proxy)**
>
> Full Agent + Proxy install if you are starting from scratch.

> **[Configuration Reference](/vmagent/ecs/configuration)**
>
> `ODIGOS_ECS_BEHIND_PROXY` and the proxy environment variables.


# Install on ECS: Multi Node (Agent + Proxy)
Source: https://docs.odigos.io/vmagent/ecs/installation-with-proxy

Deploy the Odigos ECS Agent daemon plus the cluster-scoped ECS Proxy, so a multi-node ECS cluster appears in Odigos Central as a single platform with aggregated workloads.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

This deployment runs the agent daemon on every container instance **plus one
`odigos-ecs-proxy` task per cluster**. The agents dial the proxy instead of
Central, so the proxy is the cluster's single upstream connection and the whole
cluster appears in Odigos Central as **one** platform — with an aggregated
workload list, cluster-wide configuration, a config journal for nodes that join
later, and an optional coverage check.

This guide is a **superset** of [Install: Agent Only](/vmagent/ecs/installation-agent-only):
same agent, same task definition, plus one component. If you don't need
cluster-wide aggregation, that guide is simpler.

> [!NOTE]
>
> Complete the [System Requirements](/vmagent/ecs/requirements) first. You need
> **both** images here: `odigos-ecs-agent` and `odigos-ecs-proxy`.

## Deployment order

Deploy the **proxy first**, then the agents. That way the agents connect on
their very first boot, with the correct proxy address baked into their first task
definition revision — no retry-loop noise and no redeploy.

**1. Choose the proxy's stable address**

Internal NLB (preferred), or a pinned instance for test/dev.

**2. Deploy the proxy**

One task per cluster, listening on TCP 4321.

**3. Deploy the agent daemon**

Enable the proxy, with `ODIGOS_ECS_BEHIND_PROXY=true`.

## Step 1: Set your session variables

Run every command in this guide in the **same shell**.

```bash theme={null}
export AWS_REGION=<region>
export CLUSTER=<your-ecs-cluster-name>
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
export REG=${ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
export TAG=<image-tag>
export CENTRAL_ENDPOINT='<central-host>:<port>'   # e.g. central.example.com:443

echo "ACCOUNT=$ACCOUNT REG=$REG"   # neither may be empty before continuing
```

## Step 2: License token and IAM roles

Identical to the Agent-only guide — the **agent** needs the license token; the
proxy never touches it. Both task definitions reuse the same two roles.

```bash theme={null}
export SECRET_ARN=$(aws secretsmanager create-secret --name odigos-onprem-token \
  --secret-string "<YOUR_ODIGOS_ONPREM_TOKEN>" --query ARN --output text 2>/dev/null \
  || aws secretsmanager describe-secret --secret-id odigos-onprem-token --query ARN --output text)
echo "SECRET_ARN=$SECRET_ARN"   # must be non-empty

TRUST='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ecs-tasks.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

aws iam create-role --role-name odigos-ecs-exec-role --assume-role-policy-document "$TRUST"
aws iam attach-role-policy --role-name odigos-ecs-exec-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam put-role-policy --role-name odigos-ecs-exec-role --policy-name odigos-extras \
  --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":\"logs:CreateLogGroup\",\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":\"secretsmanager:GetSecretValue\",\"Resource\":\"arn:aws:secretsmanager:${AWS_REGION}:${ACCOUNT}:secret:odigos-onprem-token-*\"}]}"

aws iam create-role --role-name odigos-ecs-task-role --assume-role-policy-document "$TRUST"
aws iam put-role-policy --role-name odigos-ecs-task-role --policy-name ssm-exec \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ssmmessages:CreateControlChannel","ssmmessages:CreateDataChannel","ssmmessages:OpenControlChannel","ssmmessages:OpenDataChannel"],"Resource":"*"}]}'
```

> **Add the coverage-check permissions to the task role (optional)**
>
> The proxy's [coverage check](/vmagent/ecs/configuration#proxy-environment-variables)
> reports ACTIVE container instances that have **no** agent connected — the check
> that catches a daemon that failed to place, a crash-looping agent, or a license
> failure. It calls the ECS/EC2 inventory APIs:
>
> ```bash theme={null}
> aws iam put-role-policy --role-name odigos-ecs-task-role --policy-name odigos-coverage \
>   --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["ecs:ListContainerInstances","ecs:DescribeContainerInstances","ec2:DescribeInstances"],"Resource":"*"}]}'
> ```
>
> Without these permissions, leave `ODIGOS_COVERAGE_CHECK` unset — everything
> else works.

## Step 3: Give the proxy a stable address

The agents need one fixed `host:4321` reachable from the container instances.
That is the proxy's **only** networking requirement — no Cloud Map, no service
discovery. Pick one of two options.

> [!WARNING]
>
> **Option B (pinned instance) is for test and development only.** If that
> instance is replaced, the proxy's address changes and you must re-point both
> the placement constraint and the agents. See
> [the pinned-IP caveat](#caveat-the-pinned-proxy-address). Use Option A
> (internal NLB) in production.

**Option A: Internal NLB (preferred)**

An internal Network Load Balancer is the production method: it survives
instance and task replacement. The proxy runs with `awsvpc` networking and
registers its ENI IP in the target group.

```bash theme={null}
aws elbv2 create-load-balancer --name odigos-ecs-proxy --type network \
  --scheme internal --subnets <SUBNET_IDS>

aws elbv2 create-target-group --name odigos-ecs-proxy --protocol TCP \
  --port 4321 --target-type ip --vpc-id <VPC_ID>

aws elbv2 create-listener --load-balancer-arn <LB_ARN> --protocol TCP \
  --port 4321 --default-actions Type=forward,TargetGroupArn=<TARGET_GROUP_ARN>
```

The NLB DNS name is the agents' endpoint:

```bash theme={null}
export PROXY_ENDPOINT='odigos-ecs-proxy-xxxx.elb.<REGION>.amazonaws.com:4321'
```

Keep `"networkMode": "awsvpc"` and the `portMappings` block in the proxy task
definition below.

**Option B: Pinned instance (test/dev)**

Run the proxy with host networking on one designated container instance; its
private IP is the endpoint. No extra AWS resources, but the address is tied
to that instance.

First confirm the cluster actually has a registered container instance:

```bash theme={null}
aws ecs describe-clusters --clusters "$CLUSTER" \
  --query 'clusters[0].{status:status,registered:registeredContainerInstancesCount}'
```

If `registered` is `0`, wait for the instances to boot and register:

```bash theme={null}
until [ "$(aws ecs describe-clusters --clusters "$CLUSTER" --query 'clusters[0].registeredContainerInstancesCount' --output text)" -ge 1 ]; do
  echo waiting...; sleep 10
done
```

Then pick the instance to pin to:

```bash theme={null}
CI=$(aws ecs list-container-instances --cluster "$CLUSTER" --query 'containerInstanceArns[0]' --output text)
if [ -z "$CI" ] || [ "$CI" = "None" ]; then
  echo "no container instances registered in $CLUSTER yet" >&2
else
  export EC2_ID=$(aws ecs describe-container-instances --cluster "$CLUSTER" --container-instances "$CI" \
    --query 'containerInstances[0].ec2InstanceId' --output text)
  export PROXY_ENDPOINT="$(aws ec2 describe-instances --instance-ids "$EC2_ID" \
    --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text):4321"
  echo "proxy → $EC2_ID @ $PROXY_ENDPOINT"
fi
```

> [!WARNING]
>
> Don't skip the guard. With no registered instances,
> `list-container-instances` returns the literal string `None`, `EC2_ID` ends
> up empty — and an **empty** `--instance-ids` makes `describe-instances`
> query every instance in the region, silently handing you the private IP of
> an unrelated EC2 instance instead of failing.

> [!NOTE]
>
> The container instances' security group must allow TCP **4321** — from the
> NLB (Option A) or from the other instances (Option B).

## Step 4: Register the proxy task definition

Save this as `odigos-ecs-proxy.json`.

> **odigos-ecs-proxy.json**
>
> ```json theme={null}
> {
>   "family": "odigos-ecs-proxy",
>   "requiresCompatibilities": ["EC2"],
>   "networkMode": "awsvpc",
>   "cpu": "256",
>   "memory": "512",
>   "executionRoleArn": "<EXEC_ROLE_ARN>",
>   "taskRoleArn": "<TASK_ROLE_ARN>",
>   "containerDefinitions": [
>     {
>       "name": "odigos-ecs-proxy",
>       "image": "<PROXY_IMAGE_URI>",
>       "essential": true,
>       "portMappings": [
>         { "containerPort": 4321, "protocol": "tcp" }
>       ],
>       "environment": [
>         { "name": "ODIGOS_CLUSTER_NAME", "value": "<CLUSTER>" },
>         { "name": "ODIGOS_CENTRAL_ENDPOINT", "value": "<CENTRAL_ENDPOINT>" },
>         { "name": "ODIGOS_PLATFORM_TYPE", "value": "aws.ecs" },
>         { "name": "ODIGOS_COVERAGE_CHECK", "value": "true" }
>       ],
>       "healthCheck": {
>         "command": ["CMD-SHELL", "bash -c '</dev/tcp/127.0.0.1/4321'"],
>         "interval": 30,
>         "timeout": 5,
>         "retries": 3,
>         "startPeriod": 10
>       },
>       "logConfiguration": {
>         "logDriver": "awslogs",
>         "options": {
>           "awslogs-group": "/odigos/ecs-proxy",
>           "awslogs-create-group": "true",
>           "awslogs-region": "<REGION>",
>           "awslogs-stream-prefix": "odigos-ecs-proxy"
>         }
>       }
>     }
>   ]
> }
> ```
>
> > [!WARNING]
> >
> > **Option A (NLB) keeps the file as-is.** Option B (pinned instance) requires
> > two edits: change `"networkMode": "awsvpc"` to `"host"`, and remove the
> > `portMappings` block.
>
> `ODIGOS_CLUSTER_NAME` becomes the platform name in Central. Drop
> `ODIGOS_COVERAGE_CHECK` if you did not grant the coverage permissions in
> Step 2. See the [proxy environment variables](/vmagent/ecs/configuration#proxy-environment-variables)
> for the full list. To keep the config journal across task replacement, add
> the EFS volume shown in [persist the config journal on EFS](#persist-the-config-journal-on-efs).

```bash theme={null}
sed -e "s|<PROXY_IMAGE_URI>|${REG}/odigos-ecs-proxy:${TAG}|" \
    -e "s|<CLUSTER>|${CLUSTER}|" \
    -e "s|<CENTRAL_ENDPOINT>|${CENTRAL_ENDPOINT}|" \
    -e "s|<EXEC_ROLE_ARN>|arn:aws:iam::${ACCOUNT}:role/odigos-ecs-exec-role|" \
    -e "s|<TASK_ROLE_ARN>|arn:aws:iam::${ACCOUNT}:role/odigos-ecs-task-role|" \
    -e "s|<REGION>|${AWS_REGION}|" \
    odigos-ecs-proxy.json > /tmp/proxy-td.json

grep -c '<' /tmp/proxy-td.json   # want 0
jq -r '.containerDefinitions[0].environment[] | select(.name=="ODIGOS_CENTRAL_ENDPOINT").value' /tmp/proxy-td.json
# confirm this is your REAL Central host:port, not a placeholder

aws ecs register-task-definition --cli-input-json file:///tmp/proxy-td.json >/dev/null && echo "proxy task definition registered"
```

## Step 5: Create the proxy service

The proxy is a **singleton** — one task per cluster. Use the tab for the address
option you chose in [Step 3](#step-3-give-the-proxy-a-stable-address).

**Option A: Internal NLB**

```bash theme={null}
aws ecs create-service \
  --cluster "$CLUSTER" \
  --service-name odigos-ecs-proxy \
  --task-definition odigos-ecs-proxy \
  --desired-count 1 \
  --launch-type EC2 \
  --enable-execute-command \
  --network-configuration "awsvpcConfiguration={subnets=[<SUBNET_IDS>],securityGroups=[<SG_ID>]}" \
  --load-balancers "targetGroupArn=<TARGET_GROUP_ARN>,containerName=odigos-ecs-proxy,containerPort=4321"
```

**Option B: Pinned instance**

The pinned proxy needs three flags a normal service doesn't:

1. **A JSON placement constraint.** The AWS CLI's shorthand parser cannot
   handle `in [...]` expressions (it fails with `Expected ',', received: '''`).
   Pass pure JSON:

   ```bash theme={null}
   PLACEMENT="[{\"type\":\"memberOf\",\"expression\":\"ec2InstanceId == '${EC2_ID}'\"}]"
   ```

2. **A stop-then-start deployment configuration.** ECS's default rollout
   starts the new task before stopping the old one. With host networking both
   tasks want port `4321` on the same instance, so the new task can never be
   placed and the deployment retries indefinitely.

3. **Availability Zone Rebalancing disabled.** AZ rebalancing defaults to
   `ENABLED` on new services and explicitly rejects `maximumPercent <= 100`,
   failing with `Availability Zone Rebalancing does not support maximumPercent <= 100%`.

```bash theme={null}
aws ecs create-service \
  --cluster "$CLUSTER" \
  --service-name odigos-ecs-proxy \
  --task-definition odigos-ecs-proxy \
  --desired-count 1 \
  --launch-type EC2 \
  --enable-execute-command \
  --placement-constraints "$PLACEMENT" \
  --deployment-configuration "minimumHealthyPercent=0,maximumPercent=100" \
  --availability-zone-rebalancing DISABLED
```

Wait for the proxy to come up and connect to Central before continuing:

```bash theme={null}
until [ "$(aws ecs describe-services --cluster "$CLUSTER" --services odigos-ecs-proxy --query 'services[0].runningCount' --output text)" = "1" ]; do
  echo "waiting for proxy..."; sleep 10
done

aws logs tail /odigos/ecs-proxy --since 2m --filter-pattern central
# want: "Connected to central"
```

## Step 6: Register the agent task definition

Use the **same task definition** as the Agent-only guide, with two differences:
the endpoint is the proxy's, and `ODIGOS_ECS_BEHIND_PROXY` is `true`.

> **odigos-ecs-agent.json (proxy mode)**
>
> ```json theme={null}
> {
>   "family": "odigos-ecs-agent",
>   "requiresCompatibilities": ["EC2"],
>   "networkMode": "host",
>   "pidMode": "host",
>   "executionRoleArn": "<EXEC_ROLE_ARN>",
>   "taskRoleArn": "<TASK_ROLE_ARN>",
>   "volumes": [
>     { "name": "docker-sock", "host": { "sourcePath": "/var/run/docker.sock" } },
>     { "name": "docker-lib", "host": { "sourcePath": "/var/lib/docker" } },
>     { "name": "sys-kernel-debug", "host": { "sourcePath": "/sys/kernel/debug" } },
>     { "name": "odigos-config", "host": { "sourcePath": "/etc/odigos-ecs-agent" } },
>     { "name": "odigos-instrumentations", "host": { "sourcePath": "/var/odigos" } }
>   ],
>   "containerDefinitions": [
>     {
>       "name": "odigos-ecs-agent",
>       "image": "<AGENT_IMAGE_URI>",
>       "essential": true,
>       "privileged": true,
>       "cpu": 512,
>       "memoryReservation": 512,
>       "memory": 2048,
>       "stopTimeout": 120,
>       "mountPoints": [
>         { "sourceVolume": "docker-sock", "containerPath": "/var/run/docker.sock", "readOnly": true },
>         { "sourceVolume": "docker-lib", "containerPath": "/var/lib/docker", "readOnly": true },
>         { "sourceVolume": "sys-kernel-debug", "containerPath": "/sys/kernel/debug", "readOnly": false },
>         { "sourceVolume": "odigos-config", "containerPath": "/etc/odigos-ecs-agent", "readOnly": false },
>         { "sourceVolume": "odigos-instrumentations", "containerPath": "/var/odigos", "readOnly": false }
>       ],
>       "environment": [
>         {
>           "name": "ODIGOS_DEFAULT_CONFIG",
>           "value": "{\"central\":{\"enabled\":true,\"endpoint\":\"<PROXY_ENDPOINT>\"}}"
>         },
>         { "name": "ODIGOS_ECS_BEHIND_PROXY", "value": "true" }
>       ],
>       "secrets": [
>         { "name": "ODIGOS_ONPREM_TOKEN", "valueFrom": "<TOKEN_SECRET_ARN>" }
>       ],
>       "ulimits": [
>         { "name": "memlock", "softLimit": -1, "hardLimit": -1 },
>         { "name": "nofile", "softLimit": 65536, "hardLimit": 65536 }
>       ],
>       "linuxParameters": { "initProcessEnabled": true },
>       "logConfiguration": {
>         "logDriver": "awslogs",
>         "options": {
>           "awslogs-group": "/odigos/ecs-agent",
>           "awslogs-create-group": "true",
>           "awslogs-region": "<REGION>",
>           "awslogs-stream-prefix": "odigos-ecs-agent"
>         }
>       }
>     }
>   ]
> }
> ```

```bash theme={null}
sed -e "s|<AGENT_IMAGE_URI>|${REG}/odigos-ecs-agent:${TAG}|" \
    -e "s|<PROXY_ENDPOINT>|${PROXY_ENDPOINT}|" \
    -e "s|<TOKEN_SECRET_ARN>|${SECRET_ARN}|" \
    -e "s|<EXEC_ROLE_ARN>|arn:aws:iam::${ACCOUNT}:role/odigos-ecs-exec-role|" \
    -e "s|<TASK_ROLE_ARN>|arn:aws:iam::${ACCOUNT}:role/odigos-ecs-task-role|" \
    -e "s|<REGION>|${AWS_REGION}|" \
    odigos-ecs-agent.json > /tmp/agent-td.json

grep -c '<' /tmp/agent-td.json   # want 0
jq -r '.containerDefinitions[0].environment[] | select(.name=="ODIGOS_DEFAULT_CONFIG").value' /tmp/agent-td.json | jq .
# → {"central": {"enabled": true, "endpoint": "10.0.1.23:4321"}}  ← the PROXY, not Central

aws ecs register-task-definition --cli-input-json file:///tmp/agent-td.json >/dev/null && echo "agent task definition registered"
```

> [!WARNING]
>
> **`ODIGOS_ECS_BEHIND_PROXY=true` matters.** The agent cannot tell a proxy
> address from a Central address — both are just `host:port` — and the two modes
> report different platform names. Behind the proxy, each agent identifies itself
> by **node hostname** (the proxy is what reports the cluster name upstream, and
> its coverage check matches agent names against the instances' private DNS).
> If you leave it `false` here and every agent registers under the cluster name instead,
> which makes the coverage check report **all** instances as uncovered.

## Step 7: Create the agent daemon service

```bash theme={null}
aws ecs create-service \
  --cluster "$CLUSTER" \
  --service-name odigos-ecs-agent \
  --task-definition odigos-ecs-agent \
  --scheduling-strategy DAEMON \
  --launch-type EC2 \
  --enable-execute-command
```

Because the proxy was already live, agents connect on their first boot. For
later changes, use [`update-service`](/vmagent/ecs/maintenance#upgrade) —
`create-service` is one-time.

## Step 8: Verify

**1. Every instance runs an agent, on a supported kernel**

```bash theme={null}
aws ecs describe-services --cluster "$CLUSTER" --services odigos-ecs-agent \
  --query 'services[0].{running:runningCount,desired:desiredCount,events:events[:3].message}'

TASK=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name odigos-ecs-agent \
  --desired-status RUNNING --query 'taskArns[0]' --output text)

aws ecs execute-command --cluster "$CLUSTER" --task "$TASK" \
  --container odigos-ecs-agent --interactive --command "uname -r"   # want 6.1.x
```

**2. Agents are connected to the proxy**

```bash theme={null}
aws logs tail /odigos/ecs-agent --since 5m
```

Expect a connection to `<PROXY_ENDPOINT>` and **no** `runtime-detector stopped`
errors.

**3. The proxy sees every agent**

The proxy's `/healthz` returns the full membership view: connected agents (id,
node, version, connected-at), tombstones for departed nodes, and the last
coverage result.

```bash theme={null}
PTASK=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name odigos-ecs-proxy \
  --query 'taskArns[0]' --output text)

aws ecs execute-command --cluster "$CLUSTER" --task "$PTASK" \
  --container odigos-ecs-proxy --interactive --command "/bin/bash"
```

The proxy image is slim and ships **no `curl`** — use bash's `/dev/tcp`
inside the session:

```bash theme={null}
exec 3<>/dev/tcp/127.0.0.1/4321
printf 'GET /healthz HTTP/1.0\r\n\r\n' >&3
cat <&3
```

The number of connected agents should match your container-instance count.

**4. Central shows one platform for the cluster**

Open the Central UI: **one** platform named after the ECS cluster (not one
per node), with workloads aggregated across all nodes.

## Caveat: the pinned proxy address

With **Option B**, if the pinned instance is replaced (instance failure, ASG
replacement, manual termination) the proxy's address changes. You must:

1. Re-run [Step 3](#step-3-give-the-proxy-a-stable-address) against a live
   instance to get a new `EC2_ID` and `PROXY_ENDPOINT`.
2. Re-create the proxy service with the new placement constraint.
3. Re-point every agent's `ODIGOS_DEFAULT_CONFIG` at the new endpoint, register a
   new revision, and `update-service --force-new-deployment`.

If that fragility is unacceptable, use **Option A** (internal NLB) — it survives
instance and task replacement at the cost of one extra AWS resource.

## Persist the config journal on EFS

By default the journal is a file on the proxy task's local storage
(`/var/lib/odigos-ecs-proxy/journal.jsonl`). Task replacement loses it. Agents
already persist their own config, so a healthy cluster is unaffected — the
journal only closes the gap for a node that was **absent** when a change was
made. If that gap matters, put the journal on EFS.

`ODIGOS_PROXY_JOURNAL` is a **file path**, not a directory. Mount EFS at a
directory and point the variable at a file inside it. Per-agent replay cursors
are written next to that file (`journal.jsonl.cursors.json`), so the whole
directory must be on the volume.

Add a volume and mount to the [proxy task definition](#step-4-register-the-proxy-task-definition),
and set the environment variable:

```json theme={null}
{
  "volumes": [
    {
      "name": "odigos-proxy-journal",
      "efsVolumeConfiguration": {
        "fileSystemId": "<EFS_FILE_SYSTEM_ID>",
        "transitEncryption": "ENABLED"
      }
    }
  ],
  "containerDefinitions": [
    {
      "mountPoints": [
        {
          "sourceVolume": "odigos-proxy-journal",
          "containerPath": "/mnt/odigos-journal",
          "readOnly": false
        }
      ],
      "environment": [
        { "name": "ODIGOS_PROXY_JOURNAL", "value": "/mnt/odigos-journal/journal.jsonl" }
      ]
    }
  ]
}
```

Merge those fields into `odigos-ecs-proxy.json` — do not replace the rest of
the task definition. Then:

* Create the file system in the same VPC as the proxy, with a mount target in
  each subnet the task can land in.
* Allow **NFS (TCP 2049)** from the proxy's security group to the EFS mount
  targets (Option A: the task's `awsvpc` security group; Option B: the container
  instances' security group).
* Grant the **task role** `elasticfilesystem:ClientMount` and
  `elasticfilesystem:ClientWrite` on that file system. If you enable IAM
  authorization on the volume (`authorizationConfig.iam: ENABLED`), also set
  `elasticfilesystem:ClientRootAccess` or use an access point whose POSIX user
  the proxy can write as.

An existing EFS file system is fine. This is optional: skip it if replaying
missed mutations from Central after a proxy replacement is acceptable.

## Current limits

* **Journal durability** — without EFS, the config journal is lost when the
  proxy task is replaced. See [persist the config journal on EFS](#persist-the-config-journal-on-efs).
* **Per-node workload entries** — workload lists are concatenated across nodes,
  so a service with tasks on several nodes currently appears once per node.
* **Single-node targeting** — profiling jobs and diagnose bundles are served by
  one agent; targeting a specific node is not yet supported.
* **Coverage matching** — instances are correlated to agents by hostname; an
  instance whose private DNS cannot be resolved is conservatively reported as
  uncovered.

## Next steps

> **[Add Sources](/vmagent/setup/configuration/add-sources)**
>
> Choose which ECS tasks to instrument — cluster-wide.

> **[Add Destinations](/vmagent/setup/configuration/add-destinations)**
>
> Send the telemetry to your observability backend.

> **[Configuration Reference](/vmagent/ecs/configuration)**
>
> Agent and proxy environment variables.

> **[Troubleshooting](/vmagent/ecs/troubleshooting)**
>
> Logs, `odictl`, `/healthz`, and common failures.


# Upgrade and Uninstall on ECS
Source: https://docs.odigos.io/vmagent/ecs/maintenance

Roll out a new Odigos ECS Agent version and remove Odigos from an ECS cluster.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

## Upgrade

An upgrade is a new image tag plus a new task definition revision. The agent's
configuration and identity live on the container instance at
`/etc/odigos-ecs-agent`, so they survive the rollout.

**1. Mirror the new image tag**

Pull the new version and push it to your ECR repository, as in
[System Requirements](/vmagent/ecs/requirements#license-and-images). Use the
architecture that matches your container instances.

**2. Register a new task definition revision**

Re-run the `sed` and `register-task-definition` steps from your install guide
with the new `TAG`. This creates a new revision of the same family
(`odigos-ecs-agent`).

```bash theme={null}
grep -c '<' /tmp/agent-td.json   # want 0
aws ecs register-task-definition --cli-input-json file:///tmp/agent-td.json >/dev/null
```

**3. Roll the service**

```bash theme={null}
aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-agent \
  --task-definition odigos-ecs-agent --force-new-deployment
```

ECS replaces the agent task on each instance. Instrumented workloads are not
restarted — instrumentation is direct-attach and the agent re-attaches to the
running tasks when it comes back up.

**4. Upgrade the proxy too (Agent + Proxy mode)**

Same pattern for `odigos-ecs-proxy`:

```bash theme={null}
aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-proxy \
  --task-definition odigos-ecs-proxy --force-new-deployment
```

> [!WARNING]
>
> With a **pinned** proxy (host networking), the service must already have
> `minimumHealthyPercent=0,maximumPercent=100` and
> `--availability-zone-rebalancing DISABLED`, or the rollout gets stuck: ECS
> tries to place the new task before stopping the old one, and both want port
> `4321` on the same instance. If the deployment is already stuck, apply the
> deployment configuration and stop the old task once to unblock it.

Upgrade the proxy and agents to the same version.

> [!NOTE]
>
> Changing an environment variable follows the exact same flow: new revision,
> then `update-service --force-new-deployment`. Remember that values already
> persisted in `config.yaml` on the instance
> [take precedence](/vmagent/ecs/configuration#configuration-precedence-and-persistence)
> over `ODIGOS_DEFAULT_CONFIG`.
>
> To move an existing Agent-only cluster onto the proxy without reinstalling,
> see [Switch to Multi Node](/vmagent/ecs/installation-switch-to-multi-node).

## Uninstall

Removing Odigos does not touch your workloads: instrumentation is attached at
runtime, so stopping the agent stops the telemetry — your tasks keep running
untouched.

**1. Delete the ECS services**

```bash theme={null}
aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-agent --desired-count 0
aws ecs delete-service --cluster "$CLUSTER" --service odigos-ecs-agent --force

# Agent + Proxy mode
aws ecs update-service --cluster "$CLUSTER" --service odigos-ecs-proxy --desired-count 0
aws ecs delete-service --cluster "$CLUSTER" --service odigos-ecs-proxy --force
```

**2. Deregister the task definition revisions**

```bash theme={null}
for family in odigos-ecs-agent odigos-ecs-proxy; do
  for arn in $(aws ecs list-task-definitions --family-prefix "$family" --query 'taskDefinitionArns[]' --output text); do
    aws ecs deregister-task-definition --task-definition "$arn" >/dev/null
  done
done
```

**3. Remove the leftover state on each container instance**

The agent's config directory and the synced instrumentation files are host
directories and outlive the task. On each instance:

```bash theme={null}
sudo rm -rf /etc/odigos-ecs-agent /var/odigos
```

> [!WARNING]
>
> `/etc/odigos-ecs-agent` holds the agent's identity, license, and
> configuration. Removing it makes any future install start fresh — leave it
> in place if you plan to reinstall and want the configuration back.

**4. Clean up the supporting AWS resources (optional)**

```bash theme={null}
aws logs delete-log-group --log-group-name /odigos/ecs-agent
aws logs delete-log-group --log-group-name /odigos/ecs-proxy

aws secretsmanager delete-secret --secret-id odigos-onprem-token --force-delete-without-recovery

aws iam delete-role-policy --role-name odigos-ecs-exec-role --policy-name odigos-extras
aws iam detach-role-policy --role-name odigos-ecs-exec-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name odigos-ecs-exec-role

aws iam delete-role-policy --role-name odigos-ecs-task-role --policy-name ssm-exec
aws iam delete-role-policy --role-name odigos-ecs-task-role --policy-name odigos-coverage 2>/dev/null || true
aws iam delete-role --role-name odigos-ecs-task-role

aws ecr delete-repository --repository-name odigos-ecs-agent --force
aws ecr delete-repository --repository-name odigos-ecs-proxy --force
```

Also delete the internal NLB, listener, and target group if you created them
for the proxy.

> [!NOTE]
>
> Deleting and recreating the license secret gives it a **new** random ARN
> suffix. The install guides' IAM policy uses a `odigos-onprem-token-*`
> wildcard so it keeps working — but any task definition still referencing the
> old ARN fails to start with
> `ResourceNotFoundException ... can't find the specified secret`.

Your ECS cluster and container instances are untouched by all of the above.


# Odigos on Amazon ECS
Source: https://docs.odigos.io/vmagent/ecs/overview

Instrument Amazon ECS tasks with eBPF using the Odigos ECS Agent — the VM Agent packaged as an ECS daemon service. No task restarts, no application changes.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

The **Odigos ECS Agent** is the same agent as the [VM Agent](/vmagent/overview),
packaged as a container image and deployed as an ECS **daemon service** — one
agent task on every EC2 container instance in the cluster.

Each agent discovers the ECS tasks running on its instance through the
instance's Docker daemon, instruments them **in place** with eBPF, and runs the
`odigos-otelcol` collector as a supervised child process to export the
telemetry.

> [!NOTE]
>
> Instrumentation is **direct attach only**: the agent never restarts,
> recreates, or modifies your task definitions. Deploying Odigos does not
> redeploy your services.

## How it works

```mermaid theme={null}
%%{init: {'themeVariables': {'fontSize': '22px'}}}%%
flowchart LR
    Tasks["Your ECS tasks"]
    Agent["odigos-ecs-agent<br/>+ odigos-otelcol"]
    Central["Odigos Central"]

    Agent == "eBPF attach<br/>no restart" ==> Tasks
    Agent -. "direct, or via proxy<br/>see below" .-> Central

    classDef centralStyle fill:#30638E,color:#fff,stroke:#123C60,stroke-width:2px
    classDef agentStyle fill:#C0E0DE,color:#123C60,stroke:#30638E,stroke-width:2px
    classDef taskStyle fill:transparent,color:#475569,stroke:#94a3b8,stroke-width:2px,stroke-dasharray: 4 4
    class Central centralStyle
    class Agent agentStyle
    class Tasks taskStyle
```

One agent per container instance, repeated across every instance in the
cluster. Each agent connects upstream either **directly to Central** (single
node) or **through the proxy** (multi node) — see
[Choose a deployment mode](#choose-a-deployment-mode) below.

* **Discovery** — the agent reads the instance's Docker socket to map processes
  to ECS tasks, services, and task ARNs. Only ECS-managed containers are
  instrumented; host processes and `systemd` services on the container instance
  are ignored (unlike the [VM Agent](/vmagent/setup/installation) and
  [Docker deployment](/vmagent/setup/docker)).
* **Instrumentation** — Java, Go, and C++ attach through their dynamic eBPF
  SDKs. Every other language is covered by
  [OBI](/oss/instrumentations/obi), which also attaches with no restart.
  Restart-based instrumentation (Python/Node.js/.NET environment injection) is
  intentionally not used on ECS, since ECS owns the container lifecycle.
* **Export** — the collector runs inside the agent container with host
  networking, so instrumented tasks reach it on the instance's network
  namespace.
* **Configuration** — the task definition only bootstraps the agent (Central
  or proxy endpoint, license). Sources, destinations, actions, and
  instrumentation rules are managed from Odigos Central or with
  [`odictl`](/vmagent/setup/configuration/odictl), exactly as on any VM Agent.
  On ECS, `odictl` runs inside the agent container over ECS Exec.

## Choose a deployment mode

Both modes run the same agent daemon and instrument workloads identically. The
difference is **how the cluster appears in Odigos Central** and whether you
deploy one extra component.

| | **Agent only** | **Agent + Proxy** |
| - | - | - |
| What runs | the agent daemon service | the agent daemon service **+** one `odigos-ecs-proxy` task per cluster |
| Upstream connections | every agent dials Central directly | only the proxy dials Central; agents dial the proxy |
| How it appears in Central | one platform **per node** | **one** platform, named after the cluster |
| Cluster-wide workload list | no — per node | yes, aggregated across nodes |
| Config replay for new nodes | each node is configured on its own | yes — the proxy journals config changes and replays them to nodes that join later |
| Coverage check (nodes missing an agent) | — | yes, opt-in |
| Extra AWS resources | none | a stable address for the proxy: none if pinned to an instance, or one internal NLB |
| Guide | [Single Node](/vmagent/ecs/installation-agent-only) | [Multi Node](/vmagent/ecs/installation-with-proxy) |

Select your cluster shape for the installation flow:

**Single node (Agent only)**

**One container instance — the agent alone is all you need.**

The agent daemon runs on the instance and connects straight to Odigos
Central, which shows the cluster as a single platform named after it. No
proxy, no load balancer, no extra AWS resources.

**1. Store the license token**

AWS Secrets Manager (or a plain env var for a test cluster).

**2. Create two IAM roles**

Task execution role, plus a task role for `odictl` over ECS Exec.

**3. Register the agent task definition**

Privileged, `pidMode: host`, host networking, with your Central endpoint in `ODIGOS_DEFAULT_CONFIG`.

**4. Create the DAEMON service**

One agent task per container instance, now and for instances that join later.

**5. Verify**

Kernel 5.10+, agent connected, platform visible in Central.

> **[Install: Single Node](/vmagent/ecs/installation-agent-only)**
>
> Follow the full Agent-only flow.

> [!NOTE]
>
> This flow also works unchanged on a **multi-node** cluster — you simply get
> one platform per node instead of one per cluster. Choose it whenever you
> don't need cluster-wide aggregation.

**Multi node (Agent + Proxy)**

**Several container instances that should behave as one cluster.**

The agent daemon still runs on every instance, but the agents dial a single
`odigos-ecs-proxy` task instead of Central. The proxy is the cluster's only
upstream connection, so Central shows **one** platform with aggregated
workloads, cluster-wide configuration, config replay for nodes that join
later, and an optional coverage check.

**1. Store the license token and create the IAM roles**

Same two roles as the single-node flow, plus optional coverage permissions for the proxy.

**2. Give the proxy a stable address**

Pin it to one instance (no extra AWS resources) or front it with an internal NLB (survives instance replacement).

**3. Deploy the proxy first**

One task per cluster, listening on TCP 4321.

**4. Register the agent task definition**

Endpoint set to the proxy, plus `ODIGOS_ECS_BEHIND_PROXY=true`.

**5. Create the DAEMON service**

Agents connect to the already-live proxy on first boot.

**6. Verify**

Agents connected, proxy `/healthz` lists every node, one platform in Central.

> **[Install: Multi Node](/vmagent/ecs/installation-with-proxy)**
>
> Follow the full Agent + Proxy flow.

> [!TIP]
>
> **Not sure? Start single node.** The flow is identical on a 1-node or a 50-node
> cluster, and moving to Agent + Proxy later is just re-pointing the agents'
> endpoint at the proxy and rolling the daemon service — no reinstall. See
> [Switch to Multi Node](/vmagent/ecs/installation-switch-to-multi-node).

## What the proxy adds

`odigos-ecs-proxy` is the ECS counterpart of the Kubernetes `central-proxy`:
one task per cluster that represents the whole cluster to Odigos Central as a
single compute platform.

```mermaid theme={null}
%%{init: {'themeVariables': {'fontSize': '22px'}}}%%
flowchart LR
    agents["odigos-ecs-agent<br/>one per node"]
    proxy["odigos-ecs-proxy<br/>one per cluster"]
    central["Odigos Central"]

    agents == "unchanged<br/>protocol" ==> proxy == "one connection<br/>per cluster" ==> central

    classDef centralStyle fill:#30638E,color:#fff,stroke:#123C60,stroke-width:2px
    classDef proxyStyle fill:#123C60,color:#fff,stroke:#30638E,stroke-width:2px
    classDef agentStyle fill:#C0E0DE,color:#123C60,stroke:#30638E,stroke-width:2px
    class central centralStyle
    class proxy proxyStyle
    class agents agentStyle
```

* **Request routing** — configuration queries go to any one healthy agent (the
  config is identical on all of them); live state such as workload reports fans
  out to every agent and is merged; configuration changes are broadcast to all
  agents.
* **Config journal** — configuration changes are recorded, so a node that joins
  the cluster later (or a task that migrates to a fresh instance) converges to
  the cluster configuration automatically.
* **Membership and coverage** — `GET /healthz` lists the connected agents,
  departed nodes, and — when
  [coverage checking](/vmagent/ecs/configuration#proxy-environment-variables) is
  enabled — ACTIVE container instances that have no agent connected.

The proxy is unprivileged: no host PID, no Docker socket, no host mounts.

## Not supported: Fargate

The ECS Agent requires the **EC2 launch type**. It needs host PID access, the
instance's Docker socket, and the ability to load eBPF programs — none of which
exist on Fargate, where AWS controls the host. There is no Fargate deployment
mode.

## Next steps

> **[System Requirements](/vmagent/ecs/requirements)**
>
> Launch type, kernel and AMI requirements, IAM, and images.

> **[Install: Single Node](/vmagent/ecs/installation-agent-only)**
>
> Agent only — daemon service, direct to Central.

> **[Install: Multi Node](/vmagent/ecs/installation-with-proxy)**
>
> Agent + Proxy — one platform per cluster, with aggregation and config replay.

> **[Configuration Reference](/vmagent/ecs/configuration)**
>
> Every environment variable, platform naming, and where configuration is persisted.


# System Requirements
Source: https://docs.odigos.io/vmagent/ecs/requirements

What your ECS cluster, container instances, IAM roles, and registry need before installing the Odigos ECS Agent.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

Review this page before installing. The launch type and kernel requirements are
hard requirements — the agent cannot work around them.

## Cluster and launch type

* An existing **Amazon ECS cluster** with **EC2 container instances**.
* **EC2 launch type only.** [Fargate is not supported](/vmagent/ecs/overview#not-supported-fargate) —
  the agent needs host PID, the instance's Docker socket, and eBPF.
* Ability to run a **privileged task with `pidMode: host`** and host bind mounts
  on the container instances (see [required permissions](#required-permissions)).

## Required permissions

Each elevated setting in the agent task definition maps to a specific
capability. These are hard requirements.

| Setting | Reason |
| - | - |
| `pidMode: host` | Classifying and attaching to other tasks' processes via `/proc/<pid>`. |
| `privileged: true` | eBPF program loading, perf events, and writing instrumentation files into target containers via `/proc/<pid>/root`. |
| `networkMode: host` | The collector and OpAMP server listen on the instance's network namespace, reachable by instrumented workloads on localhost or the instance IP. |
| `/var/run/docker.sock` (read-only) | Container metadata: mapping PIDs to ECS tasks, service names, task ARNs. |
| `/var/lib/docker` (read-only) | Resolving executable paths through overlay2 for language detection. |
| `/var/odigos` (read-write) | Instrumentation files synced to the instance on startup; the source seeded into workload containers when attaching Java. |
| `/etc/odigos-ecs-agent` (read-write) | Agent configuration and identity, persisted across task restarts and upgrades. |
| `/sys/kernel/debug` | kprobes and tracepoints used by the eBPF instrumentations. |
| `ulimits.memlock: -1` | eBPF maps on kernels that account them against `RLIMIT_MEMLOCK`. |
| `linuxParameters.initProcessEnabled` | Reaps child processes left by attach tooling. |

The **proxy** needs none of these.

## Container instance kernel and AMI

> [!WARNING]
>
> The container instance kernel must be **5.10 or newer with BTF**
> (`/sys/kernel/btf/vmlinux` present). The agent's process detector and eBPF
> probes require BTF/CO-RE and a modern verifier.
>
> **Amazon Linux 2's stock kernel (4.14) does not work.** On it, the agent logs
> `runtime-detector stopped: ... verifier error ... permission denied`, never
> discovers processes, and nothing is instrumented.

* **Recommended:** the **Amazon Linux 2023** ECS-optimized AMI (kernel 6.1).
* **Minimum:** the Amazon Linux 2 **kernel-5.10** ECS-optimized AMI.
* Both `amd64` and `arm64` (Graviton) instances are supported — pull the image
  for the matching architecture.

Run this preflight check on a container instance before deploying:

```bash theme={null}
uname -r                    # want 5.10+ (6.1.x on AL2023) — 4.14.x will fail
ls /sys/kernel/btf/vmlinux  # must exist
```

## Resources per instance

The agent task reserves modest resources on each container instance. The
defaults in the reference task definition are a good starting point:

| Setting | Default | Notes |
| - | - | - |
| `cpu` | `512` | CPU units for the agent and its collector child process. |
| `memoryReservation` | `512` | Soft limit. |
| `memory` | `2048` | Hard limit. Raise it on instances with many high-throughput tasks. |

Make sure your instances have that headroom free, otherwise the daemon task
cannot be placed.

## License and images

* An **Odigos Pro license token** for the VM Agent, delivered to the task as
  `ODIGOS_ONPREM_TOKEN` (recommended: from AWS Secrets Manager).
* The **`odigos-ecs-agent`** container image, and — for
  [Agent + Proxy](/vmagent/ecs/installation-with-proxy) — the
  **`odigos-ecs-proxy`** image. Both are published multi-arch (`amd64`,
  `arm64`).
  \[!NOTE] Obtain the images and registry credentials from your Odigos support team.
* We recommend **mirroring both images into your own Amazon ECR repository** in
  the cluster's account and region. ECS then pulls them with the task execution
  role and needs no external registry credentials, and pulls stay in-region:

  ```bash theme={null}
  export REGION=<region>
  export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
  export REG=${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com
  export TAG=<version>

  aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin "$REG"

  for r in odigos-ecs-agent odigos-ecs-proxy; do
    aws ecr create-repository --repository-name "$r" --region "$REGION" >/dev/null 2>&1 || true
    docker pull --platform linux/amd64 "<ODIGOS_REGISTRY>/${r}:${TAG}"   # or linux/arm64
    docker tag "<ODIGOS_REGISTRY>/${r}:${TAG}" "${REG}/${r}:${TAG}"
    docker push "${REG}/${r}:${TAG}"
  done
  ```

  Pull the architecture that matches your container instances. The proxy image
  is only needed for the Agent + Proxy mode.

## IAM roles

Two roles, both trusting `ecs-tasks.amazonaws.com`:

> **Task execution role — required**
>
> Used by ECS to pull the image, write logs, and read the license secret.
>
> * Managed policy `AmazonECSTaskExecutionRolePolicy`
> * `logs:CreateLogGroup` (the reference task definitions set
>   `awslogs-create-group: "true"`)
> * `secretsmanager:GetSecretValue` on the license secret, when the token comes
>   from Secrets Manager

> **Task role — recommended**
>
> Used by the running task itself.
>
> * `ssmmessages:CreateControlChannel`, `ssmmessages:CreateDataChannel`,
>   `ssmmessages:OpenControlChannel`, `ssmmessages:OpenDataChannel` — required
>   for [ECS Exec](/vmagent/ecs/troubleshooting#inspect-a-running-agent-with-odictl),
>   which is how you run `odictl` against a running agent.
> * **Proxy only, for the optional coverage check:**
>   `ecs:ListContainerInstances`, `ecs:DescribeContainerInstances`,
>   `ec2:DescribeInstances`.

The install guides create both roles with copy-pasteable commands.

## Network

* Outbound connectivity from the container instances to your **Odigos Central**
  endpoint (`host:port`; `:443` implies `wss`). The connection is
  **outbound-only** — a Central instance in another AWS account or a public
  endpoint works with no inbound rules on the cluster.
* **Agent + Proxy mode only:** the instances' security group must allow TCP
  **4321** to the proxy (instance-to-instance when the proxy is pinned to an
  instance, or from the internal NLB).

## Local tooling

For the install and verification steps:

```bash theme={null}
# AWS CLI v2, jq, and the Session Manager plugin (for ECS Exec / odictl)
brew install jq
brew install --cask session-manager-plugin
```

> [!WARNING]
>
> **zsh users:** always brace variables when a `:` follows them — write
> `"${REG}/odigos-ecs-agent:${TAG}"`, not `"$REG/odigos-ecs-agent:$TAG"`, and
> `"arn:aws:iam::${ACCOUNT}:role/..."`, not `"arn:aws:iam::$ACCOUNT:role/..."`.
> Unbraced `$VAR:letter` is parsed by zsh as a history modifier and **silently
> corrupts the value** — producing image names like `odigos-ecs-agentatest` and
> truncated role ARNs.

## Next steps

> **[Install: Agent Only](/vmagent/ecs/installation-agent-only)**
>
> Daemon service, each node connects to Central directly.

> **[Install: Agent + Proxy](/vmagent/ecs/installation-with-proxy)**
>
> One aggregated platform per cluster.


# Export profiles to Pyroscope
Source: https://docs.odigos.io/vmagent/instrumentations/profiling/export-pyroscope

Configure a Pyroscope destination on the VM Agent and view OTLP Profiles in the Pyroscope UI.

The VM Agent exports continuous profiles to external backends using the same **destinations** model as traces.

For profiling, add a destination whose **signals** include `PROFILES`. Any backend which accepts OTLP Profiles is supported.

## Pyroscope destination fields

See the [Pyroscope backend reference](/enterprise/backends/pyroscope) for all fields. Defaults for a local Pyroscope container on the same host:

| YAML / env key | UI label | Example | Notes |
| - | - | - | - |
| `PYROSCOPE_URL` | Pyroscope Endpoint | `localhost:4040` | Distributor `host:port`. Odigos appends `/v1development/profiles` — do **not** include any path suffix. |
| `PYROSCOPE_TLS_ENABLED` | Enable TLS | `false` | Set to `true` when Pyroscope is fronted by TLS. |
| `signals` | Profiles checkbox | `[PROFILES]` | Required — Pyroscope only emits profiles. |

## Add destination with odictl

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Open the destination type list**

Press `d` to focus **Destinations**, then `Enter` (or click **+ Add Destination**) and search for **profiles** to filter the list to backends that support the `PROFILES` signal.

![odictl destination type selector filtered by profiles](https://mintcdn.com/odigos/Z3K5Ztm4CqA3lhsa/images/vmagent/profiling/2-profile-destination.png?fit=max\&auto=format\&n=Z3K5Ztm4CqA3lhsa\&q=85\&s=ba6fca5e5cafa507b29424e237eca499)

Highlight **Pyroscope** and press `Enter`.

**3. Configure the destination**

Fill in **Pyroscope Endpoint** and toggle **Enable TLS** if needed. Leave **Profiles: Yes**.

![odictl Pyroscope destination form with localhost:4040 and TLS disabled](https://mintcdn.com/odigos/Z3K5Ztm4CqA3lhsa/images/vmagent/profiling/3-pyroscope-example.png?fit=max\&auto=format\&n=Z3K5Ztm4CqA3lhsa\&q=85\&s=b387f5bd0ad1803b89ae16893c6f6275)

Press `Ctrl+S` (or focus **Apply** with `Tab` and press `Enter`) to save.

**4. Confirm**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep 'Destination signals configured'
```

Look for a line containing `signals=[PROFILES]` for the new destination.

## Add destination with YAML

Create a file under `/var/lib/odigos-agent/config/destinations.d/`, for example `pyroscope.yaml`:

```yaml theme={null}
- name: pyroscope
  type: pyroscope
  signals:
    - PROFILES
  config:
    PYROSCOPE_URL: "localhost:4040"
    PYROSCOPE_TLS_ENABLED: "false"
```

The destination controller picks the file up automatically.

## Run Pyroscope on the host

Quick local run, useful for testing:

```shell theme={null}
docker run -d --name pyroscope -p 4040:4040 grafana/pyroscope:latest
```

If Pyroscope listens only on localhost, port-forward from your laptop:

```shell theme={null}
ssh -L 4040:localhost:4040 <your-vm>
```

## View profiles

Open Pyroscope at `http://localhost:4040`. Pick profile type **`process_cpu`** and filter by **`service.name`** matching one of your enabled sources.

![Pyroscope UI showing a process\_cpu flame graph for odigos-demo-coupon](https://mintcdn.com/odigos/Z3K5Ztm4CqA3lhsa/images/vmagent/profiling/4-pyroscope-ui.png?fit=max\&auto=format\&n=Z3K5Ztm4CqA3lhsa\&q=85\&s=57c62acae93b5a2b69c6b6dbcae48f55)

## Other OTLP Profiles backends

Any backend that accepts **OTLP Profiles** can be used with destination type `otlp` (gRPC) or `otlphttp` and `signals: [PROFILES]`. Pyroscope is documented here because it is easy to setup self-hosted target for VM deployments, the configuration shape for other backends is identical aside from their own endpoint fields.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Enable continuous profiling
Source: https://docs.odigos.io/vmagent/instrumentations/profiling/getting-started

Enable the VM Agent eBPF profiling pipeline, add sources, and export profiles to Pyroscope.

This guide walks you through enabling the eBPF profiling pipeline on a host, and adding a destination that accepts OTLP profiles.

## Step 1: Enable the profiling pipeline

Turn on the eBPF profiles pipeline on **odigos-otelcol**. You can use `odictl` or edit `/var/lib/odigos-agent/config/config.yaml` directly.

**odictl**

**1. Open the Configuration Editor**

```shell theme={null}
odictl
```

Press c to open the Configuration Editor, then navigate to **Profiling**.

**2. Set profiling.enabled to true**

Set **profiling.enabled** to `true` and press Ctrl+S to save. The collector reloads automatically.

**YAML**

Add to `/var/lib/odigos-agent/config/config.yaml`:

```yaml theme={null}
profiling:
  enabled: true
```

## Step 2: Enable sources for the workloads you want to profile

Profiling only retains samples for **enabled sources**. Open `odictl`, focus the **Sources** pane (press `o`), and instrument the services you want sampled. The screenshot below shows the demo workloads enabled (`Instrumented`) alongside other host processes that stay untouched (`Not Instrumented`).

![Sources panel in odictl showing the demo services instrumented](https://mintcdn.com/odigos/Z3K5Ztm4CqA3lhsa/images/vmagent/profiling/1-ensure-source-enabled.png?fit=max\&auto=format\&n=Z3K5Ztm4CqA3lhsa\&q=85\&s=938295b5d8a6f8fbabda13a8e541747d)

See [Sources](../../setup/configuration/add-sources) for the full walkthrough (TUI and YAML).

> [!TIP]
>
> Toggling a source on or off takes effect within seconds — no vm-agent restart is needed. The collector keeps streaming profiles, but only samples for currently-enabled PIDs are exported.

## Step 3: Add a destination that accepts PROFILES

Open **Destinations → + Add Destination** and search for **profiles**. Any destination whose **Signals** column includes `PROFILES` or they accept OTLP Profiles signal works.

![odictl destination type selector filtered by profiles](https://mintcdn.com/odigos/Z3K5Ztm4CqA3lhsa/images/vmagent/profiling/2-profile-destination.png?fit=max\&auto=format\&n=Z3K5Ztm4CqA3lhsa\&q=85\&s=ba6fca5e5cafa507b29424e237eca499)

As an example for step-by-step Pyroscope configuration, jump to [Export to Pyroscope](./export-pyroscope). For any other OTLP Profiles backend (e.g. `otlp` or `otlphttp`), follow the standard [Add destinations](../../setup/configuration/add-destinations).

## Step 4: View profiles in the destination

Open your destination's UI and you should be able to access the application profiles collected.
eg in Pyroscope, select profile type **`process_cpu`** and filter by **`service.name`** values that match your sources. See [Export to Pyroscope → View profiles](./export-pyroscope#view-profiles) for detailed example.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Continuous profiling overview
Source: https://docs.odigos.io/vmagent/instrumentations/profiling/overview

How continuous CPU profiling works on the Odigos VM Agent: eBPF collection, service attribution, and export to Pyroscope.

## See it in action!

The clip below shows the full path on a host: enabling `profiling.enabled` via `odictl`, adding a Pyroscope destination, and the resulting flame graph in the Pyroscope UI.

![End-to-end VM Agent profiling demo: enable profiling, add destination, view flame graph](https://mintcdn.com/odigos/Z3K5Ztm4CqA3lhsa/images/vmagent/profiling/vm-profiling-demo.gif?s=061fc7e186e7f8caeb7ed8c07a32ead2)

## What is continuous CPU profiling?

CPU profiling samples what your processes are doing on the CPU. Over time, those samples form a **flame graph** that shows which functions and libraries consume the most CPU time.

Unlike traces, which tell you *that* a request was slow, profiles show *where* CPU time was spent. Use profiles after traces (or metrics) point you at a hot service.

## How it works on the VM Agent

1. You enable the profiling pipeline in agent config (`profiling.enabled: true`).
2. **odigos-otelcol** runs the OpenTelemetry [eBPF profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler) receiver on the host.
3. Profiles are exported as using **OTLP Profiles** protocol to a destination which supports OTLP profiles.

## Prerequisites

* Linux host meeting [VM Agent system requirements](../../setup/system-requirements) (eBPF-capable kernel).
* VM Agent installed and running ([Installation](../../setup/installation)).
* At least one [source](../../setup/configuration/add-sources) enabled for the workloads you want to profile.
* A destination that accepts the **PROFILES** signal (see [Export to Pyroscope](./export-pyroscope) for the recommended setup).

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# View profiles in odictl
Source: https://docs.odigos.io/vmagent/instrumentations/profiling/view-in-odictl

Open, navigate, and inspect continuous CPU profiles directly in the odictl terminal interface.

`odictl` includes a built-in profiling view so you can inspect continuous CPU profiles directly in the terminal—without opening an external backend. Use it to find the functions consuming the most CPU in a service, and to drive [profile-driven custom instrumentation](../../setup/configuration/instrumentation-rules/instrument-from-profiles).

## Prerequisites

* VM Agent installed and running ([Installation](../../setup/installation)).
* At least one [source](../../setup/configuration/add-sources) enabled for the workloads you want to profile.
* The profiling pipeline enabled. You can enable it ahead of time (see [Getting started](./getting-started) or the Configuration Editor's **Profiling** section in [odictl](../../setup/configuration/odictl)), or directly from the profiling view as described below.

## Open the profiling view

From the main `odictl` dashboard, press Shift+P to open the profiling view.

If profiling is not yet enabled, `odictl` shows an **Odigos profiling is not enabled** prompt. Press Enter to enable it.

![odictl profiling view prompting to enable profiling](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/profiling/odictl_prompt_enable_profiling.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=93061d652f676cd999c3d30f8ed15d53)

> [!NOTE]
>
> You can also enable profiling without opening this view by setting **profiling.enabled** in the Configuration Editor. See [Getting started](./getting-started) for the full setup.

After profiling is enabled, the agent collects samples before any data appears. This can take up to a minute.

> [!TIP]
>
> The first load may be empty while samples are gathered. Wait a few moments and the services and functions populate automatically—no restart is needed.

![odictl profiling view collecting profile data](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/profiling/odictl_collecting_profile_data.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=74d2b17e18e6d3c0d53c89b4f70ab288)

## Navigating the profiling view

The profiling view has three panes: a **Services** list on the left, a **Filter modules** bar across the top, and a **Functions** list. Use `Tab` to move between them and the arrow keys to move within the focused pane.

| Key | Action |
| - | - |
| `Shift+P` | Open the profiling view from the dashboard |
| `Tab` | Cycle between the Services, Filter modules, and Functions panes |
| `↑` / `↓` | Move between items in the focused pane |
| `Space` | Toggle the highlighted module in the Filter modules pane |
| `a` | Show app-only functions |
| `i` | Instrument the highlighted function (or remove its instrumentation) |
| `Esc` | Return to the main dashboard |

> [!NOTE]
>
> Mouse selection is not supported in the profiling view. Use the arrow keys to select items and `Tab` to switch panes.

### Services pane

The left pane lists the services that have profile data. Use the arrow keys to highlight the service you want to inspect.

### Filter modules

Press `Tab` to focus the **Filter modules** bar at the top. It controls which modules and sample types appear in the Functions list:

* **Module** — selects which modules to show in the profile. This is set based on the language of the selected service (for example, Java). You can also choose `vmlinux` or other libraries based on the runtime/language. Press `Space` to toggle the highlighted module.
* **Type** — defaults to `app`. You can also select `library` or `kernel`, or press `a` to show app-only functions.

![odictl profiling view module and type filters](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/profiling/odictl_module_filters.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=5f451c9c6d04c82ec64a23021e6844cb)

### Functions pane

Press `Tab` again to focus the list of profiled functions for the selected service and filter. Use the arrow keys to highlight a function. From here, press `i` to instrument the highlighted function (the panel then shows **\[i] remove** to undo it)—see [Instrument from Profiles](../../setup/configuration/instrumentation-rules/instrument-from-profiles) for the full workflow.

## Getting Help

If you have any issues, or require our assistance, please open an issue in [GitHub](https://github.com/odigos-io/odigos), or reach out to us in [Odigos Slack](https://join.slack.com/t/odigos/shared_invite/zt-24u91yknm-0fLXu6qnqYfNHS_GSUgTJw)


# Overview
Source: https://docs.odigos.io/vmagent/overview

The Odigos VM Agent instruments Linux processes and systemd services using eBPF. Because instrumentation runs in the kernel, impact on applications is negligible. Traces, logs, and profiles are exported to any of the supported destinations of your choice.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

## Features

* Instrument [systemd](https://systemd.io/) services and Linux processes
* Automatic OpenTelemetry instrumentation via eBPF with no code changes
* Distributed traces with seamless context propagation
* Collect application logs from instrumented sources (process, Docker, systemd) and export them when a destination accepts the `LOGS` signal
* Continuous CPU profiling (eBPF) with export to [Pyroscope](/vmagent/instrumentations/profiling/export-pyroscope) or other OTLP Profiles backends

## Deployment options

The VM Agent can be deployed in three ways. All of them instrument workloads with
the same eBPF agent and are configured the same way (via Odigos Central or
`odictl`):

* **Linux package** — install as an RPM/deb system package managed by `systemd`.
  This is the standard option for a normal Linux host. See [Installation](/vmagent/setup/installation).
* **Docker container** — run the agent as a privileged container. Use this for
  hosts where installing a system package is not an option, such as immutable hosts
  or Compose-managed fleets. See [Docker Deployment](/vmagent/setup/docker).
* **Amazon ECS daemon service** — run the agent as an ECS daemon so every EC2
  container instance instruments the ECS tasks on it, with no task restarts. See
  [Odigos on Amazon ECS](/vmagent/ecs/overview).

## Key concepts

* **Sources**: The systemd services or Linux processes you instrument with the VM Agent.
  See [Add Sources](/vmagent/setup/configuration/add-sources) to get started.
* **Destinations**: The backends that receive your telemetry data.
  See [Add Destinations](/vmagent/setup/configuration/add-destinations) to get started.
* **Actions**: Processors that transform, filter, or enrich telemetry before it is sent to destinations.
  See [Actions overview](/vmagent/setup/configuration/actions/overview) to add and manage actions.
* **Instrumentation rules**: Rules that control how telemetry is recorded from your sources—for example, which attributes, headers, or payloads are collected and how custom instrumentations behave.
  See [Instrumentation rules overview](/vmagent/setup/configuration/instrumentation-rules/overview) to add and manage rules.


# Overview
Source: https://docs.odigos.io/vmagent/setup/configuration/actions/overview

What Odigos VM Agent actions are, their benefits, and the three types: processor addition, span metrics, and URL templatization.

Actions modify OpenTelemetry Signals produced by your sources before it is exported to destinations. They run in the VM Agent, so processing is decoupled from your applications and applies consistently to all signals from the agent.

## Benefits

* **Language and source agnostic** — Actions run in the OpenTelemetry pipeline and can transform, filter, or enrich OpenTelemetry signals regardless of the instrumented process or language.
* **Lower impact on workloads** — Processing happens in the agent instead of inside your services, so CPU and memory stay on the agent.
* **Control cost and volume** — You can reduce or shape the data sent to backends (e.g., sampling, dropping attributes, aggregating spans into metrics) to limit export volume and cost.
* **Reusable building blocks** — The VM Agent supports the OpenTelemetry Collector processors defined in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61); you can combine and configure those processors to match your needs.

## Action types

The VM Agent supports three kinds of actions. Each type has a dedicated page with configuration steps for [odictl](/vmagent/setup/configuration/odictl) and YAML:

### Processor addition

Add an OpenTelemetry Collector processor to transform, filter, or enrich signals (e.g., add or delete attributes, batch, sample, mask PII). Processors defined in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported; for each processor's configuration options, see the [OpenTelemetry Collector Contrib documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor). See [Processor addition](/vmagent/setup/configuration/actions/processor-addition) for configuration steps.

### Span metrics

Convert tracing spans into aggregated metrics (e.g., request counts, latency histograms) to reduce volume and simplify dashboards and alerts. See [Span metrics](/vmagent/setup/configuration/actions/span-metrics) for configuration steps.

### URL templatization

Replace variable parts of URLs with parameter placeholders (e.g., `/users/123` → `/users/{id}`) before recording in signals to control cardinality and group traffic by endpoint pattern. See [URL templatization](/vmagent/setup/configuration/actions/url-templatization) for configuration steps.


# Processor Addition
Source: https://docs.odigos.io/vmagent/setup/configuration/actions/processor-addition

Add OpenTelemetry Collector processors to transform, filter, or enrich OpenTelemetry signals in the VM Agent

[Processor addition](/vmagent/setup/configuration/actions/overview#processor-addition) lets you add an OpenTelemetry Collector processor to transform, filter, or enrich signals (e.g., add or delete attributes, batch, sample, mask PII). Processors defined in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported.

There are two ways to add a processor addition action: using `odictl` or using YAML files.

**odictl**

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Select the actions menu**

Use `Tab` to focus on the Actions pane or press `a`, then press `Enter` or click `+ Add Action` with your mouse.

![Select Actions menu](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=ce5670b32757477053e84051653a4f2c)

**3. Select Processor addition**

Use the arrow keys to move through the list of action types. When **Processor addition** is highlighted, press `Enter`.

![Select Processor addition action](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-type-processor.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=e46e98be90aa2ffc5700b8bbbacd880a)

**4. Configure the processor addition**

1. Press `i` to enter INSERT mode.
2. Set the processor type and options. Only processors listed in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported. Set the processor name (e.g., `attributes`, `batch`) and fill in any processor-specific configuration.
3. When finished, press `Esc`, then type `:wq` to save and exit.

![Configure Processor addition](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-edit-processor.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=24f8bf1bc445dbcde9c46f043bdb3608)

> [!NOTE]
>
> To cancel creating the action, press `Esc` if you are in INSERT mode, then type `:q!` to exit without saving.

**5. Verify the action has been created**

![Verify the action has been created](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-processor-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=487d86bb45b3bfec60a40842b6a37851)

**YAML**

**1. Navigate to the actions configuration folder**

```shell theme={null}
cd /var/lib/odigos-agent/config/actions.d
```

**2. Create an action YAML file**

Create a YAML file for your action using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi processor-addition.yaml
```

**3. Add the processor addition configuration**

Add a processor addition action with the desired processor type and config. Processors listed in the [Odigos collector builder config](https://github.com/odigos-io/odigos/blob/main/collector/builder-config.yaml#L61) are supported. Set `config.type` to one of those processor names (e.g., `attributes`, `batch`), and use `config.processorConfig` for processor-specific options. See the [OpenTelemetry Collector Contrib documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor) for each processor's configuration.

For example:

```yaml theme={null}
name: processor-addition
actionName: processor-addition
notes: ""
signals:
  - TRACES
# pipelines: []  # optional: target specific pipelines (e.g. traces/otlp-datadog)
orderHint: 0
type: processorAddition
config:
  type: ""
  processorName: ""
  processorConfig: {}
```

> [!NOTE]
>
> Replace `config.type` with the processor type (e.g., `attributes`, `batch`), set `config.processorName` if desired, and fill `config.processorConfig` with the processor's configuration.

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Verify the action has been created**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep 'Action created'
```

```
Mar 11 21:59:43 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T21:59:43.580Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/tower/mutations/create_action_handler.go:41 msg="Action created" name=processor-addition
```


# Span Metrics
Source: https://docs.odigos.io/vmagent/setup/configuration/actions/span-metrics

Convert tracing spans into aggregated metrics in the Odigos VM Agent

[Span metrics](/vmagent/setup/configuration/actions/overview#span-metrics) is a processor that turns tracing spans into aggregated metrics (e.g., request counts, latency histograms), reducing export volume and making it easier to build dashboards and alerts.

There are two ways to add a span metrics action: using `odictl` or using YAML files.

**odictl**

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Select the actions menu**

Use `Tab` to focus on the Actions pane or press `a`, then press `Enter` or click `+ Add Action` with your mouse.

![Select Actions menu](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=ce5670b32757477053e84051653a4f2c)

**3. Select Span metrics**

Use the arrow keys to move through the list of action types. When **Span metrics** is highlighted, press `Enter`.

![Select Span metrics action](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-type-span.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=da6d35feb64c8067f23225ce8dc072fa)

**4. Configure span metrics**

1. Press `i` to enter INSERT mode.
2. Adjust options such as `interval`, `metricsExpiration`, and histogram settings as needed.
3. When finished, press `Esc`, then type `:wq` to save and exit.

![Configure Span metrics action](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-edit-span.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=947182fc897b65a65b91e245b29746f0)

> [!NOTE]
>
> To cancel creating the action, press `Esc` if you are in INSERT mode, then type `:q!` to exit without saving.

**5. Verify the action has been created**

![Verify the action has been created](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-span-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=ce840fa545c610400bc960a5db618b83)

**YAML**

**1. Navigate to the actions configuration folder**

```shell theme={null}
cd /var/lib/odigos-agent/config/actions.d
```

**2. Create an action YAML file**

Create a YAML file for your action using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi span-metrics.yaml
```

**3. Add the span metrics configuration**

Add a span metrics action. You can adjust `interval`, `metricsExpiration`, `additionalDimensions`, and other resource attributes to match your needs.

For example:

```yaml theme={null}
name: span-metrics
actionName: span-metrics
signals:
  - TRACES
  - METRICS
# pipelines: []  # optional: target specific pipelines
type: spanMetrics
config:
  disabled: false
  interval: "60s"
  metricsExpiration: "5m"
  additionalDimensions: []
  histogramDisabled: false
  explicitHistogramBuckets: []
  resourceMetricsKeyAttributes: []
  includedProcessInDimensions: false
  excludedResourceAttributes: []
```

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Verify the action has been created**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep 'Action created'
```

```
Mar 11 21:59:43 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T21:59:43.580Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/tower/mutations/create_action_handler.go:41 msg="Action created" name=span-metrics
```


# URL Templatization
Source: https://docs.odigos.io/vmagent/setup/configuration/actions/url-templatization

Replace dynamic URL parts with parameter placeholders in VM Agent OpenTelemetry signals

[URL templatization](/vmagent/setup/configuration/actions/overview#url-templatization) replaces variable parts of URLs with parameter placeholders before the URL is recorded in OpenTelemetry signals (e.g., `/users/123` → `/users/{id}`), keeping cardinality under control and making it easier to group traffic by endpoint pattern.

There are two ways to add a URL templatization action: using `odictl` or using YAML files.

**odictl**

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Select the actions menu**

Use `Tab` to focus on the Actions pane or press `a`, then press `Enter` or click `+ Add Action` with your mouse.

![Select Actions menu](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=ce5670b32757477053e84051653a4f2c)

**3. Select URL Templatization**

Use the arrow keys to move through the list of action types. When **URL templatization** is highlighted, press `Enter`.

![Select URL Templatization action](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-type-url.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=37dc41d760d3989a8f2c1f44972a4e46)

**4. Configure URL templatization**

1. Press `i` to enter INSERT mode.
2. Add one or more templatization rules with a `template` (e.g., `/api/v1/resource/{id}`) and optional `examples`.
3. When finished, press `Esc`, then type `:wq` to save and exit.

![Configure URL templatization](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-edit-url.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=b3be0309fc69b49f9459f584906809d5)

> [!NOTE]
>
> To cancel creating the action, press `Esc` if you are in INSERT mode, then type `:q!` to exit without saving.

**5. Verify the action has been created**

![Verify the action has been created](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/actions/odictl-action-url-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=36a6abca986d88f1374fb7c014927d8b)

**YAML**

**1. Navigate to the actions configuration folder**

```shell theme={null}
cd /var/lib/odigos-agent/config/actions.d
```

**2. Create an action YAML file**

Create a YAML file for your action using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi url-templatization.yaml
```

**3. Add the URL templatization configuration**

Add a URL templatization action with one or more rule groups. Each rule has a `template` (e.g., `/api/v1/resource/{id}`), optional `examples` of matching URLs, and optional `notes`.

For example:

```yaml theme={null}
name: url-templatization
actionName: url-templatization
signals:
  - TRACES
# pipelines: []  # optional: target specific pipelines (e.g. traces/otlp-datadog)
type: urlTemplatization
config:
  templatizationRulesGroups:
    - notes: ""
      templatizationRules:
        - template: "/api/v1/resource/{id}"
          examples:
            - "/api/v1/resource/123"
          notes: ""
```

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Verify the action has been created**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep 'Action created'
```

```
Mar 11 21:59:43 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T21:59:43.580Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/tower/mutations/create_action_handler.go:41 msg="Action created" name=url-templatization
```


# Destinations
Source: https://docs.odigos.io/vmagent/setup/configuration/add-destinations

Add destinations to the Odigos VM Agent so telemetry can be sent to your backends.

Enable the signals you want to export on each destination (`TRACES`, `LOGS`, `PROFILES`). The agent turns on collection for those signals automatically—for example, configuring `LOGS` starts [log collection](/vmagent/setup/configuration/overview#log-collection) for instrumented sources.

There are two ways to add [destinations](/vmagent/overview#key-concepts) to the Odigos VM Agent: use `odictl` or use YAML files.

**odictl**

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Select the destination menu**

Use `Tab` to focus on the Destinations pane or press `d`, then press `Enter` or click `+ Add Destination` with your mouse.

![Select Destination section of the odictl interface](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/destination/odictl-destination-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=7508d738a9ce547a6b723fa4119413bb)

**3. Select a destination**

Type the name of the destination in the search bar at the top, and/or press `Tab` and scroll through the list
of destinations. Once your destination is highlighted, press `Enter`.

![Select Destination](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/destination/odictl-destination-type.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=b13a9cab15b5fd33aae5370954ee9840)

**4. Configure destination**

Set each field according to your destination type. See [Destinations](/enterprise/backends-overview)
for field details and requirements.

![Configure Destination](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/destination/odictl-destination-configure.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=2c9f89ba774cd30eccb91f494ca9d672)

Fill in the required fields, then press `Ctrl+S` to save your destination.

**5. Verify the destination**

Your destination appears in the `Destinations` section in `odictl`.

![Verify Destination](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/destination/odictl-destination-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=f55158680a809a60e8dbad53554cf783)

**YAML**

**1. Navigate to the /var/lib/odigos-agent/config/destinations.d folder**

```shell theme={null}
cd /var/lib/odigos-agent/config/destinations.d
```

**2. Create destination YAML file**

Create a YAML file for your destination's configuration using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi example.yaml
```

**3. Add the destination configuration**

Set each field according to your destination type. See [Destinations](/enterprise/backends-overview) for field details and requirements.

For example:

```yaml theme={null}
- name: otlphttp
  type: otlphttp
  config:
    OTLP_HTTP_COMPRESSION: none
    OTLP_HTTP_ENDPOINT: https://otlp-example.com
    OTLP_HTTP_INSECURE_SKIP_VERIFY: "false"
    OTLP_HTTP_OAUTH2_ENABLED: "false"
    OTLP_HTTP_TLS_ENABLED: "false"
  signals:
    - TRACES
    - LOGS
```

> [!NOTE]
>
> The above is an example of an [OTLP http](/enterprise/backends-overview) destination.

For continuous CPU profiling, add a [Pyroscope](/enterprise/backends/pyroscope) destination with `signals: [PROFILES]`. See [Export to Pyroscope](/vmagent/instrumentations/profiling/export-pyroscope).

> [!NOTE]
>
> You can define multiple destinations in a single YAML file, and you can organize destinations across multiple YAML files.

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Verify the destination has been added successfully**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep 'Destination signals configured'
```

```
Mar 11 20:50:18 ip-10-0-1-51 odigos-vmagent[611]: time=2026-03-11T20:50:18.401Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/destination/controller.go:177 msg="Destination signals configured" destination=otlphttp signals=[TRACES LOGS]
```


# Sources
Source: https://docs.odigos.io/vmagent/setup/configuration/add-sources

Add or instrument sources in the Odigos VM Agent so the agent can collect their telemetry.

There are two ways to add [sources](/vmagent/overview#key-concepts) to the Odigos VM Agent: use `odictl` or use YAML files.

> [!TIP]
>
> It is recommended to [add a destination](/vmagent/setup/configuration/add-destinations) before adding a source.

> [!WARNING]
>
> **Instrumenting a source restarts its underlying service or process.** Regardless of whether you use `odictl` or YAML, the VM Agent restarts each service, process, or container when it applies instrumentation, so expect a brief interruption.
>
> **The only exception is Go and Java processes running directly on the host (not in a container).** These are instrumented in place and are **not** restarted.

> [!NOTE]
>
> A **non-root** agent requires additional permissions to restart services, containers, and processes. See [Additional permissions](/vmagent/setup/permissions#additional-permissions).

**odictl**

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Open the Add Source menu**

Use `Tab` to focus on the Sources pane or press `o`, then press `Enter` or click `+ Add Source` with your mouse.

![Select Sources pane of the odictl interface](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/source/odictl-sources-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=e19f987c17fc8381f643c736dfc37db5)

Once the source list opens, choose how you want to instrument. Select a **single source** to review and adjust its
properties before instrumenting, or select **multiple sources** to instrument them all at once with default settings.

**Single source**

**1. Search for and select a source**

Click in the search bar at the top and type the name of the Linux process, and/or press `Tab` and scroll
through the list of sources. Once your source is highlighted, press `Enter`.

![Select Source](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/source/odictl-sources-process.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=29a79f41e7a291cbf352f704fbc17c31)

**2. Review properties and instrument**

Review and adjust the properties if needed:

**Properties**

**Name** `string`

The identity the VM Agent uses to match this source to the discovered process, service, or container. It is auto-populated at discovery time and is required — it also drives source lookup and log collection, so avoid changing it (contact support if you must).

**Service Name** `string`

The OpenTelemetry `service.name` reported for this source's telemetry — the service that its spans, metrics, and logs are attributed to in your backend (for example, the service node in a service map). It is auto-populated from `Name` at discovery time, but you can edit it to report telemetry under a different service name. If left empty, it defaults to `Name`. This does not change process matching or the individual span (operation) names.

**Kind** `string`

The process type: either `systemd` for services managed by systemd, `process` for standalone Linux processes, or `docker` for docker containers.
This field is auto-populated at discovery time — contact support before changing it.

**Enabled** `boolean`

Controls whether the VM Agent instruments this source. Check to enable instrumentation, or uncheck to pause it without removing the source.

**Runtime Language** `string`

The programming language of the process as detected by the VM Agent. This field is auto-populated at discovery time — contact support before changing it.

**Runtime Version** `string`

The version of the runtime language as detected by the VM Agent. This field is auto-populated at discovery time — contact support before changing it.

![Instrument Source](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/source/odictl-sources-properties.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=3ad60cb3c57517578057bc9d07fa1c28)

Press `Ctrl+S` to save and instrument your source.

**3. Verify instrumentation**

Instrumentation can take a few moments to complete. Once it is applied, the source appears in the `Sources` list with a green `Health` status.

![Verify Instrumentation](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/source/odictl-sources-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=2b7c3d519b33cb189b6b77d8557b7cb8)

**Multiple sources (defaults)**

**1. Select multiple sources**

Move through the list with the arrow keys or `Tab`, and press `Space` to select each process, service, or
container you want to instrument. Repeat until all the sources you want are selected.

![Select Multiple Sources](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/source/odictl-sources-multi-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=5a4a6614358310a7ffee91a3d6a2bbb2)

**2. Instrument all selected sources**

Press `Enter` to instrument every selected source at once. This flow does not show the properties dialog —
each source is instrumented with its default settings, which is what most deployments want.

**3. Verify instrumentation**

Instrumentation can take a few moments to complete. Once it is applied, the sources appear in the `Sources` list with a green `Health` status.

![Verify Instrumentation](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/source/odictl-sources-multi-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=dc899fd32f0cd15d5b1cca281640acc3)

**YAML**

**1. Navigate to the /var/lib/odigos-agent/config/sources.d folder**

```shell theme={null}
cd /var/lib/odigos-agent/config/sources.d
```

**2. Create source YAML file**

Create a YAML file for your source's configuration using the editor of your choice. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi example.yaml
```

**3. Add the source configuration**

Define each service or process you want to instrument using the properties below.

**Properties**

**name** `string`

The identity the VM Agent uses to match this source to the discovered process, service, or container. Required, and also used for source lookup and log collection, so it should match the detected workload.

**enabled** `boolean`

Controls whether the VM Agent instruments this source. Set to `true` to enable instrumentation, or to `false` to pause it without removing the source.

**serviceName** `string`

Optional. The OpenTelemetry `service.name` reported for this source's telemetry — the service that its spans, metrics, and logs are attributed to in your backend. If omitted, it defaults to `name`. This does not change process matching or the individual span (operation) names.

**kind** `string`

The process type: either `systemd` for services managed by systemd, `process` for standalone Linux processes, or `docker` for Docker containers.

**config** `object | null`

Optional runtime configuration (e.g., language and version). Auto-populated at discovery time — contact support before changing this field.

For example:

```yaml theme={null}
# serviceName omitted: telemetry is reported under service.name = odigos-demo-frontend (the name)
- name: odigos-demo-frontend
  enabled: true
  kind: systemd
  config: null
# serviceName set: telemetry is reported under service.name = membership-service
- name: odigos-demo-membership
  enabled: true
  serviceName: membership-service
  kind: systemd
  config: null
```

> [!NOTE]
>
> You can define multiple sources in a single YAML file, and you can organize sources across multiple YAML files.

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Verify the source has been instrumented**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep 'Successfully attached eBPF probes to process'
```

```
Mar 10 20:14:35 ip-10-0-1-51 odigos-vmagent[4203]: time=2026-03-10T20:14:35.846Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/instrumentor/ebpf_manager.go:179 msg="Successfully attached eBPF probes to process" pid=7251 service=odigos-demo-frontend
Mar 10 20:21:42 ip-10-0-1-51 odigos-vmagent[4203]: time=2026-03-10T20:21:42.098Z level=INFO source=/go/src/github.com/keyval/odigos-vmagent/pkg/components/controller/instrumentor/ebpf_manager.go:179 msg="Successfully attached eBPF probes to process" pid=8308 service=odigos-demo-membership
```


# Custom Instrumentation
Source: https://docs.odigos.io/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation

Define custom eBPF-based instrumentations for arbitrary functions in your Java, Go, or C++ application or its dependencies using the VM Agent.

[Custom instrumentation](/vmagent/setup/configuration/instrumentation-rules/overview#custom-instrumentation) lets you define custom eBPF-based instrumentations for specific functions or methods in your application or its dependencies. Unlike automatic instrumentation, which covers common frameworks and libraries, custom instrumentation targets the exact code you choose—for example, a business-critical function or an internal helper that you want to see as a span in your traces. The VM Agent supports **Java**, **Go**, and **C++**. For Go, you can instrument either a **function** (by package and function name) or a **receiver method** (by package, receiver type, and method name); you must use one form or the other, not both. For C++, you target a function by its **signature**.

There are two ways to add a custom instrumentation rule: using `odictl` or using YAML files.

> [!TIP]
>
> Prefer to find hot functions visually instead of typing class and method names? Use the odictl profiling view to instrument a function straight from a CPU profile. See [Instrument from Profiles](/vmagent/setup/configuration/instrumentation-rules/instrument-from-profiles).

**odictl**

**1. Launch odictl**

```shell theme={null}
odictl
```

**2. Open the Instrumentation Rules pane**

Use `Tab` to cycle to the **Instrumentation Rules** pane, or click it with your mouse. Press `Enter` or click **+ Add Rule** to add a new rule.

![Open the Instrumentation Rules pane](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/rules/odictl-rule-select.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=5ab5e1ab4b0696a79e77693efe61986f)

**3. Choose Custom instrumentation**

Select **Custom instrumentation** as the rule type and press `Enter`.

![Choose Custom instrumentation](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/rules/odictl-rule-custominstrumentation.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=5bae7fbc46dae6823177d3ba1a3058e1)

**4. Configure the rule**

1. Press `i` to enter INSERT mode.
2. For **Java**: set `className` (fully qualified class name) and `methodName` (method name).
3. For **Go function**: set `packageName` and `functionName`. Do not set `receiverName` or `receiverMethodName`.
4. For **Go receiver method**: set `packageName`, `receiverName`, and `receiverMethodName`. Do not set `functionName`.
5. For **C++**: set `signature` (the function signature, for example `std::vector::push_back` or `SSL_write`).
6. When finished, press `Esc`, then type `:wq` to save and exit.

**Examples**

> **Java**
>
> Instrument a method by fully qualified class and method name:
>
> ```yaml theme={null}
> name: my-custom-instrumentation
> type: CustomInstrumentation
> config:
>   java:
>     - className: "com.example.MyClass"
>       methodName: "myMethod"
> ```

> **Go — function**
>
> Instrument a function by package and function name:
>
> ```yaml theme={null}
> name: my-custom-instrumentation
> type: CustomInstrumentation
> config:
>   golang:
>     - packageName: "github.com/example/ecommerce"
>       functionName: "ProcessOrder"
> ```

> **Go — receiver method**
>
> Instrument a method by package, receiver type, and method name:
>
> ```yaml theme={null}
> name: my-custom-instrumentation
> type: CustomInstrumentation
> config:
>   golang:
>     - packageName: "github.com/example/ecommerce"
>       receiverName: "OrderProcessor"
>       receiverMethodName: "ProcessOrder"
> ```

> **C++**
>
> Instrument a function by its signature:
>
> ```yaml theme={null}
> name: my-custom-instrumentation
> type: CustomInstrumentation
> config:
>   cpp:
>     - signature: "ecommerce::OrderProcessor::processOrder"
> ```

> [!TIP]
>
> The `java:`, `golang:`, and `cpp:` keys accept a list; add multiple entries to instrument several methods or functions in one rule. Use only one language per rule (do not combine `java:`, `golang:`, and `cpp:`). For Go, use either functions or receiver methods in a single rule, not both.

> [!NOTE]
>
> To cancel, press `Esc` if in INSERT mode, then type `:q!` to exit without saving.

**5. Verify the rule**

The rule appears in the **Rules** list.

![Rules list with the new rule](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/rules/odictl-rule-custominstrumentation-end.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=374fc76d1b8b3216c3db9db643e8b71c)

**YAML**

**1. Navigate to the rules configuration folder**

```shell theme={null}
cd /var/lib/odigos-agent/config/rules.d
```

**2. Create a rule YAML file**

Create a YAML file for your custom instrumentation rule. The example below uses [vi](https://en.wikipedia.org/wiki/Vi).

```shell theme={null}
sudo vi custom-instrumentation.yaml
```

**3. Add the custom instrumentation configuration**

Add a rule with `CustomInstrumentation` and one of `java`, `golang`, or `cpp`. For Go, use either a **function** (package + functionName) or a **receiver method** (package + receiverName + receiverMethodName), not both.

> **Java — class and method**
>
> ```yaml theme={null}
> name: custom-instrumentation
> type: CustomInstrumentation
> config:
>   java:
>     - className: "com.example.ecommerce.OrderService"
>       methodName: "processOrder"
> ```

> **Go — package and function**
>
> ```yaml theme={null}
> name: custom-instrumentation
> type: CustomInstrumentation
> config:
>   golang:
>     - packageName: "github.com/example/ecommerce"
>       functionName: "ProcessOrder"
> ```

> **Go — receiver method**
>
> ```yaml theme={null}
> name: custom-instrumentation
> type: CustomInstrumentation
> config:
>   golang:
>     - packageName: "github.com/example/ecommerce"
>       receiverName: "OrderProcessor"
>       receiverMethodName: "ProcessOrder"
> ```

> **C++ — signature**
>
> ```yaml theme={null}
> name: custom-instrumentation
> type: CustomInstrumentation
> config:
>   cpp:
>     - signature: "ecommerce::OrderProcessor::processOrder"
> ```

**4. Save the file**

```shell theme={null}
:wq!
```

**5. Verify the rule has been loaded**

```shell theme={null}
sudo journalctl -u odigos-vmagent | grep -i rules
```

```
Mar 16 21:49:38 ip-10-0-1-51 odigos-vmagent[36872]: {"level":"info","ts":"2026-03-16T21:49:38.580Z","caller":"logger/logger.go:177","msg":"Publishing rules snapshot","component":"vm-agent","controller":"rules","files":4,"total_rules":4}
Mar 16 21:49:38 ip-10-0-1-51 odigos-vmagent[36872]: {"level":"info","ts":"2026-03-16T21:49:38.580Z","caller":"logger/logger.go:177","msg":"Received RulesChange update","component":"vm-agent","controller":"coordinator","rules":4,"has_rules":true}
```


# Instrument from Profiles
Source: https://docs.odigos.io/vmagent/setup/configuration/instrumentation-rules/instrument-from-profiles

Create custom instrumentation rules from a function in the odictl continuous profiling view.

The odictl [profiling view](/vmagent/instrumentations/profiling/view-in-odictl) lets you turn a profiled function into a [custom instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation) rule with a single keystroke. Instead of typing a class, package, or method name by hand, you highlight a hot function in the profile and press `i`—`odictl` generates the matching `CustomInstrumentation` rule for you.

This UI-driven flow supports **Java**, **Go**, and **C++**.

> [!NOTE]
>
> This page covers the profile-driven, UI-only workflow. To define a rule manually (entering class/method or package/function yourself) or with YAML files, see [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation).

## Before you start

* Profiling must be enabled and collecting data. See [View profiles in odictl](/vmagent/instrumentations/profiling/view-in-odictl).
* A [source](/vmagent/setup/configuration/add-sources) must be enabled for the service you want to instrument.

## Create a rule from a profile

**1. Open the profiling view and select a function**

From the main `odictl` dashboard, press Shift+P to open the [profiling view](/vmagent/instrumentations/profiling/view-in-odictl). Use `Tab` to move between the three panes—**Services**, **Filter modules**, and **Functions**—and the arrow keys to move within the focused pane.

1. In the **Services** pane, select a service. Its runtime (Java, Go, or C++) determines the modules shown.
2. In the **Filter modules** pane, adjust the module or type filter if needed.
3. In the **Functions** pane, highlight the function you want to instrument.

In this example, the `odigos-demo-frontend` service is selected and the highlighted function is the Java method `sun.net.www.protocol.http.HttpURLConnection.getHeaderField`.

![odictl profiling view with the getHeaderField function selected, before instrumenting](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/profiling/odictl_profiling_before_instrumenting.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=03fe5870ff6b2870a49db3d17d1aa428)

**2. Instrument the function**

Press i. The bottom-right panel updates to show the function is **Instrumented**. In this example, `getHeaderField` is now instrumented under the `odigos-profiling-odigos-demo-frontend` rule.

![odictl profiling view showing the getHeaderField function marked as Instrumented](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/profiling/odictl_profiling_after_instrumenting.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=85e778682c2320318371ea614a829cb1)

**3. Return to the dashboard**

Press Esc to return to the main dashboard. A new rule appears in the **Rules** section, named `odigos-profiling-<service-name>` for the service you selected—`odigos-profiling-odigos-demo-frontend` in this example.

**4. Review the generated rule**

Highlight the rule to view its configuration. The generated config matches the runtime of the profiled function:

* **Java** — the fully qualified class name and method name.
* **Go** — the package and function name, or the package, receiver type, and method name.
* **C++** — the function signature.

For the underlying field reference, see [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation).

![odictl Rules section showing the generated profiling rule and its config](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/profiling/odictl_profiling_instrumentation_rule.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=924c2839e64c6b1bc5182b9b5514656c)

> [!NOTE]
>
> The rule name and config are generated automatically from the profile—you don't enter the class, package, or method by hand. To author a rule manually instead, see [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation).

**5. Verify the new span**

Once the rule is applied, new traces include a span for the instrumented function. Open your trace backend and search for the function name (for example, `getHeaderField`) to confirm the span is being recorded.

## Remove a profile-driven instrumentation

To stop instrumenting a function, reopen the profiling view (`Shift+P`), highlight the same function, and press i again. The panel shows **\[i] remove** for an already-instrumented function.

## Related

* [View profiles in odictl](/vmagent/instrumentations/profiling/view-in-odictl) — open and navigate the profiling view.
* [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation) — define rules manually with `odictl` or YAML.
* [Instrumentation rules overview](/vmagent/setup/configuration/instrumentation-rules/overview) — all rule types and their benefits.


# Overview
Source: https://docs.odigos.io/vmagent/setup/configuration/instrumentation-rules/overview

What Odigos VM Agent instrumentation rules are, their benefits, and rule types: code attributes, headers collection, payload collection, and custom instrumentation.

Instrumentation rules control how telemetry is recorded from your [sources](/vmagent/overview#key-concepts). You can apply rules to specific sources and instrumentation libraries to control which attributes, headers, and payloads are collected and how custom instrumentations behave.

## Benefits

* **Control what is recorded** — Decide which code attributes, HTTP headers, and payloads are added to spans so you capture the right context without excess volume.
* **Enrich without code changes** — Get better context without modifying or redeploying your application.
* **Trace back to code** — Code attributes link spans to the code that produced them, so you can debug and analyze with confidence.
* **Custom instrumentation** — Define custom eBPF-based instrumentations for arbitrary functions in your application or its dependencies.

## Rule types

You can manage instrumentation rules via [odictl](/vmagent/setup/configuration/odictl) or YAML.

### Custom instrumentation

Define custom eBPF-based instrumentations for arbitrary functions in your application or its dependencies. Use custom instrumentation with Java, Go, and C++ applications to instrument application-specific code. See [Custom Instrumentation](/vmagent/setup/configuration/instrumentation-rules/custom-instrumentation) for configuration, or [Instrument from Profiles](/vmagent/setup/configuration/instrumentation-rules/instrument-from-profiles) to create a rule directly from a function in the odictl profiling view.

### Code attributes

Add [semantic code attributes](https://opentelemetry.io/docs/specs/semconv/attributes-registry/code/) to spans so you can trace an operation back to the code that generated it.

> [!NOTE]
>
> **Planned for a future release.**

### Headers collection

Add span attributes by collecting HTTP header data from requests and responses.

> [!NOTE]
>
> **Planned for a future release.**

### Payload collection

Add span attributes by collecting payload data from HTTP requests and responses, database queries, and messaging systems.

> [!NOTE]
>
> **Planned for a future release.**


# odictl
Source: https://docs.odigos.io/vmagent/setup/configuration/odictl

Launching and navigating the terminal user interface (TUI); `odictl`

## Introduction

The Odigos VM Agent is installed with a terminal user interface (TUI) called `odictl`. `odictl` is one method you can use
to instrument processes and services, set up destinations, and add actions to Odigos.

> [!NOTE]
>
> Want centralized visibility and management? See [Connecting VM Agent](/central/adding-connections/vmagent) to connect this host to Odigos Central.

To launch `odictl`, run the following command from the command-line:

```shell theme={null}
odictl
```

You will then see a screen similar to the one below:

![Initial odictl TUI screen](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/odictl/odictl-init-screen.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=b2e0b5ff8583f324884235a027852f7f)

## Install a license

Apply the Odigos Pro token the support team issued. `odictl` validates the JWT before writing `/var/lib/odigos-agent/config/license.jwt`, then you restart the agent:

***Alternative***:

Create a file called `/var/lib/odigos-agent/config/license.jwt` with the JWT token and restart the agent.

```shell theme={null}
sudo odictl -odigos-license '<JWT>'
sudo systemctl restart odigos-vmagent
```

See [Installation](/vmagent/setup/installation#apply-the-license).

## Navigation

The odictl interface has five main sections: Services, Logs,
Sources, Destinations, and Actions.
Use the keyboard to move between them; if you are connected through an `ssh` session from a terminal that supports
mouse input, you can navigate the interface with your mouse.

## Navigating the Interface

> **Keyboard Navigation**
>
> The following describes how to navigate using the keyboard:
>
> | Key | Action |
> | - | - |
> | `Tab` / `Shift+Tab` | Cycle panes |
> | `↑` / `↓` | Move between items in the current section |
> | `Enter` | Select the highlighted item |
> | `q` / `Esc` | Quit |
> | `s` | Focus Services |
> | `o` | Focus Sources |
> | `d` | Focus Destinations |
> | `a` | Focus Actions |
> | `c` | Open Configuration Editor |
> | `r` | Reload status + config |
> | `e` | Edit selected |
> | `Del` / `Backspace` | Delete selected |
> | `l` | Show service logs |
> | `y` | Show YAML config |
> | `?` / `h` | Display full keymap |
>
> > [!TIP]
> >
> > The currently selected item has a secondary border around it. Press `Enter` to select it.

> **Mouse Navigation**
>
> #### Requirements
>
> Mouse support requires:
>
> * A terminal emulator that supports mouse input
>
> * An SSH session that forwards mouse events
>
> > [!TIP]
> >
> > Most modern terminal emulators support mouse interaction automatically.
>
> To navigate:
>
> * Click a **section** to switch sections
> * Click an **item** to select it

## Services Panel

The Services section shows the status of the two services that run the Odigos VM Agent:
odigos-vmagent and odigos-otelcol. When you highlight a service,
the log window shows that service's logs.

![Services section of the odictl interface](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/odictl/odictl-services-screen.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=2416bcbf09bf823781ce4f63be4d8ab3)

The following animation shows switching between viewing logs for odigos-vmagent
and odigos-otelcol in the Logs section.

<video />

## Logs Panel

The Logs section shows a live stream of logs for the odigos-vmagent and
odigos-otelcol Odigos services. When the section is focused, use the ↑ / ↓ keys
or the mouse wheel to scroll through the logs.

## Sources Panel

Sources is where you define the systemd services or Linux processes on the host system that you want to instrument with
the Odigos VM Agent. See [Add Sources](/vmagent/setup/configuration/add-sources) to learn how.

![Sources screen](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/odictl/odictl-sources-screen.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=32a119d3940125f8ae5563b62f374b41)

## Destinations Panel

[Destinations](/enterprise/backends-overview) are backends that accept data from the Odigos VM Agent.
Odigos lets you [add as many destinations](/vmagent/setup/configuration/add-destinations) as you like, enabling you to
execute different use cases such as:

* Using different backends for different types of OpenTelemetry signals
* Migrating from one backend to another
* Testing different backends for different OpenTelemetry use cases

![Destinations screen](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/odictl/odictl-destinations-screen.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=77a339076c09c8e67b39a7a938f7453c)

## Actions Panel

Actions let you modify OpenTelemetry data from Odigos Sources before it is exported to your destinations. Odigos
implements actions using OpenTelemetry Collector processors—components that transform, filter, or enrich telemetry
before it is sent to your destinations.

See [Actions overview](/vmagent/setup/configuration/actions/overview) to add or edit actions. For action types and processor options, see
[Odigos Actions](/enterprise/pipeline/actions/introduction).

![Actions screen](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/odictl/odictl-actions-screen.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=5420468f92f2a09632ba5a634309cbaa)

## Configuration Editor

The Configuration Editor lets you view and modify the VM Agent's configuration directly from the TUI
without manually editing YAML files.

### Opening the Configuration Editor

Press c from any panel to open the Configuration Editor. The editor replaces
the main dashboard with a three-column layout:

| Column | Purpose |
| - | - |
| **Section** | Top-level configuration categories such as Logging, Network, Central, Process controller, Data collection, Instrumentation controller, and Profiling. |
| **Subsection** | Groups within a section. For example, Instrumentation controller contains subsections for Java instrumentation and its top-level fields. Sections with a single group show the section name. |
| **Values** | The editable fields for the selected subsection. Each field shows its label, an optional description, and an input widget (text input, dropdown, or multi-line editor). |

### Navigating the Editor

| Key | Action |
| - | - |
| `Tab` / `Shift+Tab` | Move focus between the three columns |
| `↑` / `↓` | Move between items in the focused column |
| `Ctrl+←` | Jump back from the Values column to Subsection |
| `Enter` | Open a dropdown or confirm a selection |
| `Ctrl+S` | Save all changes to disk and reload the agent configuration |
| `Esc` | Discard changes and return to the main dashboard |

Mouse navigation is also supported: click a section, subsection, or field to focus it, and use the
scroll wheel to browse long lists of fields.

### Editing Fields

* **Text fields**: Type directly into the input to change the value.
* **Boolean / enum fields**: A dropdown appears with the available options. Use `↑` / `↓` to pick a
  value, then press `Enter` to confirm.
* **Read-only fields**: Displayed with a *(read-only)* label and cannot be modified.

### Saving Changes

Press Ctrl+S to validate and save. If validation fails (for example, a
value is out of range), an error message appears at the top of the Values column. Fix the indicated
field and save again. Saving writes the updated configuration to the YAML file on disk and triggers
an automatic reload of the affected agent components.

### Configuration Sections

> **Logging**
>
> Controls how the VM Agent writes logs to stdout and to a rotated log file on disk.
>
> | Field | Description |
> | - | - |
> | **Log level** | Minimum severity written to stdout (`debug`, `info`, `warn`, `error`). |
> | **Log format** | Output format: `console` (human-readable) or `json` (structured). |
>
> **Log rotation** subsection — settings for the on-disk rotated log file:
>
> | Field | Description |
> | - | - |
> | **Filename** | Path to the rotated log file. |
> | **File level** | Minimum severity written to the file. Captures levels that stdout suppresses. |
> | **Max size mb** | Maximum file size in MB before rotation. |
> | **Max backups** | Number of old rotated files to retain. |
> | **Max age days** | Days to keep old log files before deletion. |
> | **Compress** | Whether rotated files are gzip-compressed. |

> **Network**
>
> Port assignments for the VM Agent process and the OpenTelemetry Collector. Changing collector receiver
> ports restarts the collector and re-instruments affected workloads.
>
> **Vmagent** subsection:
>
> | Field | Description |
> | - | - |
> | **Cli port** | Port for the CLI WebSocket server used by `odictl`. |
> | **Pprof port** | Port for the VM Agent pprof profiling server. |
>
> **Collector** subsection:
>
> | Field | Description |
> | - | - |
> | **Health check port** | OTel Collector health-check endpoint. |
> | **Pprof port** | OTel Collector pprof profiling endpoint. |
> | **Zpages port** | OTel Collector zpages debugging endpoint. |
> | **Prometheus port** | OTel Collector Prometheus metrics endpoint. |
>
> **Collector receivers** subsection:
>
> | Field | Description |
> | - | - |
> | **OTLP gRPC port** | Port for the OTLP gRPC receiver. Changing this restarts instrumented sources. |
> | **OTLP HTTP port** | Port for the OTLP HTTP receiver. Changing this restarts instrumented sources. |

> **Central**
>
> Settings for connecting this VM Agent to an Odigos Central management plane. When **Enabled** is
> `false`, the remaining fields are ignored and the agent runs standalone.
>
> | Field | Description |
> | - | - |
> | **Enabled** | Enable or disable the Central controller. |
> | **Endpoint** | WebSocket endpoint of the Central server (`host:port` or URL). |
> | **Agent name** | Identifier for this VM Agent instance. |
> | **Platform type** | Platform identifier (e.g. `vm-agent`). |
> | **Version** | Reported agent version. |
> | **Deployment id** | Unique deployment identifier *(read-only)*. |
> | **Connection timeout** | Timeout in seconds for the initial connection. |
> | **Reconnection delay** | Delay in seconds between reconnection attempts. |
> | **Max reconnection attempts** | Maximum number of reconnection retries. |
> | **Verbose logging** | Enable detailed logging for Central communication. |
> | **Exit on connection failure** | Force the agent to exit if the Central connection fails. |

> **Process controller**
>
> Controls how the agent discovers and monitors processes on the host.
>
> | Field | Description |
> | - | - |
> | **Ignored exe paths** | List of executable paths excluded from process monitoring. |
> | **Enabled classifiers** | Toggle individual process classifiers (`docker`, `systemd`, `containerd`, `process`). |
> | **Service name rules** | Rules for extracting service names from process arguments using pattern/template matching. |

> **Data collection**
>
> Configuration passed to the OpenTelemetry Collector instance managed by the VM Agent.
>
> **Otel collector** subsection:
>
> | Field | Description |
> | - | - |
> | **Config** | Key-value pairs written as environment variables to the `odigos-otelcol.conf` file (e.g. `OTELCOL_OPTIONS`). |

> **Instrumentation controller**
>
> Controls how the agent instruments discovered processes. The Java instrumentation
> subsection targets Java-specific behavior.
>
> **Java instrumentation** subsection:
>
> | Field | Description |
> | - | - |
> | **Sdk type** | Java agent variant: `java-enterprise` or `java-ebpf-instrumentations`. |
> | **Use dynamic attach** | Whether to attach the Java agent at runtime without restarting the process. |
> | **Disabled instrumentations** | List of Java instrumentation names to turn off. |

> **Profiling**
>
> Enables **continuous CPU profiling** (eBPF, OTLP Profiles) on **odigos-otelcol**. This is separate from Go **pprof** (see **Pprof** below). See [Continuous Profiling](/vmagent/instrumentations/profiling/overview) for the full setup.
>
> | Field | Description |
> | - | - |
> | **profiling.enabled** | Turns on the `profiling` receiver and the `profiles` pipeline whenever a destination is configured with the `PROFILES` signal. Default `false`. |
>
> ```yaml theme={null}
> profiling:
>   enabled: true
> ```
>
> Toggling this field reloads the collector and the PID → `service.name` registry within seconds — no vm-agent restart is required.

> **Pprof**
>
> Controls the **Go `net/http/pprof`** debug endpoint for the **vm-agent process only**. This is for debugging the agent itself and has nothing to do with application continuous CPU profiling (see **Profiling** above for that).
>
> | Field | Description |
> | - | - |
> | **Enabled** | Expose pprof on `network.vmagent.pprof_port` (default `7771`). Enabled by default. |

## Instrumentation Rules Panel

Instrumentation Rules control how telemetry is recorded from your instrumented sources. A rule can be applied to
specific sources and instrumentation libraries, letting you control what attributes, headers, or payloads are collected
and how custom instrumentations behave.

For rule types and configuration options, see [Instrumentation rules overview](/vmagent/setup/configuration/instrumentation-rules/overview).

![Instrumentation Rules screen](https://mintcdn.com/odigos/vLCErpcm1crPpCjg/images/vmagent/odictl/odictl-rules-screen.png?fit=max\&auto=format\&n=vLCErpcm1crPpCjg\&q=85\&s=05ca5491bd38ef7ff5ff0795c089608d)


# Configuration Overview
Source: https://docs.odigos.io/vmagent/setup/configuration/overview

How to configure the Odigos VM Agent: odictl TUI and YAML, and where to add sources, destinations, and actions.

Configuration defines where telemetry comes from (sources), where it is sent (destinations), and optional processing (actions). You can manage all of these using the VM Agent's interactive tool or YAML files.

## Configuration methods

You can configure the Odigos VM Agent in two ways:

* **[odictl](/vmagent/setup/configuration/odictl)** — A terminal user interface (TUI) that you use to add and manage sources, destinations, and actions interactively.
* **YAML** — Each configuration task (sources, destinations, actions) can be done via YAML; the pages linked below include a YAML tab with examples.

## Log collection

When a [destination](/vmagent/setup/configuration/add-destinations) is configured with the `LOGS` signal, the VM Agent automatically collects application logs from **instrumented** sources and exports them. How logs are collected depends on the source kind:

| Source kind | How the VM Agent collects logs |
| - | - |
| `process` | Scans the process PID file descriptors for paths containing `*.log` |
| `docker` | Streams Docker logs for the instrumented container |
| `systemd` | Streams logs via the journald receiver for the service |

> [!NOTE]
>
> A non-root agent can only read the systemd journal if the `odigos` user is in `adm` and/or `systemd-journal`. See [Linux Permissions](/vmagent/setup/permissions#systemd-journal-group-logs).

> [!NOTE]
>
> The odictl **Logs** panel shows live logs for the `odigos-vmagent` and `odigos-otelcol` services themselves. That is separate from application OpenTelemetry logs exported to your destination.

## Configuration tasks

> **[Add sources](/vmagent/setup/configuration/add-sources)**
>
> Instrument systemd services, Linux processes, or Docker containers so the VM Agent collects their telemetry.

> **[Add destinations](/vmagent/setup/configuration/add-destinations)**
>
> Define where telemetry is sent (for example, OTLP endpoints or supported backends). Destinations with the `LOGS` signal turn on application log export automatically.

> **[Add actions](/vmagent/setup/configuration/actions/overview)**
>
> Transform, filter, or enrich telemetry before it is sent to destinations.

> **[Instrumentation rules](/vmagent/setup/configuration/instrumentation-rules/overview)**
>
> Control how telemetry is recorded from your sources (attributes, headers, payloads and custom instrumentation rules).

> **[Continuous profiling](/vmagent/instrumentations/profiling/overview)**
>
> Enable eBPF CPU profiling on the collector and export OTLP Profiles to Pyroscope.

> **[Connect to Odigos Central](/central/adding-connections/vmagent)**
>
> Connect this VM Agent host to Central for centralized visibility and management.

> [!TIP]
>
> Add a [destination](/vmagent/setup/configuration/add-destinations) first, then [sources](/vmagent/setup/configuration/add-sources), then [actions](/vmagent/setup/configuration/actions/overview) and [instrumentation rules](/vmagent/setup/configuration/instrumentation-rules/overview) as needed.

For definitions of sources, destinations, and actions, see [Key concepts](/vmagent/overview#key-concepts) in the VM Agent overview.


# Managing Service Resources
Source: https://docs.odigos.io/vmagent/setup/configuration/services-mem-limits-configuration

Configure systemd memory limits for the odigos-otelcol and odigos-vmagent services

The Odigos VM Agent runs as two systemd services on your host **when the agent is in root mode**:

* odigos-otelcol — the Odigos distribution of the OpenTelemetry Collector.
* odigos-vmagent — the VM Agent that discovers, instruments, and manages processes.

In the default **non-root** install, `odigos-otelcol` is a child of `odigos-vmagent` and they share one cgroup. Set memory limits on `odigos-vmagent` only (the packaged `nonroot.conf` already raises `MemoryMax` to 3G to cover both). Do not enable or edit `odigos-otelcol.service` in that mode.

You can cap how much memory each service is allowed to use with the systemd `MemoryMax` (hard limit)
and `MemoryHigh` (soft throttling limit) directives. This is useful on hosts where you need to
guarantee that the agent never competes with your application workloads for memory.

Both services are Go binaries, so you should also set the [`GOMEMLIMIT`](https://pkg.go.dev/runtime#hdr-Environment_Variables)
environment variable. This tells the Go runtime a soft memory ceiling and makes the garbage collector
run more aggressively as the process approaches it — reclaiming memory *before* systemd throttles or
OOM-kills the service.

## Default limits

Both services **ship with memory limits already configured** out of the box, so you only need the steps
on this page if you want to change them. The default values in the packaged unit files are:

```ini odigos-otelcol theme={null}
[Service]
Environment="GOMEMLIMIT=1500MiB"
LimitMEMLOCK=infinity
MemoryHigh=1800M
MemoryMax=2G
MemorySwapMax=0
```

```ini odigos-vmagent theme={null}
[Service]
Environment="GOMEMLIMIT=750MiB"
LimitMEMLOCK=infinity
MemoryAccounting=yes
MemoryHigh=750M
MemoryMax=1G
MemorySwapMax=0
```

> [!NOTE]
>
> In non-root mode the collector shares the agent's cgroup, so the relevant defaults are those in `nonroot.conf`: `MemoryHigh=2500M`, `MemoryMax=3G`, `GOMEMLIMIT=750MiB`. Override them on `odigos-vmagent`, not on `odigos-otelcol.service`.

> [!NOTE]
>
> `MemorySwapMax=0` disables swap usage for the service, and `MemoryAccounting=yes` enables per-service
> memory tracking so the limits above are enforced. `LimitMEMLOCK=infinity` is required by the eBPF
> profiler and is unrelated to the memory caps.

> [!WARNING]
>
> Editing systemd units requires **root / `sudo`** privileges. All commands on this page must be run
> as `root` or with `sudo`.

## How systemd memory limits work

| Directive | Behavior |
| - | - |
| **`MemoryHigh`** | Soft limit. When the service exceeds it, the kernel aggressively reclaims memory and throttles the process, but does **not** kill it. Use this as an early back-pressure signal. |
| **`MemoryMax`** | Hard limit. If the service cannot be kept under this value, the kernel's OOM killer terminates it (systemd then restarts it because both units use `Restart=on-failure`). |
| **`GOMEMLIMIT`** | Go runtime soft memory limit (environment variable). As the process nears this value the Go garbage collector works harder to keep usage down. It does not kill the process; it complements the systemd limits by avoiding the OOM condition in the first place. |

> [!TIP]
>
> Order your limits so the runtime reacts first: `GOMEMLIMIT` \< `MemoryHigh` \< `MemoryMax`.
> For example `GOMEMLIMIT=450MiB`, `MemoryHigh=475M`, and `MemoryMax=512M`. A common rule of thumb is
> to set `GOMEMLIMIT` to roughly 85–90% of `MemoryMax`, leaving headroom for non-heap memory (stacks,
> mmap, eBPF maps) so the GC — not the OOM killer — is the first line of defense.

> [!WARNING]
>
> `GOMEMLIMIT` accepts Go byte-size suffixes (`MiB`, `GiB`) — **not** systemd's `M`/`G`.
> For example, use `GOMEMLIMIT=450MiB`, not `GOMEMLIMIT=450M`.

## Editing the memory limits

The recommended way to change unit settings is a systemd **drop-in override**. This keeps your changes
separate from the packaged unit file, so they are preserved across upgrades of the Odigos packages.

**1. Open a drop-in override for the service**

Use `systemctl edit` to create an override file. Run the command for the service you want to limit:

```shell odigos-otelcol theme={null}
sudo systemctl edit odigos-otelcol.service
```

```shell odigos-vmagent theme={null}
sudo systemctl edit odigos-vmagent.service
```

This opens an editor for a new file at
`/etc/systemd/system/<service>.service.d/override.conf`.

**2. Add the memory limits**

Add a `[Service]` section with the limits you want. Set the systemd memory caps **and** the
`GOMEMLIMIT` environment variable for the Go runtime. For example, to cap `odigos-otelcol` at 512 MB:

```ini theme={null}
[Service]
MemoryHigh=475M
MemoryMax=512M
Environment=GOMEMLIMIT=450MiB
```

Do the same for `odigos-vmagent` with values that suit your host, for example:

```ini theme={null}
[Service]
MemoryHigh=240M
MemoryMax=256M
Environment=GOMEMLIMIT=220MiB
```

> [!NOTE]
>
> `MemoryHigh` / `MemoryMax` accept the usual systemd suffixes (`M`, `G`) or a percentage of total host
> memory (for example `MemoryMax=10%`); use `infinity` to remove a limit. `GOMEMLIMIT` uses Go
> byte-size suffixes (`MiB`, `GiB`) and can be set to `off` to disable the runtime limit.

**3. Reload systemd and restart the service**

Apply the new configuration and restart the affected service:

```shell odigos-otelcol theme={null}
sudo systemctl daemon-reload
sudo systemctl restart odigos-otelcol.service
```

```shell odigos-vmagent theme={null}
sudo systemctl daemon-reload
sudo systemctl restart odigos-vmagent.service
```

**4. Verify the applied limits**

Confirm that systemd is enforcing the values you set, including the `GOMEMLIMIT` environment variable:

```shell odigos-otelcol theme={null}
systemctl show odigos-otelcol.service -p MemoryHigh -p MemoryMax -p Environment
```

```shell odigos-vmagent theme={null}
systemctl show odigos-vmagent.service -p MemoryHigh -p MemoryMax -p Environment
```

You can also watch live memory usage of a running service with:

```shell theme={null}
systemctl status odigos-otelcol.service
```

## Editing the unit files directly (alternative)

If you prefer, you can edit the packaged unit files in place instead of using drop-ins. The unit files
are located at:

* `/etc/systemd/system/odigos-otelcol.service`
* `/etc/systemd/system/odigos-vmagent.service`

Add `MemoryHigh`, `MemoryMax`, and `Environment=GOMEMLIMIT=...` under the existing `[Service]`
section, then reload and restart:

```shell theme={null}
sudo systemctl daemon-reload
sudo systemctl restart odigos-otelcol.service
sudo systemctl restart odigos-vmagent.service
```

> [!NOTE]
>
> For `odigos-otelcol` you can alternatively set `GOMEMLIMIT` in its environment file at
> `/etc/odigos-otelcol/odigos-otelcol.conf` instead of using `Environment=` in the unit.

> [!WARNING]
>
> Changes made directly to the packaged unit files may be **overwritten when you upgrade the Odigos
> packages**. Prefer the drop-in override method above so your limits survive upgrades.

## Removing the limits

To remove your custom limits, delete the drop-in override and reload systemd:

```shell theme={null}
sudo rm /etc/systemd/system/odigos-otelcol.service.d/override.conf
sudo rm /etc/systemd/system/odigos-vmagent.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart odigos-otelcol.service
sudo systemctl restart odigos-vmagent.service
```


# Deploy as a Docker Container
Source: https://docs.odigos.io/vmagent/setup/docker

Run the Odigos VM Agent as a privileged Docker container — the containerized alternative to the Linux package.

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

The Docker deployment is the same Odigos VM Agent packaged as a privileged
container instead of an RPM/deb system package. It is meant for hosts where
installing a system package is not an option — immutable hosts, or fleets that
are managed entirely with Docker Compose.

## How it works

One agent container runs per host. Through the host's PID namespace and the
Docker socket, it discovers and instruments the host's **Docker containers**,
**systemd services**, and **regular Linux processes** with eBPF, and runs the
`odigos-otelcol` collector as a supervised child process (see
[How the collector runs](#how-the-collector-runs)).

Host `systemd` services are still first-class targets: languages that attach
directly (Go, C++, Java, OBI) need no `systemctl`, while env-injection languages
(Python, Node.js, .NET, PHP, Ruby) get a unit drop-in written to the host's
`/etc/systemd/system` and are restarted with the host's `systemctl`, which the
agent runs by entering the host's namespaces with `nsenter`.

## How the collector runs

In a [root-mode package install](/vmagent/setup/installation#root-and-non-root),
the agent manages `odigos-otelcol.service` as a separate systemd unit. A
**non-root** package install (the default) runs `odigos-otelcol` as a supervised
child of `odigos-vmagent` — the same model as this Docker deployment, where
there is no `systemd` inside the container:

* The agent process starts `odigos-otelcol` as a **supervised child process**
  when the container starts.
* The collector's lifecycle is tied to the container: stopping or restarting
  `odigos-vmagent` stops or restarts the collector with it. There is no
  `systemctl start/stop odigos-otelcol` equivalent.
* Agent and collector **share the container's stdout** (console format, same as
  the package install). Use `docker logs odigos-vmagent` to view both.
* With `--network host`, the collector listens on the host's network namespace,
  so instrumented workloads on the same host can reach it on `localhost`.

Collector configuration (receivers, exporters, processors, profiling, and so on)
is still managed through Odigos Central or `odictl`, the same as the package
install. [systemd memory limits](/vmagent/setup/configuration/services-mem-limits-configuration)
do not apply here — use Docker resource limits (for example Compose
`deploy.resources` or `docker run --memory`) if you need to cap the container.

## Requirements

> [!NOTE]
>
> Odigos eBPF-based data collection requires, at minimum, platforms that have underlying Linux kernel versions of 5.4.0.
>
> The required Linux capabilities depend on your kernel version:
>
> * **Kernel 5.4 – 5.7**: requires `CAP_SYS_ADMIN`.
> * **Kernel 5.8 and later**: requires `CAP_BPF` and `CAP_SYS_PTRACE`.
>
> We recommend running on kernel 5.8 or later when possible, as the more granular capabilities (`CAP_BPF` and `CAP_SYS_PTRACE`) follow the principle of least privilege compared to `CAP_SYS_ADMIN`.

* An Odigos Pro license token for the VM Agent.
* A **Linux** host — the agent instruments the host it runs on.
* Docker installed and running, with permission to start privileged containers.
* Ability to grant the [required permissions](#required-permissions) (privileged
  container, host namespaces, and host mounts). Review that section before
  deploying — these are hard requirements and cannot be relaxed.
* The Odigos VM Agent container image. \[!NOTE] Obtain the image from your Odigos support team.

## Step 1: One-time host setup

Create the agent's config directory on the host before starting the container.
It persists the agent's identity and configuration across container upgrades:

```shell theme={null}
sudo mkdir -p /var/lib/odigos-agent/config
```

## Step 2: Run the agent

**Docker Compose**

After [Step 1](#step-1-one-time-host-setup), start the agent with the
`compose.yml` bundled with the distribution:

```shell theme={null}
ODIGOS_ONPREM_TOKEN='<TOKEN>' ODIGOS_CENTRAL_ENDPOINT='<HOST>:<PORT>' docker compose up -d
```

> **compose.yml**
>
> ```yaml theme={null}
> services:
>   odigos-vmagent:
>     image: ${ODIGOS_VMAGENT_IMAGE:-odigos-vmagent:latest}
>     container_name: odigos-vmagent
>     restart: unless-stopped
>     # Privileged + host PID: eBPF attach and /proc/<pid> access to host processes.
>     privileged: true
>     pid: host
>     # Host cgroup namespace: docker-container classification reads /proc/<pid>/cgroup
>     # to map a PID to its container id. Without this, container processes are
>     # misclassified as regular processes.
>     cgroup: host
>     # Host networking: the collector/OpAMP listen on the host's interfaces and
>     # the default agent name is the host's hostname.
>     network_mode: host
>     init: true
>     environment:
>       # License token; required unless license.jwt is pre-provisioned in the
>       # config dir mount.
>       ODIGOS_ONPREM_TOKEN: ${ODIGOS_ONPREM_TOKEN:-}
>       # Optional: enables the Odigos Central connection (host:port or URL).
>       ODIGOS_CENTRAL_ENDPOINT: ${ODIGOS_CENTRAL_ENDPOINT:-}
>       # Optional: agent name reported to Central (defaults to the hostname).
>       ODIGOS_AGENT_NAME: ${ODIGOS_AGENT_NAME:-}
>     volumes:
>       - /var/run/docker.sock:/var/run/docker.sock:ro
>       - /var/lib/docker:/var/lib/docker:ro
>       - /var/odigos:/var/odigos
>       # Config and identity, persisted across upgrades (same path as the
>       # package install — this is the same product in a different runtime).
>       - /var/lib/odigos-agent/config:/var/lib/odigos-agent/config
>       # Host systemd unit dir: instrumentation drop-ins for env-injection
>       # systemd services land here. Omit on hosts that do not use systemd.
>       - /etc/systemd/system:/etc/systemd/system
>       - /sys/kernel/debug:/sys/kernel/debug
>     ulimits:
>       # eBPF maps on kernels that account them against RLIMIT_MEMLOCK.
>       memlock: -1
>       nofile:
>         soft: 65536
>         hard: 65536
> ```

**docker run**

After [Step 1](#step-1-one-time-host-setup), run the container directly:

```shell theme={null}
docker run -d --name odigos-vmagent \
  --restart unless-stopped \
  --privileged --pid host --network host --cgroupns host --init \
  --ulimit memlock=-1 --ulimit nofile=65536:65536 \
  -e ODIGOS_ONPREM_TOKEN='<TOKEN>' \
  -e ODIGOS_CENTRAL_ENDPOINT='<HOST>:<PORT>' \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /var/lib/docker:/var/lib/docker:ro \
  -v /var/odigos:/var/odigos \
  -v /var/lib/odigos-agent/config:/var/lib/odigos-agent/config \
  -v /etc/systemd/system:/etc/systemd/system \
  -v /sys/kernel/debug:/sys/kernel/debug \
  odigos-vmagent:latest
```

The `/etc/systemd/system` mount is only needed on a systemd host, for
env-injection systemd instrumentation; omit it otherwise.

Once the container is running, confirm the agent and collector are healthy:

```shell theme={null}
docker logs -f odigos-vmagent
```

## Configuration

The agent is configured entirely through environment variables at startup:

| Variable | Required | Effect |
| - | - | - |
| `ODIGOS_ONPREM_TOKEN` | Yes, unless `license.jwt` is pre-provisioned in the config directory | Written to `/var/lib/odigos-agent/config/license.jwt` at startup. |
| `ODIGOS_CENTRAL_ENDPOINT` | No | Enables the Odigos Central connection and sets its endpoint (`host:port` or URL). Omit to run without Central. |
| `ODIGOS_AGENT_NAME` | No | Agent name reported to Central. Defaults to the host's hostname. |

Everything else — [sources](/vmagent/setup/configuration/add-sources),
[destinations](/vmagent/setup/configuration/add-destinations), actions, and
instrumentation rules — is managed from Odigos Central or with `odictl`, exactly
as in the package deployment:

```shell theme={null}
docker exec -it odigos-vmagent odictl
```

The config tree is persisted on the host at `/var/lib/odigos-agent/config`, so the
agent keeps its deployment identity and configuration across container upgrades.
Values written to `config.yaml` take precedence over the environment-derived
defaults on later boots.

> [!NOTE]
>
> Pre-provisioning `license.jwt` in `/var/lib/odigos-agent/config` is an alternative
> to `ODIGOS_ONPREM_TOKEN`. When the token variable is set, it is the source of
> truth and overwrites an out-of-date license file.

## Connecting to Odigos Central

Unlike the package install (which connects through `odictl`/`config.yaml` and a
`systemctl` restart), the container connects to Central via the
`ODIGOS_CENTRAL_ENDPOINT` environment variable set at startup — the endpoint
must be reachable from the host. To learn more about Central, see
[Connecting VM Agent](/central/adding-connections/vmagent).

## Required permissions

The agent runs as a privileged container with host namespaces and several host
mounts. Each is required for a specific capability:

| Setting | Reason |
| - | - |
| `--privileged` | eBPF program loading, perf events, writing instrumentation files into target containers, and writing to `/var/odigos`. |
| `--pid host` | Classifying and attaching to host processes via `/proc/<pid>`. |
| `--cgroupns host` | Mapping a PID to its container id via `/proc/<pid>/cgroup`. Without it, every container process is misclassified as a regular process. |
| `--network host` | The collector and OpAMP server listen on the host's network namespace, reachable by instrumented workloads on `localhost`. |
| `--init` | Reaps child processes left by attach tooling. |
| `--ulimit memlock=-1` | eBPF maps on kernels that account them against `RLIMIT_MEMLOCK`. |
| `/var/run/docker.sock` (ro) | Container metadata: mapping PIDs to containers, service names, restart-based instrumentation. |
| `/var/lib/docker` (ro) | Executable path resolution through overlay2, and container JSON logs for the [logs pipeline](/vmagent/setup/configuration/overview#log-collection). |
| `/var/odigos` (rw) | Instrumentation files must live on the host so host processes can load them. |
| `/var/lib/odigos-agent/config` (rw) | Agent configuration and identity, persisted across container upgrades. |
| `/etc/systemd/system` (rw) | Env-injection systemd instrumentation writes unit drop-ins here so the host's `systemd` picks them up. Omit on non-systemd hosts (direct-attach systemd services still work without it). |
| `/sys/kernel/debug` | kprobes/tracepoints used by the eBPF instrumentations. |

## Upgrade

Pull the new image and recreate the container. The persisted
`/var/lib/odigos-agent/config` directory preserves the agent's identity and
configuration across the upgrade.

**Docker Compose**

```shell theme={null}
docker compose pull
docker compose up -d
```

**docker run**

```shell theme={null}
docker pull odigos-vmagent:latest
docker stop odigos-vmagent && docker rm odigos-vmagent
```

Then re-run the `docker run` command from [Step 2](#step-2-run-the-agent)
with the new image.

## Uninstall

```shell theme={null}
docker stop odigos-vmagent
docker rm odigos-vmagent
```

To fully remove the deployment, also drop the persisted identity and configuration:

```shell theme={null}
sudo rm -rf /var/lib/odigos-agent
```

## Troubleshooting

### How to view the logs

The agent and the `odigos-otelcol` collector share the container's stdout in
console format (same as the package install). Follow the live log stream:

```shell theme={null}
docker logs -f odigos-vmagent
```

### How to inspect or configure a running agent

Use `odictl` inside the container the same way you would on a package install —
list sources, add destinations, inspect instrumentation status, and so on:

```shell theme={null}
docker exec -it odigos-vmagent odictl
```

Config changes are written under `/var/lib/odigos-agent/config` on the host and persist
across container restarts.

### Why do container processes show up as regular processes?

The agent classifies a process as a Docker container by reading
`/proc/<pid>/cgroup` and mapping it to a container id. That mapping only works
when the agent shares the host's cgroup namespace.

If you started the container without `--cgroupns host` (`cgroup: host` in
Compose), every container process is misclassified as a regular process.
Recreate the container with the setting from [Step 2](#step-2-run-the-agent)
(see also [Required permissions](#required-permissions)).

### Why are systemd services in env-injection languages not instrumented?

Direct-attach languages (Go, C++, Java, OBI) do not need `systemd` drop-ins and
still work without the mount. Env-injection languages (Python, Node.js, .NET,
PHP, Ruby) need a unit drop-in written to the host's `/etc/systemd/system` so
the host's `systemd` picks up the instrumentation environment and restarts the
service.

If those services are not instrumented on a systemd host, the container is
almost certainly missing the `/etc/systemd/system` volume mount. Recreate the
container with the mount from [Step 2](#step-2-run-the-agent). Omit the mount
only on hosts that do not use `systemd`.


# Installation
Source: https://docs.odigos.io/vmagent/setup/installation

Install the Odigos VM Agent on your Linux host

> [!NOTE]
>
> The Odigos VM Agent is available as part of Odigos Pro. For access and licensing details,
> [contact us](https://odigos.io/).

> [!NOTE]
>
> This page installs the VM Agent as a Linux system package. To run the agent as
> a container instead — for example on immutable hosts or Compose-managed
> fleets — see [Docker Deployment](/vmagent/setup/docker).

## Configuring Package Sources (apt, dnf, rpm)

To install the Odigos VM Agent, configure your package manager or obtain the RPM files.

**DNF**

Using `dnf` (RHEL, Fedora, Rocky Linux):

```shell theme={null}
sudo rm /etc/yum.repos.d/odigos.repo
sudo tee /etc/yum.repos.d/odigos.repo > /dev/null <<'EOF'
[odigos]
name=Odigos Packages
baseurl=https://yum.odigos.io/projects/odigos-cloud/odigos-rpm
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://raw.githubusercontent.com/odigos-io/odigos/main/docs/keys/odigos-package-signing-keys.asc
EOF

sudo dnf makecache
```

**APT**

Using `apt` (Debian, Ubuntu):

```shell theme={null}
sudo mkdir -p /etc/apt/keyrings/

wget -q -O - https://apt.odigos.io/doc/repo-signing-key.gpg \
  | gpg --dearmor \
  | sudo tee /etc/apt/keyrings/odigos.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/odigos.gpg] https://apt.odigos.io/projects/odigos-cloud odigos-apt main" \
  | sudo tee /etc/apt/sources.list.d/odigos.list

sudo apt update
```

**RPM**

If you use a package manager other than `dnf` or `apt`, or prefer to install via RPM files, you can
install the VM Agent using RPM files. There are two files you will need to obtain:

* odigos-otelcol-\<otelcol-version>.rpm
* odigos-\<agent-version>.rpm

> [!NOTE]
>
> Obtain the rpm files from your Odigos support team

## Root and non-root

Fresh installs run as the unprivileged `odigos` user. The packaged unit still defaults to `User=root`; non-root mode is applied by the drop-in `/etc/systemd/system/odigos-vmagent.service.d/nonroot.conf` (capabilities, relocated writable trees, collector as a child process).

| Mode | When to use |
| - | - |
| **Non-root** (default) | Typical hosts. The agent runs as `odigos` with a fixed set of Linux capabilities. |
| **Root** | You need the agent to restart systemd units without a [polkit rule](/vmagent/setup/permissions#systemd-services), or the host cannot grant ambient capabilities (systemd older than 229). |

**Precedence** (first match wins):

1. Host cannot grant ambient capabilities (systemd older than 229 — RHEL 7, CentOS 7, default Amazon Linux 2) → **root**, even if you requested non-root. Check with `/usr/lib/odigos/bin/odigos-vmagent-caps --explain`.
2. `ODIGOS_AGENT_USER_MODE=root` or `nonroot` in the environment of the package script (`apt` / `dnf` / `rpm`)
3. Sticky file `/var/lib/odigos-agent/user-mode` (written automatically when the env is set; see below)
4. Heuristic: non-root — including an upgrade of a host that is still on root

> [!NOTE]
>
> Non-root extra permissions — [additional permissions](/vmagent/setup/permissions#additional-permissions) for restarting sources, journal groups for `LOGS`, and the capability set — are documented in [Linux Permissions](/vmagent/setup/permissions).

## Installation

**DNF**

Non-root (default), using `dnf` (RHEL, Fedora, Rocky Linux):

```shell theme={null}
sudo dnf install odigos
```

Root:

```shell theme={null}
sudo ODIGOS_AGENT_USER_MODE=root dnf install odigos
```

**APT**

Non-root (default), using `apt` (Debian, Ubuntu):

```shell theme={null}
sudo apt install odigos
```

Root:

```shell theme={null}
sudo ODIGOS_AGENT_USER_MODE=root apt install odigos
```

**RPM**

If you use a package manager other than `dnf` or `apt`, run the following to install the agent:



\> \[!NOTE]
\>
\> Replace \<otelcol-version> and \<agent-version> with the versions you downloaded.



Non-root (default):

```shell theme={null}
sudo rpm -ivh odigos-otelcol-<otelcol-version>.rpm odigos-<agent-version>.rpm
```

Root:

```shell theme={null}
sudo ODIGOS_AGENT_USER_MODE=root rpm -ivh odigos-otelcol-<otelcol-version>.rpm odigos-<agent-version>.rpm
```

To persist the choice independently of the install environment (the reliable method — `apt` / `dnf` may not forward `ODIGOS_AGENT_USER_MODE` into the package script):

```shell theme={null}
# Before install or upgrade
echo root | sudo tee /var/lib/odigos-agent/user-mode
# or
echo nonroot | sudo tee /var/lib/odigos-agent/user-mode
```

## Apply the license

`odictl` validates the token **before** writing it, so an invalid JWT cannot replace a working license. The file is stored at `/var/lib/odigos-agent/config/license.jwt` (the same path in root and non-root mode).

```shell theme={null}
sudo odictl -odigos-license '<JWT>'
```

Wrap the token in single quotes so the shell does not alter it. Do not write the JWT into the file by hand — use `odictl` so it is validated first.

> [!NOTE]
>
> The config directory is owned by the agent user (`odigos` in non-root mode, `root` in root mode) and is not world-writable. Use `sudo` for `odictl -odigos-license` and for any YAML you place under `/var/lib/odigos-agent/config`.

## Enable the VM Agent

Enable and start `odigos-vmagent`. It does not start itself on a fresh install.

```shell theme={null}
sudo systemctl enable --now odigos-vmagent
```

> [!WARNING]
>
> **Do not** enable `odigos-otelcol.service` on a non-root install. The package disables that unit so it does not fight for ports. The non-root agent starts `odigos-otelcol` as a **child process** of `odigos-vmagent`.
>
> In root mode the agent manages `odigos-otelcol.service` itself over systemd D-Bus. Enabling the collector unit separately is not required.

Confirm the agent is running:

```shell theme={null}
sudo systemctl status odigos-vmagent
```

If you run Odigos Central, follow [Connecting VM Agent](/central/adding-connections/vmagent) to link this host to Central.

## Layout

Config, license, sources, destinations, actions, and rules live under one tree for both root and non-root:

| Path | Purpose |
| - | - |
| `/var/lib/odigos-agent/config/` | Agent configuration and `license.jwt` |
| `/var/lib/odigos-agent/user-mode` | Sticky `root` / `nonroot` preference |
| `/var/lib/odigos-agent/log/` | Agent logs in non-root mode |
| `/var/odigos/` | Instrumentation files (both modes; do not relocate) |
| `/etc/odigos-otelcol/` | Collector configuration |
| `/etc/systemd/system/odigos-vmagent.service.d/nonroot.conf` | Non-root user, capabilities, and environment |

## Next steps

* [Linux Permissions](/vmagent/setup/permissions) — capabilities, Docker group, journal groups for `LOGS`, and polkit for systemd restarts
* [Add destinations](/vmagent/setup/configuration/add-destinations) and [add sources](/vmagent/setup/configuration/add-sources)
* [odictl](/vmagent/setup/configuration/odictl)


# Linux Permissions
Source: https://docs.odigos.io/vmagent/setup/permissions

Capabilities, Linux groups, and polkit rules the Odigos VM Agent needs when it runs as the unprivileged odigos user.

By default the VM Agent runs as the unprivileged `odigos` user, not as root. It still needs a small, fixed set of Linux capabilities, and — depending on which sources you instrument — membership in a few host groups. This page covers what is granted out of the box, why, and how to change it.

> [!NOTE]
>
> Root-mode installs (`ODIGOS_AGENT_USER_MODE=root`) already have every privilege. Skip this page unless you are tightening a non-root install or switching a host to non-root. See [Installation](/vmagent/setup/installation#root-and-non-root).

## Linux capabilities

The packaged drop-in `/etc/systemd/system/odigos-vmagent.service.d/nonroot.conf` grants the same capability set on every distribution that can run non-root.

```ini theme={null}
AmbientCapabilities=CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYSLOG CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_NET_RAW CAP_NET_ADMIN CAP_SETUID CAP_SETGID
CapabilityBoundingSet=CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYSLOG CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_NET_RAW CAP_NET_ADMIN CAP_SETUID CAP_SETGID
```

Ambient capabilities are inherited by child processes, so they also cover `odigos-otelcol` when the non-root agent runs it as a child.

| Capability | Why the agent needs it |
| - | - |
| `CAP_SYS_ADMIN` | Load eBPF programs, call `perf_event_open` (uprobes and profiling), and read `/proc/<pid>/exe`. Covers the finer `CAP_BPF` / `CAP_PERFMON` / `CAP_CHECKPOINT_RESTORE` bits that `capabilities(7)` still attributes to it. |
| `CAP_SYS_PTRACE` | Discover and instrument processes owned by other users. |
| `CAP_DAC_READ_SEARCH` | Read root-owned `/sys/kernel/tracing` and other DAC-restricted `/proc` paths. Without it, process discovery fails and the `odictl` Sources list stays empty. |
| `CAP_SYSLOG` | Read `/proc/kallsyms` addresses when `kptr_restrict=1` (the Debian/Ubuntu default), needed to resolve kernel symbols. |
| `CAP_CHOWN` | Hand `odigos-otelcol` secret and log files to the `odigos` user. |
| `CAP_DAC_OVERRIDE` | Write root-owned host files such as `/etc/ld.so.preload` (dynamic-loader instrumentation). |
| `CAP_FOWNER` | Replace root-owned scratch left by an earlier root-mode agent in sticky (`1777`) directories: `/tmp/keeplist` and each container's `/tmp/ebpf-java-*-dynamic.json`. Only the owner may unlink or rename there; `CAP_DAC_OVERRIDE` does not lift that rule. |
| `CAP_NET_RAW` | Open the `AF_PACKET` socket the eBPF instrumenter binds its socket filter to. Dropping this is not a partial loss: the instrumenter's pipelines share one error group, so it exits at startup and **no** eBPF traces are produced. `CAP_SYS_ADMIN` does not substitute for this. |
| `CAP_NET_ADMIN` | Attach the TC programs that carry trace context across hops (TCP-option propagation). |
| `CAP_SETUID` + `CAP_SETGID` | Match a target JVM's credentials for HotSpot dynamic attach, which the JVM accepts only from an identical euid/egid. Dropping them leaves JVMs owned by other users (including all containerized ones) instrumentable only by restarting them. If you choose to remove this capability, also set `use_dynamic_attach=false` in the VM Agent configuration (see [Configuration Sections](/vmagent/setup/configuration/odictl#configuration-sections)). |

> [!NOTE]
>
> The installer does **not** pick capabilities from `kernel.perf_event_paranoid`, and it never assigns `CAP_BPF`, `CAP_PERFMON`, or `CAP_CHECKPOINT_RESTORE`.

> [!NOTE]
>
> Provisioning `/var/exchange` still needs root. The drop-in handles that with `ExecStartPre=+/usr/bin/install ...` (the `+` prefix runs that line as root regardless of `User=odigos`).

### Why `CAP_SYS_ADMIN` on every distro

The capability **list** does not change between RHEL and Debian. `CAP_SYS_ADMIN` is always included because the narrower bits are not enough on the kernels and sysctl defaults these distributions ship:

**RHEL / Rocky / Fedora**

| Distro | Typical kernel | Why `CAP_SYS_ADMIN` |
| - | - | - |
| RHEL 7, CentOS 7, Amazon Linux 2 | 3.10 / 4.14 | **Non-root is not available.** systemd is older than 229, so `AmbientCapabilities=` does not exist. The installer forces root even if you request non-root. Confirm with `/usr/lib/odigos/bin/odigos-vmagent-caps --explain`. |
| RHEL 8, Rocky 8 | 4.18 | `CAP_BPF` and `CAP_PERFMON` do not exist until Linux 5.8. eBPF load and `perf_event_open` still require `CAP_SYS_ADMIN`. |
| RHEL 9, Rocky 9, Amazon Linux 2023 | 5.14+ | The finer bits exist, but `capabilities(7)` still attributes those operations to `CAP_SYS_ADMIN`, and `perf_event_open` often still needs it. The package therefore grants `CAP_SYS_ADMIN` rather than `CAP_BPF` / `CAP_PERFMON`. |

**Debian / Ubuntu**

| Distro | Typical kernel | Why `CAP_SYS_ADMIN` |
| - | - | - |
| Debian 11–13, Ubuntu 20.04 and later | 5.4+ | The default `kernel.perf_event_paranoid` is `3` or `4`. That sysctl requires `CAP_SYS_ADMIN` for `perf_event_open` (uprobes and profiling) even when `CAP_PERFMON` is present. Debian/Ubuntu also set `kptr_restrict=1`, which is why `CAP_SYSLOG` is in the set. |

Inspect what the helper decided on this host:

```shell theme={null}
/usr/lib/odigos/bin/odigos-vmagent-caps --explain
```

### Override, add, or remove capabilities

> [!WARNING]
>
> Modifying Linux capabilities may impair or break VM Agent functionality. Alter these settings at your own risk or under the guidance of Odigos Support.

systemd **merges** `AmbientCapabilities=` and `CapabilityBoundingSet=` across drop-in files. A later drop-in can **add** capabilities; it cannot remove one that `nonroot.conf` already listed. To drop or replace capabilities, edit the packaged drop-in itself.

**1. Edit the packaged drop-in**

```shell theme={null}
sudo systemctl cat odigos-vmagent.service
sudo vi /etc/systemd/system/odigos-vmagent.service.d/nonroot.conf
```

Change `AmbientCapabilities=` and `CapabilityBoundingSet=` together so they stay in sync. To **remove** a capability, delete it from both lines. To **add** one, append it to both lines — or, if you only want to add, create a later drop-in instead:

```ini theme={null}
# /etc/systemd/system/odigos-vmagent.service.d/99-extra-caps.conf
[Service]
AmbientCapabilities=CAP_YOUR_EXTRA
CapabilityBoundingSet=CAP_YOUR_EXTRA
```

**2. Stop the package from overwriting your edit**

Package upgrades re-render `nonroot.conf` only while its first line is:

```
# Managed by the odigos-vmagent package.
```

Replace that header with your own comment (for example `# Operator-managed capabilities`) so the next upgrade leaves the file alone.

**3. Restore the SELinux label (RHEL, Rocky, Fedora)**

Skip this on Debian and Ubuntu. A drop-in edited by hand can receive the wrong SELinux label, and systemd will refuse to read it:

```shell theme={null}
sudo restorecon -Rv /etc/systemd/system/odigos-vmagent.service.d
```

**4. Reload systemd and restart the agent**

```shell theme={null}
sudo systemctl daemon-reload
sudo systemctl restart odigos-vmagent
```

> [!NOTE]
>
> To restore the packaged set after experimenting, put the managed header back and run:
>
> ```shell theme={null}
> sudo /usr/lib/odigos/bin/odigos-vmagent-caps --refresh
> sudo systemctl daemon-reload
> sudo systemctl restart odigos-vmagent
> ```

Dropping a capability does not prevent the agent from starting. The matching feature is disabled or degraded, and the agent logs which capability was missing at startup.

## Additional permissions

A non-root agent needs extra host rights to instrument and restart some sources. Root-mode installs do not need these steps.

### Docker containers

Instrumenting and controlling **Docker** sources requires access to `/var/run/docker.sock`. That socket is owned by `root:docker`. The package adds the `odigos` user to the `docker` group when the group already exists.

If Docker was installed **after** the agent, or the group was missing at install time, add it yourself:

```shell theme={null}
sudo usermod -aG docker odigos
sudo systemctl restart odigos-vmagent
```

> [!WARNING]
>
> Group membership is applied when the process starts. `usermod` alone is not enough — restart `odigos-vmagent` so it picks up the new supplementary groups.

If the `docker` group does not exist yet:

```shell theme={null}
sudo groupadd docker
sudo usermod -aG docker odigos
sudo systemctl restart odigos-vmagent
```

Confirm:

```shell theme={null}
id odigos
# uid=... gid=... groups=... docker
```

Without this group, Docker discovery fails with `EACCES` on the socket: the agent cannot list containers, instrument them, or restart them.

### systemd services

Instrumenting a systemd source writes a unit drop-in (`CAP_DAC_OVERRIDE` covers the write) and then asks systemd over D-Bus to `daemon-reload` and restart the unit. systemd asks **polkit** for that. A system service has no interactive session, so without a polkit rule those calls fail with:

```
Reload daemon failed: Interactive authentication required.
```

The package does **not** install a polkit rule. Until you add one, a non-root agent cannot complete systemd instrumentation on its own. Docker and plain-process sources are unaffected.

Choose one of the following:

**Install a polkit rule**

Check which polkit your host uses. RHEL 8+, Rocky, Fedora, Debian 12+, and Ubuntu 24.04 load **JavaScript** rules from `/etc/polkit-1/rules.d/`. Debian 11 and Ubuntu 20.04 / 22.04 still use **`.pkla`** files.

**RHEL / Rocky / Fedora**

Create `/etc/polkit-1/rules.d/49-odigos-vmagent.rules` owned by `root:root`, mode `0644`:

```javascript theme={null}
polkit.addRule(function(action, subject) {
    if (subject.user !== "odigos") {
        return;
    }
    if (action.id === "org.freedesktop.systemd1.reload-daemon") {
        return polkit.Result.YES;
    }
    if (action.id === "org.freedesktop.systemd1.manage-units") {
        var verb = action.lookup("verb");
        if (verb === "start" || verb === "stop" || verb === "restart" ||
            verb === "try-restart" || verb === "reload" ||
            verb === "reload-or-restart") {
            return polkit.Result.YES;
        }
    }
});
```

```shell theme={null}
sudo chmod 0644 /etc/polkit-1/rules.d/49-odigos-vmagent.rules
sudo restorecon -v /etc/polkit-1/rules.d/49-odigos-vmagent.rules
sudo systemctl restart polkit
```

**Debian / Ubuntu**

**Debian 12+ and Ubuntu 24.04** — JavaScript rules. Create `/etc/polkit-1/rules.d/49-odigos-vmagent.rules`:

```javascript theme={null}
polkit.addRule(function(action, subject) {
    if (subject.user !== "odigos") {
        return;
    }
    if (action.id === "org.freedesktop.systemd1.reload-daemon") {
        return polkit.Result.YES;
    }
    if (action.id === "org.freedesktop.systemd1.manage-units") {
        var verb = action.lookup("verb");
        if (verb === "start" || verb === "stop" || verb === "restart" ||
            verb === "try-restart" || verb === "reload" ||
            verb === "reload-or-restart") {
            return polkit.Result.YES;
        }
    }
});
```

```shell theme={null}
sudo chmod 0644 /etc/polkit-1/rules.d/49-odigos-vmagent.rules
sudo systemctl restart polkit
```

**Debian 11 and Ubuntu 20.04 / 22.04** — `.pkla` files (polkit 0.105). Create `/etc/polkit-1/localauthority/50-local.d/49-odigos-vmagent.pkla`:

```ini theme={null}
[Allow odigos to reload systemd]
Identity=unix-user:odigos
Action=org.freedesktop.systemd1.reload-daemon
ResultAny=yes
ResultInactive=yes
ResultActive=yes

[Allow odigos to manage units]
Identity=unix-user:odigos
Action=org.freedesktop.systemd1.manage-units
ResultAny=yes
ResultInactive=yes
ResultActive=yes
```

Then restart polkit (`sudo systemctl restart polkit` or `sudo systemctl restart polkitd`).

The rule lets the `odigos` user reload systemd and start, stop, or restart units. Narrow the `verb` list further if your policy requires it. `.pkla` files cannot filter by verb; they grant the whole action.

> [!WARNING]
>
> This grants the `odigos` user the ability to restart **any** systemd unit, not only Odigos-managed sources. Scope or omit the rule if that is not acceptable, and use one of the alternatives instead.

**Run the agent as root**

Root-mode agents talk to systemd without polkit:

```shell theme={null}
echo root | sudo tee /var/lib/odigos-agent/user-mode
sudo ODIGOS_AGENT_USER_MODE=root apt install --reinstall odigos   # Debian / Ubuntu
# sudo ODIGOS_AGENT_USER_MODE=root dnf reinstall odigos          # RHEL / Rocky
```

See [Installation](/vmagent/setup/installation#root-and-non-root).

**Restart units yourself**

Leave the agent non-root. After it writes the instrumentation drop-in, reload and restart the unit as an operator:

```shell theme={null}
sudo systemctl daemon-reload
sudo systemctl restart <unit>.service
```

#### journald permissions

When a destination includes the `LOGS` signal, the collector reads **systemd** source logs from the journal (a shared `journald` receiver). A non-root `odigos` user cannot read the journal unless it is in a group that journald allows.

Add `odigos` to an appropriate group below based on your Linux Distribution, then restart the agent:

**Debian / Ubuntu**

Members of `adm` or `systemd-journal` can read the systemd journal:

```shell theme={null}
sudo usermod -aG adm,systemd-journal odigos
sudo systemctl restart odigos-vmagent
```

**RHEL / Rocky / Fedora**

journald grants read access to `systemd-journal`:

```shell theme={null}
sudo usermod -aG systemd-journal odigos
sudo systemctl restart odigos-vmagent
```

This is only required for `LOGS` on **systemd** sources. Process log files and Docker container logs do not use the journal.

> [!NOTE]
>
> The odictl **Logs** panel shows logs for the agent and collector units themselves. That is separate from application logs exported to your destination, and does not require these groups.

### Processes

No extra group or polkit rule is required. The capabilities in [Linux capabilities](#linux-capabilities) are enough to discover and instrument host processes, including restart-based instrumentation where the agent starts the process itself. Process log files are found via the process file descriptors; they do not use the systemd journal.


# System Requirements
Source: https://docs.odigos.io/vmagent/setup/system-requirements

To ensure optimal performance and compatibility when deploying the Odigos VM Agent, your environment must meet the following system requirements.

* Odigos Pro license key (or token) for the VM Agent. Apply it after install with `sudo odictl -odigos-license '<JWT>'` — see [Installation](/vmagent/setup/installation#apply-the-license).
* A Linux host, plus deployment-specific requirements:
  * **[Package installation](/vmagent/setup/installation)** — a RHEL-compatible
    distribution (DNF), a Debian-based distribution (APT), or any Linux system
    supporting RPM packages.
  * **[Docker deployment](/vmagent/setup/docker)** — Docker installed and
    running, with permission to start privileged containers. See also the
    [required permissions](/vmagent/setup/docker#required-permissions) on that
    page.
  * **[Amazon ECS](/vmagent/ecs/overview)** — an ECS cluster on the **EC2**
    launch type (Fargate is not supported) with container instances on kernel
    5.10+ with BTF. See the
    [ECS system requirements](/vmagent/ecs/requirements) for the full list.
* At least one process or application on the host that is written in a
  [supported language](/enterprise/instrumentations/overview).

## Kernel and systemd

Odigos eBPF-based data collection requires Linux kernel **5.4** or later. Prefer **5.8** or later when possible.

Non-root mode (the package default) additionally requires **systemd 229 or later**, which is when `AmbientCapabilities=` became available. On older systemd the installer runs the agent as **root**, even if you request non-root.

| Distro | Typical kernel | systemd | Default agent mode |
| - | - | - | - |
| RHEL 7, CentOS 7, Amazon Linux 2 | 3.10 / 4.14 | 219 | Root only (`AmbientCapabilities` unavailable) |
| RHEL 8, Rocky 8 | 4.18 | 239+ | Non-root |
| RHEL 9, Rocky 9, Amazon Linux 2023 | 5.14+ | 250+ | Non-root |
| Debian 11–13, Ubuntu 20.04+ | 5.4+ | 245+ | Non-root |

Confirm what this host can grant:

```shell theme={null}
/usr/lib/odigos/bin/odigos-vmagent-caps --explain
```

Expected output:

```text theme={null}
systemd version      : 257
kernel               : 6.12.0-211.46.1.el10_2.x86_64
ambient capabilities : supported
capability set       : CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_DAC_READ_SEARCH CAP_SYSLOG CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_NET_RAW CAP_NET_ADMIN CAP_SETUID CAP_SETGID
```

The non-root capability set is **the same on every distro** and always includes `CAP_SYS_ADMIN`. It does not switch to `CAP_BPF` / `CAP_PERFMON` on newer kernels. See [Linux Permissions](/vmagent/setup/permissions) for the full list, why each bit is required, and how to add or remove capabilities.

## Disk space

The host must provide the following minimum free disk capacity. If these paths
share a filesystem, the combined free space must meet the sum of the individual
requirements.

> [!NOTE]
>
> These values are planning estimates based on typical agent usage and may
> change in future releases. Size `/tmp` according to your diagnose bundle
> configuration if you raise the default bundle size.

| Path | Minimum free | Purpose |
| - | - | - |
| `/var` | 500 MB | Instrumentation tools and agent logs |
| `/usr` | 500 MB | Agent binaries and packages |
| `/etc` | 2 MB | Agent configuration |
| `/tmp` | 55 MB | Diagnose bundle generation (50 MB default bundle size) and temporary files (5 MB) |


# Uninstall
Source: https://docs.odigos.io/vmagent/setup/uninstall

Uninstall the Odigos VM Agent from your Linux host

> [!NOTE]
>
> Running the agent as a container? See [Docker Deployment](/vmagent/setup/docker#uninstall)
> for how to remove it.

To uninstall the Odigos VM Agent, run the following based on your Linux package manager:

**DNF**

Using `dnf` (RHEL, Fedora, Rocky Linux):

```shell theme={null}
sudo dnf remove odigos
```

**APT**

Using `apt` (Debian, Ubuntu):

```shell theme={null}
sudo apt remove odigos
```

**RPM**

If you use a package manager other than `dnf` or `apt`, run the following to uninstall the agent:



\> \[!NOTE]
\>
\> Replace \<otelcol-version> and \<agent-version> with the versions you downloaded.



```shell theme={null}
sudo rpm -ev odigos odigos-otelcol
```

Removing the package leaves configuration and instrumentation data on disk. To fully clean the host:

```shell theme={null}
sudo rm -rf /var/lib/odigos-agent /var/odigos
sudo rm -rf /etc/systemd/system/odigos-vmagent.service.d
sudo rm -f /etc/polkit-1/rules.d/49-odigos-vmagent.rules
```

The `odigos` user and group are not removed automatically. Remove them only if no other software uses them:

```shell theme={null}
sudo userdel odigos
sudo groupdel odigos
```


# Upgrade
Source: https://docs.odigos.io/vmagent/setup/upgrade

Upgrade the Odigos VM Agent on your Linux host

> [!NOTE]
>
> Running the agent as a container? See [Docker Deployment](/vmagent/setup/docker#upgrade)
> for how to upgrade it.

To upgrade the Odigos VM Agent, run the following based on your Linux package manager:

**DNF**

Using `dnf` (RHEL, Fedora, Rocky Linux):

```shell theme={null}
sudo dnf upgrade odigos
```

**APT**

Using `apt` (Debian, Ubuntu):

```shell theme={null}
sudo apt upgrade odigos
```

**RPM**

If you use a package manager other than `dnf` or `apt`, run the following to upgrade the agent:



\> \[!NOTE]
\>
\> Replace \<otelcol-version> and \<agent-version> with the versions you downloaded.



```shell theme={null}
sudo rpm -Uvh odigos-otelcol-<otelcol-version>.rpm odigos-<agent-version>.rpm
```

## What an upgrade changes

Starting from version v0.1.106, every upgrade switches the agent to **non-root**, including a host that has been running as root. During the upgrade, the package:

* Moves leftover `/etc/odigos-vmagent/*` into `/var/lib/odigos-agent/config/`, then deletes `/etc/odigos-vmagent`
* Moves agent logs from `/var/log` into `/var/lib/odigos-agent/log/`
* Chowns `/var/odigos` and `/etc/odigos-otelcol` to the `odigos` user
* Adds `odigos` to the `docker` group when that group exists
* Disables `odigos-otelcol.service` (the collector becomes a child of the agent)
* Re-renders `/etc/systemd/system/odigos-vmagent.service.d/nonroot.conf` unless you replaced its `# Managed by the odigos-vmagent package` header

> [!NOTE]
>
> Config and logs still move to `/var/lib/odigos-agent` even if you stay on root. Only the service user stays `root`.

> [!WARNING]
>
> To **stay on root**, set the mode before the upgrade:
>
> ```shell theme={null}
> echo root | sudo tee /var/lib/odigos-agent/user-mode
> sudo ODIGOS_AGENT_USER_MODE=root dnf upgrade odigos    # or apt upgrade odigos
> ```
>
> Hosts that cannot grant ambient capabilities (systemd older than 229) stay on root automatically.

After a root-to-non-root migration, review [Additional permissions](/vmagent/setup/permissions#additional-permissions).

## Migrate an existing root install

To switch a running root host to non-root without waiting for the next package upgrade:

```shell theme={null}
sudo /usr/lib/odigos/bin/odigos-migrate-to-nonroot --dry-run
sudo /usr/lib/odigos/bin/odigos-migrate-to-nonroot
```

`--dry-run` prints the plan only. The migrate tool performs the same steps as [an upgrade](#what-an-upgrade-changes).