Configuring the Pipeline

Configurable Components

Cluster Gateway Collector - a collector that runs as a k8s Deployment. It receives the OpenTelemetry data from the Node Collectors, processes it, and exports it to the configured destinations.
Node Data Collection Collector - a collector that runs as a k8s DaemonSet. It collects / scrape the OpenTelemetry data from the applications running on the nodes and forwards it to the Cluster Gateway Collector.

Configuring the Pipeline

Odigos offers two main options for configuring the pipeline:

When configuring the pipeline, you must use only one of the two methods — either profiles, or directly editing the odigos-configuration ConfigMap.Using both methods will result in configurations overwriting each other, potentially leading to unexpected behavior.

1. Using Sizing Configuration

To configure resource sizing at install time, set the ResourceSizePreset value in your Helm chart when installing Odigos—either in your values.yaml file or by using --set ResourceSizePreset=<preset> in the helm install command. This approach streamlines the setup process and applies optimized defaults for common use cases. The available sizing options—size_s, size_m, size_l, and size_xl—are predefined profiles designed to simplify pipeline configuration. Each profile specifies a defined set of parameters, as detailed below: Don’t know which size is the right one for your cluster? Read more in Benchmarks. Cluster Gateway Collector:

Size	Minimum Replicas	HPA Maximum Replicas	Request CPU (m)	Limit CPU (m)	Request Memory (Mi)	Limit Memory (Mi)
`size_s`	1	5	300m	300m	300Mi	300Mi
`size_m`	2	8	1000m	1000m	600Mi	600Mi
`size_l`	3	12	1250m	1250m	850Mi	850Mi
`size_xl`	5	15	1500m	1500m	2000Mi	2000Mi

Node Data Collection Collector:

Size	Request Memory (Mi)	Limit Memory (Mi)	Request CPU (m)	Limit CPU (m)
`size_s`	150Mi	300Mi	150m	300m
`size_m`	250Mi	500Mi	250m	500m
`size_l`	500Mi	750Mi	500m	750m
`size_xl`	1024Mi	1024Mi	1000m	1000m

2. Advanced Pipeline Configuration

For advanced control over the pipeline, you can override the default sizing and fine-tune resource and scaling parameters for the collectorGateway and collectorNode. The way you apply these advanced settings depends on how you installed Odigos:

If you installed Odigos using the CLI: Edit the odigos-configuration ConfigMap directly in your cluster. This allows you to set advanced options for both collectorGateway and collectorNode (such as custom CPU/memory requests, limits, and replica counts).
If you installed Odigos using Helm: Make your changes in the values.yaml file or by using the --set flag during installation or upgrade. Do not edit the odigos-configuration ConfigMap directly, as Helm manages this resource and will overwrite manual changes.

This method gives you full control over the pipeline configuration, letting you override any default or profile-based settings to match your specific requirements. Below is an example of how to configure the collectorGateway and collectorNode parameters:

  # HYBRID CONFIGURATION APPROACH:
  # By default, all values below are automatically set based on the ResourceSizePreset values.yaml attribute describe above.
  # Uncomment and modify any value to override the automatic sizing for that specific parameter.
  #
  # IMPORTANT CONSTRAINTS:
  # 1. minReplicas must be less or equal to maxReplicas
  # 2. If you set limitMemoryMiB without requestMemoryMiB, request will equal limit
  # 3. If you set requestMemoryMiB without limitMemoryMiB, limit will equal request
  # 4. Same logic applies to CPU settings

collectorGateway:
  # the memory request for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource request
  # of the form "memory: <value>Mi".
  # default value is 600Mi
  # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value.
  # requestMemoryMiB: 600

  # the memory limit for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource limit
  # of the form "memory: <value>Mi".
  # default value is 600Mi
  # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value.
  # limitMemoryMiB: 600

  # the CPU request for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource request
  # of the form "cpu: <value>m".
  # default value is 1000m
  # If you set only requestCPUm, the limitCPUm will be set to the same value.
  # requestCPUm: 1000

  # the CPU limit for the cluster gateway collector deployment.
  # it will be embedded in the deployment as a resource limit
  # of the form "cpu: <value>m".
  # default value is 1000m
  # If you set only limitCPUm, the requestCPUm will be set to the same value.
  # limitCPUm: 1000

  # The number of replicas for the cluster gateway collector deployment.
  # Also uses in MinReplicas the HPA config.
  # minReplicas: 2
  # The maxReplicas in the HPA config.
  # maxReplicas: 8

  # sets the "limit_mib" parameter in the memory limiter configuration for the collector gateway.
  # it is the hard limit after which a force garbage collection will be performed.
  # if not set, it will be 50Mi below the memory limit.
  # memoryLimiterLimitMiB: 550

  # sets the "spike_limit_mib" parameter in the memory limiter configuration for the collector gateway.
  # note that this is not the processor soft limit, but the diff in MiB between the hard limit and the soft limit.
  # if not specified, this value will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit).
  # memoryLimiterSpikeLimitMiB: 110

  # the GOMEMLIMIT environment variable value for the collector gateway deployment.
  # this is when go runtime will start garbage collection.
  # if not specified, it will be set to 80% of the hard limit of the memory limiter.
  # goMemLimitMiB: 440

collectorNode:
  # RequestMemoryMiB is the memory request for the node collector daemonset.
  # it will be embedded in the daemonset as a resource request of the form "memory: <value>Mi"
  # default value is 250Mi
  # If you set only requestMemoryMiB, the limitMemoryMiB will be set to the same value.
  # requestMemoryMiB: 250

  # LimitMemoryMiB is the memory limit for the node collector daemonset.
  # it will be embedded in the daemonset as a resource limit of the form "memory: <value>Mi"
  # default value is 500Mi
  # If you set only limitMemoryMiB, the requestMemoryMiB will be set to the same value.
  # limitMemoryMiB: 500

  # the CPU request for the node collector daemonset.
  # it will be embedded in the daemonset as a resource request
  # of the form "cpu: <value>m".
  # default value is 250m
  # If you set only requestCPUm, the limitCPUm will be set to the same value.
  # requestCPUm: 250

  # the CPU limit for the node collector daemonset.
  # it will be embedded in the daemonset as a resource limit
  # of the form "cpu: <value>m".
  # default value is 500m
  # If you set only limitCPUm, the requestCPUm will be set to the same value.
  # limitCPUm: 500

  # this parameter sets the "limit_mib" parameter in the memory limiter configuration for the node collector.
  # it is the hard limit after which a force garbage collection will be performed.
  # if not set, it will be 50Mi below the memory limit.
  # memoryLimiterLimitMiB: 450

  # this parameter sets the "spike_limit_mib" parameter in the memory limiter configuration for the node collector.
  # note that this is not the processor soft limit, but the diff in Mib between the hard limit and the soft limit.
  # if not set, this will be set to 20% of the hard limit (so the soft limit will be 80% of the hard limit).
  # memoryLimiterSpikeLimitMiB: 90

  # the GOMEMLIMIT environment variable value for the node collector daemonset.
  # this is when go runtime will start garbage collection.
  # if not specified, it will be set to 80% of the hard limit of the memory limiter.
  # goMemLimitMiB: 360

Getting Help

If you have any issues, or require our assistance, please open an issue in GitHub, or reach out to us in Odigos Slack

​Configurable Components