Odigos exports data to AWS s3 in OTLP format. Data can be exported either in JSON format or in binary format (protobuf).

​
Configuration

To configure the AWS S3 exporter, you need to provide the following configuration:

  • Bucket Region - The AWS region where the bucket is located.
  • Bucket Name - The name of the bucket where the data will be stored.

The following configuration is optional:

  • Time granularity of S3 Bucket - Wether a new subdirectory should be created every minute or every hour. default is minute.
  • Marshaller (data format) - The format in which the data will be encoded. It can be either otlp_json or otlp_proto. Default is otlp_json.

​
Setting up AWS S3 Bucket

If you haven’t already, you need to create an S3 bucket to store your data. You can do it using the AWS Management Console or the AWS CLI.

For example, to create a bucket named otel-data in region us-east-1 using the AWS CLI, you can run the following command:

aws s3api create-bucket --bucket otel-data --region us-east-1

​
Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)

The follwoing command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace $EKS_CLUSTER_REGION with the region of your EKS cluster and $EKS_CLUSTER_NAME with the name of your EKS cluster (or set them as environment variables).

eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve

Then, create an IAM policy that allows Odigos to send data to your S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<BucketName>", "arn:aws:s3:::<BucketName>/*"]
    }
  ]
}

You can do it with the aws cli.

  • Replace otel-data with the name of your bucket if you used a different name.
  • The policy name is odigos-s3 in this example, you can use it or choose a different name. Take note of the policy ARN, you will need it in the next step.
aws iam create-policy 
--policy-name odigos-s3 
--policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::otel-data", "arn:aws:s3:::otel-data/*"] } ] }'

Or use the aws management console to create the policy.

The last step is to configure this IAM policy to be assumed by the EKS cluster. Use the following cli command as template and adjust it according to your setup:

  • make sure to replace the --attach-policy-arn value with the ARN of the policy you created in the previous step.
  • use your EKS cluster name and region.
eksctl create iamserviceaccount \
--name default \
--namespace odigos-system \
--cluster otel-cluster \
--region=us-east-1 \
--attach-policy-arn arn:aws:iam::123456789012:policy/odigos-s3 \
--override-existing-serviceaccounts \
--approve

​
Format

The aws s3 exporter supports two formats: otlp_json and otlp_proto.

Each batch of logs / metrics / spans will be written as a single s3 object in the specified bucket. The content of the object will be the serialized data in the OpenTelemetry Protocol format, represented as either JSON or Protocol Buffers.

For example, the following JSON string represents a single span in otlp_json format, from Odigos demo application:

{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"service.name","value":{"stringValue":"inventory"}},{"key":"odigos.device","value":{"stringValue":"python"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"k8s.namespace.name","value":{"stringValue":"default"}},{"key":"k8s.deployment.name","value":{"stringValue":"inventory"}},{"key":"k8s.pod.name","value":{"stringValue":"inventory-7b87f7566c-2d6fq"}},{"key":"container.name","value":{"stringValue":"inventory"}},{"key":"k8s.node.name","value":{"stringValue":"ip-192-168-4-178.ec2.internal"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9ee5a0de5802d3f1e7f93440f892509c","spanId":"88a4bfb5bfcaf297","parentSpanId":"f9d7bb9e9243df06","name":"/buy","kind":2,"startTimeUnixNano":"1707600957298456744","endTimeUnixNano":"1707600958303042894","attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"8080"}},{"key":"http.host","value":{"stringValue":"inventory:8080"}},{"key":"http.target","value":{"stringValue":"/buy?id=10"}},{"key":"net.peer.ip","value":{"stringValue":"192.168.13.194"}},{"key":"http.user_agent","value":{"stringValue":"Java/17.0.9"}},{"key":"net.peer.port","value":{"intValue":"49420"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/buy"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}

​
Adding a Destination to Odigos

Odigos makes it simple to add and configure destinations, allowing you to select the specific signals [traces/logs/metrics] that you want to send to each destination. There are two primary methods for configuring destinations in Odigos:

  1. Using the UI
    To add a destination via the UI, follow these steps:
    • Use the Odigos CLI to access the UI: Odigos UI
    odigos ui

  • In the left sidebar, navigate to the Destination page.

  • Click Add New Destination

  • Select AWS S3 and follow the on-screen instructions.

  1. Using kubernetes manifests

Save the YAML below to a file (e.g., destination.yaml) and apply it using kubectl:

kubectl apply -f destination.yaml

apiVersion: odigos.io/v1alpha1
kind: Destination
metadata:
  name: s3-example
  namespace: odigos-system
spec:
  data:
    S3_BUCKET: <Bucket Name>
    S3_MARSHALER: <Marshaller (Data Format) [otlp_json, otlp_proto]>
    S3_PARTITION: <Time granularity of S3 Bucket [minute, hour]>
    S3_REGION: <Bucket Region>
  destinationName: s3
  signals:
  - TRACES
  - METRICS
  - LOGS
  type: s3
OverviewAzure Blob Storage