> ## Documentation Index
> Fetch the complete documentation index at: https://docs.odigos.io/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS S3

> Configuring the AWS S3 backend (Managed)

### Getting Started

<img src="https://d15jtxgb40qetw.cloudfront.net/awss3.svg" alt="s3" className="not-prose h-20" />

**Setting up AWS S3 Bucket**

If you haven't already, you need to create an S3 bucket to store your data. You can do it using the [AWS Management Console](https://console.aws.amazon.com/s3/home) or the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html).

For example, to create a bucket named `otel-data` in region `us-east-1` using the AWS CLI, you can run the following command:

```bash theme={null}
aws s3api create-bucket --bucket otel-data --region us-east-1
```

<Warning>
  Make sure that you have the necessary `AWS Credentials` and permissions set up.<br />
  AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).
</Warning>

**Setting up AWS credentials with AWS EKS (Elastic Kubernetes Service)**

The following command will create a link between your EKS cluster and your IAM account, effectively allowing the EKS cluster to assume IAM roles and access AWS services.

Replace `$EKS_CLUSTER_REGION` with the region of your EKS cluster and `$EKS_CLUSTER_NAME` with the name of your EKS cluster (or set them as environment variables).

```bash theme={null}
eksctl utils associate-iam-oidc-provider --region=$EKS_CLUSTER_REGION --cluster=$EKS_CLUSTER_NAME --approve
```

**Setting up the IAM policy**

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<BucketName>", "arn:aws:s3:::<BucketName>/*"]
    }
  ]
}
```

You can do it with the aws cli.

* Replace `otel-data` with the name of your bucket if you used a different name.
* The policy name is `odigos-s3` in this example, you can use it or choose a different name. Take note of the policy ARN, you will need it in the next step.

```bash theme={null}
aws iam create-policy
--policy-name odigos-s3
--policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject" ], "Resource": ["arn:aws:s3:::otel-data", "arn:aws:s3:::otel-data/*"] } ] }'
```

Or use the aws management console to create the policy.

The last step is to configure this IAM policy to be assumed by the EKS cluster. Use the following cli command as template and adjust it according to your setup:

* make sure to replace the `--attach-policy-arn` value with the ARN of the policy you created in the previous step.
* use your EKS cluster name and region.

```bash theme={null}
eksctl create iamserviceaccount \
--name default \
--namespace odigos-system \
--cluster otel-cluster \
--region=us-east-1 \
--attach-policy-arn arn:aws:iam::123456789012:policy/odigos-s3 \
--override-existing-serviceaccounts \
--approve
```

**Format**

The aws s3 exporter supports two formats: `otlp_json` and `otlp_proto`.

Each batch of logs / metrics / spans will be written as a single s3 object in the specified bucket.
The content of the object will be the serialized data in the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto),
represented as either JSON or Protocol Buffers.

For example, the following JSON string represents a single span in `otlp_json` format, from Odigos demo application:

```json theme={null}
{"resourceSpans":[{"resource":{"attributes":[{"key":"telemetry.sdk.language","value":{"stringValue":"python"}},{"key":"telemetry.sdk.name","value":{"stringValue":"opentelemetry"}},{"key":"telemetry.sdk.version","value":{"stringValue":"1.19.0"}},{"key":"service.name","value":{"stringValue":"inventory"}},{"key":"odigos.device","value":{"stringValue":"python"}},{"key":"telemetry.auto.version","value":{"stringValue":"0.40b0"}},{"key":"k8s.namespace.name","value":{"stringValue":"default"}},{"key":"k8s.deployment.name","value":{"stringValue":"inventory"}},{"key":"k8s.pod.name","value":{"stringValue":"inventory-7b87f7566c-2d6fq"}},{"key":"container.name","value":{"stringValue":"inventory"}},{"key":"k8s.node.name","value":{"stringValue":"ip-192-168-4-178.ec2.internal"}}]},"scopeSpans":[{"scope":{"name":"opentelemetry.instrumentation.flask","version":"0.40b0"},"spans":[{"traceId":"9ee5a0de5802d3f1e7f93440f892509c","spanId":"88a4bfb5bfcaf297","parentSpanId":"f9d7bb9e9243df06","name":"/buy","kind":2,"startTimeUnixNano":"1707600957298456744","endTimeUnixNano":"1707600958303042894","attributes":[{"key":"http.method","value":{"stringValue":"POST"}},{"key":"http.server_name","value":{"stringValue":"0.0.0.0"}},{"key":"http.scheme","value":{"stringValue":"http"}},{"key":"net.host.port","value":{"intValue":"8080"}},{"key":"http.host","value":{"stringValue":"inventory:8080"}},{"key":"http.target","value":{"stringValue":"/buy?id=10"}},{"key":"net.peer.ip","value":{"stringValue":"192.168.13.194"}},{"key":"http.user_agent","value":{"stringValue":"Java/17.0.9"}},{"key":"net.peer.port","value":{"intValue":"49420"}},{"key":"http.flavor","value":{"stringValue":"1.1"}},{"key":"http.route","value":{"stringValue":"/buy"}},{"key":"http.status_code","value":{"intValue":"200"}}],"status":{}}]}]}]}
```

### Configuring Destination Fields

<Accordion title="Supported Signals:">
  ✅ Traces
  ✅ Metrics
  ✅ Logs
  ❌ Profiles
</Accordion>

* **S3\_BUCKET** `string` : Bucket Name. The name of the bucket where the data will be stored
  * This field is required
* **S3\_REGION** `string` : Bucket Region. The AWS region where the bucket is located
  * This field is required
* **S3\_PARTITION** `string` : Time granularity of S3 Bucket (Deprecated, use Partition Format). Deprecated field for time granularity. Use S3\_PARTITION\_FORMAT for custom formats.
  * This field is optional and defaults to `minute`
* **S3\_PARTITION\_FORMAT** `string` : S3 Partition Format. Custom partition format using strftime syntax (e.g., '%Y/%m/%d/%H/%M' for minute-level, '%Y/%m/%d/%H' for hour-level). If not specified, S3\_PARTITION will be used.
  * This field is optional
  * Example: `%Y/%m/%d/%H/%M`
* **S3\_MARSHALER** `string` : Marshaller (Data Format). The format in which the data will be encoded. It can be either `otlp_json` or `otlp_proto`. Default is `otlp_json`
  * This field is required and defaults to `otlp_json`

### Adding Destination to Odigos

There are two primary methods for configuring destinations in Odigos:

##### **Using the UI**

<Steps>
  <Step>
    Use the [Odigos CLI](https://docs.odigos.io/cli/odigos_ui) to access the UI

    ```bash theme={null}
    odigos ui
    ```
  </Step>

  <Step>
    Click on `Add Destination`, select `AWS S3` and follow the on-screen instructions
  </Step>
</Steps>

##### **Using Kubernetes manifests**

<Steps>
  <Step>
    Save the YAML below to a file (e.g. `s3.yaml`)

    ```yaml theme={null}
    apiVersion: odigos.io/v1alpha1
    kind: Destination
    metadata:
      name: s3-example
      namespace: odigos-system
    spec:
      data:
        S3_BUCKET: <Bucket Name>
        S3_MARSHALER: '<Marshaller (Data Format) (default: otlp_json) (options: [otlp_json,
          otlp_proto])>'
        S3_REGION: '<Bucket Region (options: [af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2,
          ap-northeast-3, ap-south-1, ap-south-2, ap-southeast-1, ap-southeast-2, ap-southeast-3,
          ap-southeast-4, ap-southeast-5, ap-southeast-7, ca-central-1, ca-west-1, eu-central-1,
          eu-central-2, eu-north-1, eu-south-1, eu-south-2, eu-west-1, eu-west-2, eu-west-3,
          il-central-1, me-central-1, me-south-1, mx-central-1, sa-east-1, us-east-1,
          us-east-2, us-gov-east-1, us-gov-west-1, us-west-1, us-west-2])>'
        # Note: The commented fields below are optional.
        # S3_PARTITION: <Time granularity of S3 Bucket (Deprecated, use Partition Format) (default: minute) (options: [minute, hour])>
        # S3_PARTITION_FORMAT: <S3 Partition Format>
      destinationName: s3
      signals:
      - TRACES
      - METRICS
      - LOGS
      type: s3
    ```
  </Step>

  <Step>
    Apply the YAML using `kubectl`

    ```bash theme={null}
    kubectl apply -f s3.yaml
    ```
  </Step>
</Steps>