Container Logs Collector Configuration

The Container Logs collector configuration launches a preconfigured OpenTelemetry Collector custom resource to tail all Kubernetes pod logs with the filelog receiver. By opting into the available pipeline you can control where the collected logs are sent. In this preconfigured custom resource, the collector will run as a DaemonSet in order to capture all logs on each node.

Prerequisites

Setup an IAM role to associate with the service account

An IAM Role with the following policies must be created for the following Kubernetes service account and namespace.

Note: Only attach the minimum set of policies necessary for your advanced configuration.

The IAM Roles for Service Accounts documentation (IRSA) contains instructions for creating the IAM role. The following examples will use eksctl to achieve this.

To create this IAM role, run the following command:

Copy
eksctl create iamserviceaccount \
    --name adot-col-container-logs \
    --namespace opentelemetry-operator-system \
    --cluster <your_cluster_name> \
    --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
    --approve \
    --role-only

This IAM role generated by the above command needs to be inserted into the annotations field of the advanced configuration as seen below:

Copy
collector:
  containerLogs:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: <iam_role_arn>

Container Logs pipelines

The following pipelines are available for the Container Logs preconfigured custom resource. Pipelines can be enabled by setting their enabled field to true.

Logs

• cloudwatchLogs

  Logs collected by a filelog receiver are sent to Amazon CloudWatch Logs.

Container Logs Advanced Configuration pipeline

Copy
collector:
  containerLogs:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: <iam_role_arn>
    pipelines:
      logs:
        cloudwatchLogs:
          enabled: true

Container Logs exporters

The following exporters can be configured for the Container Logs preconfigured custom resource.

awscloudwatchlogs

• log_group_name

  The name for the Amazon CloudWatch log group. Must follow pattern: [\\.\\-_/#A-Za-z0-9]+

• log_stream_name

  The name for the Amazon CloudWatch log stream. Must follow pattern: [^:*]*

Container Logs Advanced Configuration exporters

Copy
collector:
  containerLogs:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: <iam_role_arn>
    pipelines:
      logs:
        cloudwatchLogs:
          enabled: true
    exporters:
      awscloudwatchlogs:
        log_group_name: <log_group_name>
        log_stream_name: <log_stream_name>

Container Logs configurable values

Shown below is the complete list of configurable fields, along with their default values, for the containerLogs resource. $CLUSTER_NAME refers to the name of the EKS cluster and $NODE_NAME refers to the name of the Kubernetes node in the daemonset.

Copy
collector:
  containerLogs:
    resources:
      limits:
        cpu: 1000m
        memory: 750Mi
      requests:
        cpu: 300m
        memory: 512Mi
    serviceAccount:
      annotations:
    pipelines:
      logs:
        cloudwatchLogs:
          enabled: false
    exporters:
      awscloudwatchlogs:
        log_group_name: $CLUSTER_NAME/container/logs
        log_stream_name: $NODE_NAME

*Note that in Fargate, resource requests and limits must be equal, see this troubleshooting guide for more information.

Previous Topic: Add-on Advanced Configuration: Collector Deployment

Related Topics:

Collector Configuration for Prometheus Metrics

Collector Configuration for OTLP Ingest

Next Topic: Updating and Cleanup