Using CloudWatch Container Insights with AWS Distro for OpenTelemetry

CloudWatch Container Insights collects, aggregates, and summarizes metrics from your containerized applications and microservices. Data is collected as log events using embedded metric format. These log events are entries that use a structured JSON schema that enables high-cardinality data to be ingested and stored at scale. Amazon CloudWatch can create the aggregated CloudWatch metrics from the received EMF data.

CloudWatch Container Insights uses the CloudWatch agent to collect both infrastructure metrics and Prometheus metrics. To migrate existing customers to use AWS Distro for OpenTelemetry, we have enhanced the ADOT Collector to support the same CloudWatch Container Insights experience.

Collecting infrastructure metrics

CloudWatch Container Insights collects metrics for many resources such as CPU, memory, disk, and network. It also provides diagnostic information such as container restart failures. The metrics are aggregated at the cluster, node, pod, task, and service level as CloudWatch metrics.

Note: Collecting infrastructure metrics through awscontainerinsightreceiver requires running ADOT Collector with privileged root access as it also collects container-related data from cadvisor. For EKS deployments, use the Security Context to specify the root user as owner of the Collector process:

Copy
containers:
    - name: aws-otel-collector
      image: public.ecr.aws/aws-observability/aws-otel-collector:latest
      securityContext:
          runAsUser: 0
          runAsGroup: 0

For ECS deployments, use the Task Definition to specify the user owner of the collector process:

Copy
{
  "name": "aoc-collector",
  "image": "public.ecr.aws/aws-observability/aws-otel-collector:latest",
  "user": "root",
  ...
 }

The following platforms are supported:

• Amazon EKS and Kubernetes platforms on Amazon EC2
• Amazon EKS and Kubernetes platforms on Amazon Fargate
• Amazon ECS with cluster and service-level metrics
• Amazon ECS on Amazon EC2 with instance-level metrics (upcoming)

Collecting Prometheus metrics

CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. To collect the Prometheus metrics, the ADOT Collector uses ECSObserver extension, Prometheus Receiver and the CloudWatch embedded metric format Exporter.

The following platforms are supported:

• Amazon EKS and Kubernetes platforms on Amazon EC2
• Amazon EKS and Kubernetes platforms on Amazon Fargate
• Amazon ECS on Amazon EC2 and Fargate

Pre-built dashboards for workloads with Prometheus metrics

CloudWatch console has auto dashboard for workloads listed in following table.

Workload without pre-build dashboard can still send metrics to CloudWatch. However, you need to build your CloudWatch dashboard manually.

Export Prometheus metrics format

Prometheus metrics in Container Insights are exported as both logs and metrics. The embedded metric format exporter matches metrics based on the metrics_declaration configuration to add extra metadata in structured logs. CloudWatch extracts embedded metric format log data to metrics automatically. For example assume that you have an EKS cluster called eks-1 and a ECS cluster called ecs-1. You will see the following resources in the CloudWatch console. The names of the log groups contain the cluster name. In CloudWatch, all clusters are saved in one metric namespace, with different dimension values used to differentiate metrics from different clusters.