AWS Open Distro for OpenTelemetry

Using CloudWatch Container Insights with AWS Distro for OpenTelemetry

Using CloudWatch Container Insights with AWS Distro for OpenTelemetry

CloudWatch Container Insights collects, aggregates, and summarizes metrics from your containerized applications and microservices. Data is collected as log events using embedded metric format. These log events are entries that use a structured JSON schema that enables high-cardinality data to be ingested and stored at scale. Amazon CloudWatch can create the aggregated CloudWatch metrics from the received EMF data.

CloudWatch Container Insights uses the CloudWatch agent to collect both infrastructure metrics and Prometheus metrics. To migrate existing customers to use AWS Distro for OpenTelemetry, we have enhanced the AWS OTel Collector to support the same CloudWatch Container Insights experience.

Collecting infrastructure metrics

CloudWatch Container Insights collects metrics for many resources such as CPU, memory, disk, and network. It also provides diagnostic information such as container restart failures. The metrics are aggregated at the cluster, node, pod, task, and service level as CloudWatch metrics. To collect the infrastructure metrics, AWS OTel Collector uses the AWS Container Insights Receiver and the CloudWatch embedded metric format Exporter.

The following platforms are supported:

Collecting Prometheus metrics

CloudWatch Container Insights monitoring for Prometheus automates the discovery of Prometheus metrics from containerized systems and workloads. To collect the Prometheus metrics, the AWS OTel Collector uses ECSObserver extension, Prometheus Receiver and the CloudWatch embedded metric format Exporter.

The following platforms are supported:

Pre-built dashboards for workloads with Prometheus metrics

CloudWatch console has auto dashboard for workloads listed in following table.

EnvironmentAWS App MeshJava JMXNGINXNGINX PlusHAProxyMemcached
EKSYYYNYY
ECSYYYYNN

Workload without pre-build dashboard can still send metrics to CloudWatch. However, you need to build your CloudWatch dashboard manually.

Export Prometheus metrics format

Prometheus metrics in Container Insights are exported as both logs and metrics. The embedded metric format exporter matches metrics based on the metrics_declaration configuration to add extra metadata in structured logs. CloudWatch extracts embedded metric format log data to metrics automatically. For example assume that you have an EKS cluster called eks-1 and a ECS cluster called ecs-1. You will see the following resources in the CloudWatch console. The names of the log groups contain the cluster name. In CloudWatch, all clusters are saved in one metric namespace, with different dimension values used to differentiate metrics from different clusters.

EnvironmentLog groupMetrics Namespace
EKS/aws/containerinsights/eks-1/prometheusContainerInsights/Prometheus
ECS/aws/ecs/containerinsights/ecs-1/prometheusECS/ContainerInsights/Prometheus