Container Insights EKS Prometheus

This section explains how to set up Prometheus monitoring with the ADOT Collector in a cluster running Amazon EKS or Kubernetes. With default configurations, the ADOT Collector automatically scrapes and imports metrics for the following workloads running in a cluster.

• AppMesh
• HAProxy
• JMX
• Memcached
• Nginx

To deploy these sample workloads, please follow the above links for guidance.

Set Up IAM Policy

To use the ADOT Collector to collect Prometheus metrics, make sure the following policies are attached to your node instance role.

Copy
AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly
CloudWatchAgentServerPolicy
AmazonSSMManagedInstanceCore
AWSXrayWriteOnlyAccess
AWSXRayDaemonWriteAccess
AmazonS3ReadOnlyAccess
AmazonPrometheusFullAccess

Deploy the ADOT Collector with default configuration

The default all-in-one configuration is built into the image of ADOT Collector and can be used to collect Prometheus metrics for all the sample workloads.

1. Set up variables to export metrics of your EKS cluster to the region where the logs should be published to.

Copy
export CLUSTER_NAME=<eks-cluster-name>
export AWS_REGION=<aws-region>

1. Download the example deployment file otel-container-insights-prometheus.yaml.

2. Deploy it to the cluster.

   Copy
   cat otel-container-insights-prometheus.yaml |
   sed "s/{{region}}/$AWS_REGION/g" | 
   sed "s/{{cluster_name}}/$CLUSTER_NAME/g" |
   kubectl apply -f -

3. Check the resources in the aws-otel-eks namespace.

   • Make sure Pods are running.

   Copy
   kubectl get all -naws-otel-eks

   • You should be able to see the message Everything is ready. Begin running and processing data by fetching logs from the Pod if the ADOT Collector is started correctly.

   Copy
   kubectl logs -naws-otel-eks $POD_NAME

Customize the ADOT Collector

To collect Prometheus metrics that are different from those specified in the default configuration, you can configure ADOT Collector with your own configuration.

Prepare Customized Configuration

1. In general, the processors section should be the same as what is shown below.

   Copy
   processors:
     resourcedetection/ec2:
       detectors: [ env ]
       timeout: 2s
       override: false
     resource:
       attributes:
         - key: TaskId
           from_attribute: job
           action: insert
         - key: receiver
           value: "prometheus"
           action: insert

   • resourcedetection/ec2 is a processor that reads environment variables and converts them to resource attributes. In the example of otel-container-insights-prometheus.yaml, it injects ClusterName which is later used by awsemf exporter to generate log_group_name .
   • resource is a processor that injects additional resource attributes. We copy the value from job to TaskId to generate log_stream_name. The attribute receiver: prometheus is an identifier for optimizing Prometheus metrics before awsemf exporter sending metric logs to CloudWatch.

2. Create ConfigMap otel-config in the aws-otel-eks namespace with your customized configuration so that it can be consumed by aws-otel-collector Pods.

   Be sure to rename {{filename}} with the actual filename.

   Copy
   kubectl create configmap otel-config --from-file={{filename}} -naws-otel-eks

Run ADOT Collector

Edit otel-container-insights-prometheus.yaml with the following changes:

1. Add volume otel-config.

   Copy
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   ...
   spec:
     replicas: 1
     ...
     template:
       spec:
         containers:
         ...
         volumes:
         - configMap:
             defaultMode: 420
             name: otel-config
           name: otel-config

2. Mount volume to the container and define command and arguments.

   Make sure that {{filename}} matches the key in the otel-config ConfigMap.

   Copy
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   ...
   spec:
     replicas: 1
     ...
     template:
       spec:
         containers:
         - name: aws-otel-collector
           command: ["/awscollector"]
           args: [ "--config", "/etc/otel/{{filename}}"]
           ...
           volumeMounts:
           - mountPath: /etc/otel/
             name: otel-config
             readOnly: true
         volumes:
         - configMap:
             defaultMode: 420
             name: otel-config
           name: otel-config

3. Finally roll out the updated deployment to the EKS cluster.

   Copy
   cat otel-container-insights-prometheus.yaml |
   sed "s/{{region}}/$AWS_REGION/g" | 
   sed "s/{{cluster_name}}/$CLUSTER_NAME/g" |
   kubectl apply -f -

View Your Data

1. View your Container Insights auto-dashboard

   Open the CloudWatch console. In the AWS Region where your cluster is running, choose Container Insights - Performance monitoring in the navigation panel. Then select EKS Prometheus {{workload_type}} and {{cluster_name}} from the drop down menu in the main panel.

   

2. View Your Metrics

   To see the CloudWatch metrics, choose Metrics in the navigation panel. The metric are in the ContainerInsights/Prometheus namespace.

   

3. View your logs

   To see the CloudWatch Logs events, choose Log groups in the navigation panel. The events are in the log group /aws/containerinsights/{{cluster_name}}/prometheus.