Target Allocator

Installing the EKS Add-on gives you access to the Target Allocator (TA). Enabling the Target Allocator for an OpenTelemetry Collector custom resource decouples Prometheus service discovery and metrics collection. When enabled, the Target Allocator manages the configuration of the Prometheus receiver. Specifically, the Target Allocator allows for distribution of Prometheus scrape targets across a pool of Collectors. This functionality can be used to avoid the manual sharding of Prometheus scrape targets that can be required when horizontally scaling a single Prometheus receiver configuration. Visit the Target Allocator Readme for a deeper introduction.

Out of Order Samples

In some cases the Target Allocator can be used to solve Prometheus remote write out of order sample errors. A common reason Collector users will see out of order sample erorrs is due to multiple collectors scraping the same metric endpoint. Collector users may experience this when horizontally scaling Collectors with Prometheus receivers or daemonset deployments without manually sharding scrape targets. Both of these scenarios can lead to out of order sample erorrs without sharding of Prometheus scrape targets.

Getting Started

Enabling the target allocator requires setting OpenTelemeteryCollector.spec.targetAllocator.enabled to true. This will cause the Prometheus configuration within the Collector CRD to be rewritten to use an http_sd_config from the Target Allocator. The operator will also create a new deployment and service for this http_sd_config directive.

Copy
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
spec:
  mode: statefulset
  servicAccount: adot-collector
  targetAllocator:
    enabled: true
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]
            metric_relabel_configs:
            - action: labeldrop
              regex: (id|name)
              replacement: $$1
            - action: labelmap
              regex: label_(.+)
              replacement: $$1
    
    processors:
      batch:

    exporters:
      prometheusremotewrite:
        endpoint: ${AMP_REMOTE_WRITE_URL}
        auth:
          authenticator: sigv4auth
    
    extensions:
      sigv4auth:
        region: ${AWS_REGION}
        service: "aps"


    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [batch]
          exporters: [prometheusremotewrite]

Please reference the OpenTelemetry Operator documentation for more information on using the Target Allocator.

Useful links

Target Allocator API Spec

Scaling OpenTelmetery Collectors

Previous Topic: Kubernetes Attributes Processor

Next Topic: Updating and Cleanup