AWS Distro for OpenTelemetry Collector Deployment Types

When setting up the ADOT Collector, you will generally decide between two different types of deployment, sidecar or service. A sidecar deployment runs a process of the Collector next to each process of each of your applications while a service would have a Collector process shared by multiple applications in your system.

Sidecar

In a sidecar deployment, you will run an instance of the Collector for each instance of each application in your system. For example, if you are using Kubernetes, then you will commonly have several deployments, each corresponding to a different microservice, and each deployment will have several pods distributed across zones for scalability and reliability. With a sidecar deployment, each pod will have a process running your application as well as a process running the collector, communicating with each over localhost.

By having the Collector running with your application, it has the most visibility into the state of the system and will be able to provide the most functionality from the collector. For example, observability data can be tagged with the Kubernetes pod which the application is running on because it is the same as the application.

The tradeoff of the sidecar approach is the increased resource usage of the collector. By deploying in all your pods, it effectively adds memory and CPU usage that goes up at the same rate as the number of application instances you have. The collector can run fine with relatively low dedicated resources, but you may need to tweak some settings to make sure it doesn't have an oversized impact on your resource utilization. In particular, you should always use the memory limiter to ensure the Collector does not use up memory needed by your application. For example,

Copy
processors:
  memory_limiter:
    limit_mib: 20

will configure the Collector to keep memory usage around 20MiB.

Service

In a service deployment, you will run a set number of instances of the Collector which are accessed from all the applications in your system. For example, if you are using Kubernetes, then you will commonly have several deployments, each corresponding to a different microservice, and each deployment will have several pods distributed across zones for scalability and reliability. By deploying the Collector as a service, it will have its own deployment, independent of the others in the system. Applications will communiate with the Collector through a service endpoint.

Because the Collector runs independently from applications, it will not have visiblity into application-specific state, for example the Kubernetes pod running the application.

When running the Collector as a service, you will need to configure TLS as well to ensure communication from applications is secure. Receivers can be configured with TLS certificates, for example,

Copy
receivers:
  otlp:
    grpc:
      tls_settings:
        cert_file: /path/to/tls.crt
        key_file: /path/to/tls.key

Applications will also need to be configured with the certificate using the language-specific configuration for it.