Set up AWS Distro for OpenTelemetry Collector On-Premises

In this tutorial, we will demonstrate installing AWS Distro for OpenTelemetry Collector (ADOT Collector) binaries for on-premises hosts. For supported platforms, see the ADOT Collector README on GitHub.

Install the ADOT Collector

1. Set up the required AWS credential file.
2. Logon your on-premise host and download ADOT Collector source code from AWS Observability GitHub Repository and build RPM file with the following command.

Copy
git clone https://github.com/aws-observability/aws-otel-collector.git
make package-rpm

3. Install the ADOT Collector RPM by the following command on the host.

Copy
sudo rpm -Uvh ./aws-otel-collector.rpm

4. After RPM is installed, it will create ADOT Collector in directory /opt/aws/aws-otel-collector/ with the following folder layout.

   

5. After ADOT Collector is installed on the host, the following commands can be used to start, stop, and check the running status. a) Start ADOT Collector with CTL script. The config.yaml is optional; if it’s not provided, the default config is applied.

   Copy
   sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -c </path/config.yaml> -a start

   b) Stop ADOT Collector process

   Copy
   sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl -a stop

   c) Check the status of ADOT Collector

   Copy
   sudo /opt/aws/aws-otel-collector/bin/aws-otel-collector-ctl  -a status

Configure AWS IAM Roles Anywhere

AWS Introduced IAM Roles Anywhere to allow workloads to obtain temporary security credentials in IAM. ADOT can leverage this service to obtain the credentials needed for the exporters that target AWS (CloudWatch EMF, X-Ray, Amazon Managed Service for Prometheus).

In order to leverage IAM Roles Anywhere on your on-premises environment you'll need to create:

• A trust anchor that is trusted by both AWS and your Certificate Authority of choice.
• An IAM Role for ADOT Collector with proper permissions to interact with Amazon Managed Services for Prometheus.
• A profile to specify what roles can be assumed by your workload through the trust anchor in IAM Roles Anywhere.
• End user certificate used by ADOT Collector to obtain temporary AWS credentials.

1. A Trust Anchor is a reference to a Certificate Authority Certificate trusted by you. You have two options:

• Use AWS Private Certificate Authority (AWS Private CA) that integrates with IAM Roles Anywhere on the same account.
• Use an external Certificate Authority by importing the CA Certificate Body in AWS in Privacy Enhanced Mail (PEM) format.

You can create an IAM Roles Anywhere trust anchor following the AWS documentation:

2. Create an IAM Role with the permissions needed for your workload. An example of the IAM policies can be found in the AWS Documentation for Amazon Managed Service for Prometheus, and create a trust policy to allow IAM Roles Anywhere service to assume the role on behalf of your workload as described in the AWS Documentation

It's recommended to include conditions in the trust policy using attributes from the X.509 certificate. For example the following trust policy restricts the actions by using the certificate Subject Common Name (CN) attribute.

Copy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:SetSourceIdentity",
                "sts:TagSession"
            ],
            "Condition": {
                "StringEquals": {
                "aws:PrincipalTag/x509Subject/CN": "VM01"
            }
        }            
        }
    ]
}

3. Create a profile on IAM Roles Anywhere to match the IAM Role created in the previous step with the Trust Anchor created on Step 1.

4. Create a private key pair and end user certificate for your workload. Instructions to perform this operation depends on your OS as well as the Certificate Authority of choice. An example of how to generate and end user certificate for AWS Private CA can be found in the AWS Documentation.

Configuring ADOT Collector to use IAM Roles Anywhere

1. Install credential helper tool (aws_signing_helper) as instructed in the AWS documentation. Ensure the tool is included in the system PATH.

2. Create a home folder for the aoc user, copy the X509 certificate and private key.

   Copy
   mkdir /home/aoc/.x509
   mv <x509_private_key> /home/aoc/.x509/private-key.pem
   mv <x509_certificate> /home/aoc/.x509/cert.pem
   chown -R aoc:aoc /home/aoc/.x509/

3. Create an AWS SDK configuration (config) to use credential helper tool to generate temporary credentials using the provided X509 key and certificate. You'll need to provide the following values from your AWS Environment. You can find more information in the AWS documentation:

• TA_ARN: --trust-anchor-arn ARN of Trust anchor to to use for authentication.
• PROFILE_ID_ARN: --profile-arn ARN of the profile to pull policies from.
• REMOTE_ROLE: --role-arn Target role to assume.

Note that we stored the certificate and private keys in the aoc user home folder inside the .x509 directory. If you use a different path you'll need to update the configuration accordingly.

Also note that - due to a limitation in the AWS Go SDK, the entire credential_process line must be on a single line in the config file.

Copy
export TA_ARN=<Trust Anchor ARN>
export PROFILE_ID_ARN=<Profile ID ARN>
export REMOTE_ROLE=<Role ARN with AWS permission>

cat > config << EOF
[default]
credential_process = aws_signing_helper credential-process --certificate /home/aoc/.x509/cert.pem --private-key /home/aoc/.x509/private-key.pem --trust-anchor-arn $TA_ARN --profile-arn $PROFILE_ID_ARN --role-arn $REMOTE_ROLE
EOF

sudo chown aoc:aoc config
sudo mv config /home/aoc/.x509/

4. Add AWS_CONFIG_FILE and AWS_SDK_LOAD_CONFIG environment variable to the ADOT Collector configuration by adding an entry in the .env file used to load the service. Note that this file is only loaded for systemd enabled Linux distributions. For other systems you might need to make additional modifications to set the environment variable before running the service.

Copy
echo "AWS_CONFIG_FILE=/home/aoc/.x509/config" | sudo tee -a /opt/aws/aws-otel-collector/etc/.env
echo "AWS_SDK_LOAD_CONFIG=true" | sudo tee -a /opt/aws/aws-otel-collector/etc/.env

5. Force systemd to reload configuration from disk (installing the RPM above added the service to systemd) and restart the ADOT Collector to use the newly configured role. Note that the commands below assume you are using a systemd enabled Linux distribution. For other systems you may need to make additional modifications to leverage your operating system's service controller.

Copy
sudo systemctl daemon-reload
sudo systemctl restart aws-otel-collector.service