Tracing and Metrics with the AWS Distro for OpenTelemetry Python Auto-Instrumentation

Introduction

OpenTelemetry Python supports automatic instrumentation. It automatically produces spans with telemetry data describing the values used by the Python frameworks in your application without adding a single line of code. It is preconfigured for compatibility with AWS X-Ray and propagates the trace context across AWS services, it can also be used with any other tracing backend.

In this guide, we walk through the steps needed to trace an application and produce metrics with auto-instrumentation.

Requirements

Python versions 3.8 to 3.11 are required to run an application using OpenTelemetry.

Note: You’ll also need to have the ADOT Collector running to export traces and metrics.

Installation

The easiest way to download the packages needed for auto-instrumentation is using pip:

Copy
$ pip install aws-opentelemetry-distro

The aws-opentelemetry-distro package provides methods which configure the OpenTelemetry SDK with some basic defaults. These methods are used by Auto Instrumentation. The Instrumentors , opentelemetry-api, opentelemetry-sdk, and opentelemetry-instrumentation dependency packages are installed by default along with aws-opentelemetry-distro. And opentelemetry-instrumentation provides commands to detect, install, and initialize all instrumentation packages supported for your application’s dependencies. Notably, it installs the opentelemetry-instrument executables on your system. Check out the OpenTelemetry registry for a full list of instrumentation packages provided by OpenTelemetry Python.

Running an Application with Auto-Instrumentation

Auto-instrumentation uses the opentelemetry-instrument executable functions as a wrapper to automatically initialize the Instrumentors and start the provided application.

The AWS Distro is configured using two environment variables as OTEL_PYTHON_DISTRO="aws_distro" and OTEL_PYTHON_CONFIGURATOR="aws_configurator".

Start your application using auto-instrumentation can be as simple as the following:

Copy
$ OTEL_PYTHON_DISTRO="aws_distro" \
OTEL_PYTHON_CONFIGURATOR="aws_configurator" \
opentelemetry-instrument python3 ./path/to/your/app.py

Configuring Auto-Instrumentation

Environment variables are the primary way in which the OpenTelemetry SDK for Python is configured. For example:

• By default, aws-opentelemetry-distro uses the OTLP exporter and is configured to send data to a OpenTelemetry collector at http://localhost:4317 for both metrics and traces. The configuration of your SDK exporter depends on how you have configured your ADOT Collector. To learn more about how the ADOT Collector can be configured, refer to the ADOT Collector Documentation.

• If the Collector the application will connect to is running with TLS configured, the OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/my-cert.crt environment variable is used to provide a path to credentials to be used to establish a secure connection for the app’s exporter. The credentials at this path should be the public certificate of the collector, or one of its root certificates. If no certificate is found, the gRPC method ssl_channel_credentials() will attempt to “retrieve the PEM-encoded root certificates from a default location chosen by gRPC runtime” as explained in the gRPC Python Documentation.

• The random sampling rate for creating traces can be set through the environment variables OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.3 to configure a sampling rate of 30%. Sampling related configuration can be found in opentelemetry.sdk.trace.sampling submodule public doc.

More SDK configuration can be found in upstream opentelemetry SDK config public doc.

Using CloudWatch Application Signals

You can use CloudWatch Application Signals to automatically instrument your applications on AWS using ADOT python auto-instrumentation so that you can monitor current application health and track long-term application performance against your business objectives. Application Signals provides you with a unified, application-centric view of your applications, services, and dependencies, and helps you monitor and triage application health.

Get started with CloudWatch Application Signals

Using AWS X-Ray Remote Sampling

The ADOT Python auto-instrumentation can be configured to use AWS X-Ray remote sampling by setting the environment variable OTEL_TRACES_SAMPLER=xray. You will also need to configure the OpenTelemetry collector to allow the application to fetch sampling configuration. By default the sampler sends requests to http://localhost:2000. By setting OTEL_TRACES_SAMPLER_ARG environment variable, you can change the endpoint the sampler talks with when getting sampling configuration from AWS X-Ray Console. For example setting OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:4000 would configure the sampler to communicate with http://localhost:4000.

Using Manual Instrumentation

Because there can only be one global TracerProvider and MeterProvider, manual instrumentation should not instantiate its own TracerProvider or MeterProvider if used together alongside auto-instrumentation. Given that the same TracerProvider and MeterProvider is used, custom tracing and metrics works the same way when using automatic instrumentation or manual instrumentation. For information about custom trace instrumentation, see our docs on manual instrumentation.

Sample Application

See a Sample App using OpenTelemetry Python SDK Automatic Instrumentation.

NOTE:

• Python Web Frameworks like Flask and Django normally include a "reloader" when running in debug mode so that you can apply changes during development. This reloader will break auto-instrumentation because the app is restarted without giving OpenTelemetry a chance to wrap the instrumented libraries. When using debug mode, set the use_reloader=False as is done in the referenced sample app:

Copy
# See more: https://github.com/open-telemetry/opentelemetry-python-contrib/issues/546
if __name__ == "__main__":
    app.run(port=8082, debug=True, use_reloader=False)

• For Django applications, there are a few things that customer needs to be aware of when enabling ADOT Python auto-instrumentation. Some of them are called out in OTel Python Django Instrumentation doc:
  1. Use --noreload when running the Django application.
  2. Set the DJANGO_SETTINGS_MODULE environment variable to the location of your Django application’s settings.py file.
  3. When running a Django application in a container, set the PYTHONPATH environment variable to the location of your application’s working directory. Note that this is not called out in the OTel Python doc but is a known issue in Otel.