AWS Distro for OpenTelemetry

Tracing with the AWS Distro for OpenTelemetry Ruby SDK and X-Ray

Tracing with the AWS Distro for OpenTelemetry Ruby SDK and X-Ray

Introduction

With OpenTelemetry Ruby manual instrumentation, you configure the OpenTelemetry SDK within your application with just a few lines of code. OpenTelemetry Ruby then automatically produces trace spans with telemetry data describing the values used by the Ruby gems in your application. This telemetry data can then be exported to a backend like AWS X-Ray using the OpenTelemetry::Propagator::XRay::IDGenerator found in the ADOT Ruby opentelemetry-propagator-xray gem. We also strongly recommend using the OpenTelemetry::Propagator::XRay::TextMapPropagator propagator found in the same gem to support propagating the trace context across AWS services. This propagator handles the extraction and injecting of the AWS X-Ray Tracing header for requests from or to remote services.

In this guide, we walk through the steps needed to trace an application with manual instrumentation.



Requirements

Ruby 2.5 or later is required to run an application using OpenTelemetry according to the OpenTelemetry Ruby Documentation.

Note: You’ll also need to have the ADOT Collector running to export traces to X-Ray.



Installation

If you are using bundler, include the following gems in your Ruby application's Gemfile:

gem 'opentelemetry-exporter-otlp'
gem 'opentelemetry-sdk'


gem 'opentelemetry-propagator-xray'

Or, install them directly:

$ gem install opentelemetry-exporter-otlp \
              opentelemetry-sdk \
              opentelemetry-propagator-xray

Next, we'll use bundler to install gems that automatically instrument your application code.

OpenTelemetry Ruby distributes many gems that instrument well-known Ruby dependencies. You need to install the relevant instrumentation package for every dependency you want to generate traces for. To see supported gems, check out the OpenTelemetry Registry.

For example, use bundler and add the follow instrumentation gems to your Gemfile:

gem 'opentelemetry-instrumentation-faraday', '~> 0.19'
gem 'opentelemetry-instrumentation-rails', '~> 0.20'

Or, install them directly:

$ gem install opentelemetry-instrumentation-faraday -v '~> 0.19' \
              opentelemetry-instrumentation-rails -v '~> 0.20'


Setting up the Global Tracer

Sending Traces to AWS X-Ray

Manual Instrumentation with OpenTelemetry Ruby involves configuring the OpenTelemetry Ruby SDK. Below we discuss different methods you have for configuring the OpenTelemetry Ruby SDK.

Basic Configuration

This section describes recommended configuration to initialize OpenTelemetry Ruby SDK for tracing with AWS X-Ray.

For a ruby on rails application, OpenTelemetry Ruby Initialization Documentation recommends placing your configuration code in a Rails initializer. ADOT provides a working example of such an initializer in our sample app repo.

In a Ruby on Rails app, you will not need to require packages in your application code because of autoloading. This assumes you are using bundler and a Gemfile. Otherwise, if you included the gems with the require: false option or you are not using bundler, you will need to "require" the gems distributed by OpenTelemetry manually.

For all manually instrumented Ruby programs, you must use the OpenTelemetry::SDK.configure method below to configure the OpenTelemetry Ruby SDK.

The default OpenTelemetry OTLP Exporter with the Batch Processor is a great way to group traces and export them in a way that the ADOT Collector can receive them.

Additionally, using the X-Ray ID Generator is required to make your OpenTelemetry traces appear in X-Ray, while the X-Ray Propagator is strongly recommended in order to inject and extract the X-Ray Tracing header for downstream requests made by your application.

Putting this all together, we come up with the following:

1# Basic packages for your application
2require 'aws-sdk'
3require 'faraday'
4

5# Add imports for OTel components into the application
6require 'opentelemetry-api'
7require 'opentelemetry-exporter-otlp'
8require 'opentelemetry-sdk'
9

10# Import the gem containing the AWS X-Ray for OTel Ruby ID Generator and propagator
11require 'opentelemetry-propagator-xray'
12

13# Configure OpenTelmetry Ruby SDK
14OpenTelemetry::SDK.configure do |c|
15  # Set the service name to identify your application in the X-Ray backend service map
16  c.service_name = 'aws-otel-manual-rails-sample'
17

18  c.span_processors = [
19      # Use the BatchSpanProcessor to send traces in groups instead of one at a time
20      Trace::Export::BatchSpanProcessor.new(
21          # Use the default OLTP Exporter to send traces to the ADOT Collector
22          OpenTelemetry::Exporter::OTLP::Exporter.new(
23            # The ADOT Collector is running as a sidecar and listening on port 4318
24            endpoint="http://localhost:4318"
25          )
26      )
27    ]
28

29  # The X-Ray ID Generator generates spans with X-Ray backend compliant IDs
30  c.id_generator = OpenTelemetry::Propagator::XRay::IDGenerator
31

32  # The X-Ray Propagator injects the X-Ray Tracing Header into downstream calls
33  c.propagators = [OpenTelemetry::Propagator::XRay::TextMapPropagator.new]
34end

With this, your Ruby application has configured OpenTelemetry Ruby for compatibility with the AWS X-Ray service! To automatically trace popular Ruby gems, jump to the next section to learn about instrumenting with OpenTelemetry Ruby Instrumentations.

Advanced Configuration

From above, we learned that configuring OpenTelemetry Ruby required specifying 3 core steps

  • which Exporter to use to export to the ADOT Collector
  • which ID Generator to use to generate Trace IDs
  • which Propagator to use to propagate Trace Context to downstream calls

By default, OpenTelemetry Ruby SDK is already configured to initialize an OTLP exporter. The exporter can also be completely configured using environment variables.

export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

The OTEL_EXPORTER_OTLP_ENDPOINT value allows you to set the address that the exporter will use to connect to the collector. If unset, the SDK will try to connect to http://localhost:4318 by default. Note that because the scheme is http by default, you have to explicitly set it to be https if necessary.

If the Collector the application will connect to is running without TLS configured, the http scheme is used to disable client transport security for the OTLP exporter's connection. This option should never be used in production, non-sidecar deployments.

If the Collector the application will connect to is running with TLS configured, the https scheme and the certificate_file=/path/to/cert.pem argument should be used to give a path to credentials that allow the application to establish a secure connection for the app's exporter. The credentials at this path should be the public certificate of the collector, or one of its root certificates.

Next, because the AWS X-Ray ID Generator can only be configured through code, you cannot use an environment variable to select it. The ID Generator must be used at the time OpenTelemetry Ruby SDK is configured.

Finally, to allow the span context to propagate downstream when the application makes calls to external services, configure the global propagator to use the AWS X-Ray Propagator.

The OTEL_PROPAGATORS environment variable can be configured to have the OpenTelemetry Ruby SDK automatically find and initialize the propagator.

export OTEL_PROPAGATORS=xray

The propagator should be configured as soon as possible in your application's code so that subsequent downstream requests get the OpenTelemetry trace context injected into its HTTP headers. This is what allows your traces to be connected and for you to see a complete Service Graph in the X-Ray console.

Likewise, configuring the X-Ray Propagator means incoming requests to your application can parse out an OpenTelemetry Trace context and use the same Trace ID to pick up tracing where the upstream service left off.

You can combine the xray propagator with other propagators like tracecontext and b3 just fine, but it is recommended you put xray last because the propagator later in the list will override previous propagators.

Configuring Sampling

By default, the OpenTelemetry Ruby SDK follows the parent span's sampling decision if it exists, and samples 100% of incoming requests otherwise. This is known as the parentbased_always_on sampler.

Reduce Sampling Rate

To reduce the sampling rate, configure OpenTelemetry Ruby SDK to use the parentbased_traceidratio sampler. This can be configured using the OpenTelemetry Specification defined environment variables. For instance, to reduce the sampling rate to 10% of requests, set the following environment variables:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.10

Alternatively, this can be set on the global TracerProvider after the OpenTelemetry Ruby SDK has been configured:

1OpenTelemetry.tracer_provider.sampler = Samplers.parent_based(root: Samplers.trace_id_ratio_based(0.10))

Currently, OpenTelemetry Ruby does not support centralized sampling.

Debug Logging

By default, OpenTelemetry Ruby SDK logs at the info level. Its level can be configured using the OTEL_LOG_LEVEL environment variable.

export OTEL_LOG_LEVEL=debug

Separate from OpenTelemetry, you can use code and set the Base Logger to modify the logging level throughput your application.

1ActiveJob::Base.logger = Logger.new(STDOUT, level=Logger::DEBUG)

Additionally, you can create your own logger that logs at the log level you set.

1require 'logger'
2

3logger = Logger.new(STDOUT)
4logger.level = Logger::WARN
5

6logger.warn("This log message is visible!")
7logger.debug("This one is not.")


Instrumenting an Application

Warning: Some instrumentations are not yet stable and the attributes they collect are subject to change until the instrumentation reaches 1.0 stability. It is recommended to pin a specific version of an instrumentation

OpenTelemetry provides a wide range of instrumentations for popular Ruby libraries such as Rails, Sinatra, Faraday, the AwsSdk and many more. Instrumenting a library means that every time the library is used to make or handle a request, that library call is automatically wrapped with a populated span contain the relevant values that were used. Web framework, downstream HTTP, SQL, gRPC, and other requests can all be recorded using OpenTelemetry.

A full list of supported instrumentation packages and configuration instructions can be found on the instrumentation folder of the OpenTelemetry Ruby repo.

To enable tracing of the calls made by your package dependencies, you need to include the relevant Instrumentation classes during OpenTelemetry Ruby SDK initialization. Instrumentations have individual initialization configurability, so refer to the Instrumentation's documentation for configuration details.

1OpenTelemetry::SDK.configure do |c|
2  c.use 'OpenTelemetry::Instrumentation::Rails'
3  c.use 'OpenTelemetry::Instrumentation::Rack'
4  c.use 'OpenTelemetry::Instrumentation::ActionPack'
5  c.use 'OpenTelemetry::Instrumentation::ActiveSupport'
6  c.use 'OpenTelemetry::Instrumentation::ActionView'
7  # c.use 'OpenTelemetry::Instrumentation::ActiveRecord'
8

9  c.use 'OpenTelemetry::Instrumentation::Faraday'
10end

Alternatively, you can enable all Instrumentations which have been downloaded for this Ruby project. Not that you still need to download the Instrumentation gem for it to be initialized in the OpenTelemetry Ruby SDK.

1OpenTelemetry::SDK.configure do |c|
2  c.use_all()
3end

Instrumenting the AWS SDK

To instrument the AWS Ruby SDK and its dependencies, install the opentelemetry-instrumentation-aws_sdk OpenTelemetry Ruby Instrumentation gem for the AWS SDK.

If you are using bundler, you can include it in in the Gemfile.

gem 'opentelemetry-instrumentation-aws_sdk', '~> 0.2.1'

Otherwise you can install it directly using your shell.

$ gem install opentelemetry-instrumentation-aws_sdk -v '~> 0.2.1'

NOTE: Since these instrumentations are not yet stable, we recommend installing it at a pinned version.

To instrument requests made to services with the AWS SDK, configure the Ruby SDK as shown. We set suppress_internal_instrumentation to true because we want calls that go into the AWS SDK to be terminal requests without tracing underlying HTTP calls and other things which would make the trace noise-y.

1OpenTelemetry::SDK.configure do |c|
2  c.use 'OpenTelemetry::Instrumentation::AwsSdk', {
3    suppress_internal_instrumentation: true
4  }
5end

For more information refer to the upstream documentation for OpenTelemetry Ruby AWS SDK Instrumentation.



Custom Instrumentation

Creating Custom Spans

You can use custom spans to monitor the performance of internal activities that are not captured by instrumentation libraries. Note that only spans of kind Server are converted into X-Ray segments, all other spans are converted into X-Ray subsegments. For more on segments and subsegments, see the AWS X-Ray developer guide.

1require 'aws-sdk'
2require 'opentelemetry-api'
3

4# Get a tracer from the Global Tracer Provider
5tracer = OpenTelemetry.tracer_provider.tracer('my-tracer')
6

7tracer.in_span('Root Span', kind: :server) do |root_span|
8

9    p 'Started a root span, this will be a segment in the X-Ray console'
10

11    tracer.in_span('Child Span') do |child_span|
12

13        p 'Started a child span, this will be a subsegment in the X-Ray console'
14

15        ec2_client = Aws::EC2::Client.new
16        result = ec2_client.describe_instances
17

18        p "EC2 Describe Instances: #{result}"
19

20        p '<h1>Good job! Traces recorded!</h1>'
21    end
22end

See OpenTelemetry Ruby's own documentation on creating spans manually for more information.

Adding custom attributes

You can also add custom key-value pairs as attributes onto your spans. Attributes are converted to metadata by default. If you configure your collector, you can convert some or all of the attributes to annotations. To read more about X-Ray annotations and metadata see the AWS X-Ray Developer Guide.

One way to add custom attributes is as follows:

1require 'opentelemetry-api'
2

3# Get a tracer from the Global Tracer Provider
4tracer = OpenTelemetry.tracer_provider.tracer('my-tracer')
5

6tracer.in_span('Root Span',
7    attributes: {
8        'hello' => 'world',
9        'some.number' => 1024,
10        'tags' => [
11            'bug',
12            'enhancement'
13        ]
14    },
15    kind: :server) do |root_span|
16

17    p 'Started a root span'
18

19    span.set_attribute('my_attribute', 'foo')
20    span.set_attribute('more_items', ['bar', 'baz'])
21

22    span.add_attributes({
23        "yet.another.attribute" => "attribute value",
24        "and.another.one" => "has a value"
25    })
26end

See OpenTelemetry Ruby's own documentation on adding attributes to spans for more information.



Sample Application

See the sample Ruby on Rails App using OpenTelemetry Ruby SDK Manual Instrumentation.

On this page