Manual Instrumentation for Traces and Metrics with the AWS Distro for OpenTelemetry .NET SDK

The AWS Distro for OpenTelemetry .NET SDK contains an extension library for instrumenting the AWS SDK. In this tutorial, we will introduce how to manually instrument your application for traces and metrics step-by-step using AWS Distro for OpenTelemetry .NET SDK.

Requirements

The AWS Distro for OpenTelemetry .NET SDK is compatible for all the officially supported versions of .NET and .NET Framework.

Note: You’ll also need to have the AWS Distro for OpenTelemetry Collector running to export traces and metrics.

Installation

In order to instrument your .NET application for traces and metrics, start by downloading the OpenTelemetry nuget package to your application.

Copy
dotnet add package OpenTelemetry

The OpenTelemetry SDK for .NET deals with concerns such as sampling, a processing pipeline, and exporting telemetry to a particular backend. This generally requires additional packages to be downloaded for specific instrumentation or exporter.

The OpenTelemetry SDK generates traces with W3C random ID which X-Ray backend doesn’t currently support. You need to install the OpenTelemetry.Contrib.Extensions.AWSXRay to be able to use the AWSXRayIdGenerator which generates X-Ray compatible trace IDs. If you plan to call another application instrumented with AWS X-Ray SDK, you’ll need to configure the AWSXRayPropagator as well.

Copy
dotnet add package OpenTelemetry.Contrib.Extensions.AWSXRay

In order to export traces and metics from your application to ADOT Collector, you need to install OpenTelemetry.Exporter.OpenTelemetryProtocol.

Copy
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

By default the OpenTelemetry exporter sends data to an OpenTelemetry collector at localhost:4317.

Setting up the Global Tracer and Meter

Sending Traces and Metrics

Configure AWS X-Ray ID generator, propagator and OpenTelemetry Protocol (OTLP) exporter globally in your application as follows. Make sure to call AddXRayTraceId() in the very beginning when creating TracerProvider Also configure the meter provider and add a meter of your choice as well as the OpenTelemetry Protocol (OTLP) exporter.

Copy
using OpenTelemetry;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Trace;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
                        .AddSource("ActivitySourceName")
                        .AddXRayTraceId() // for generating AWS X-Ray compliant trace IDs
                        .AddOtlpExporter() // default address localhost:4317
                        .Build();

var meterProvider = Sdk.CreateMeterProviderBuilder()
                        .AddMeter("example_meter")
                        .AddOtlpExporter()
                        .Build();

Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator()); // configure AWS X-Ray propagator

Using the AWS resource Detectors

The ADOT .NET SDK supports automatically recording metadata in EC2, Elastic Beanstalk, ECS, and EKS environments. You can configure the corresponding resource detector to the TracerProvider following the EC2 example below.

Copy
using OpenTelemetry;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Resources;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
                        // other configurations
                        .SetResourceBuilder(ResourceBuilder
                            .CreateDefault()
                            .AddDetector(new AWSEC2ResourceDetector()))
                        .Build();

Instrumenting .NET Applications

Once you have configured all necessary X-Ray components to the TracerProvider, you can proceed to OpenTelemetry .NET SDK's developer guide to instrument your .NET application.

OpenTelemetry provides a wide range of instrumentations for popular .NET libraries: Asp.Net, Asp.Net Core, Http, Grpc, Redis, Sql and EntityFramework. Instrumenting a library means that every time the library is used to make or handle a request, that library's calls are automatically wrapped within a span. That span is automatically populated with attributes describing the values used by the library call.

AWS SDK Instrumentation

For tracing downstream calls to AWS services from your .NET application, you will need to install the OpenTelemetry.Contrib.Instrumentation.AWS package.

Copy
dotnet add package OpenTelemetry.Contrib.Instrumentation.AWS

Call AddAWSInstrumentation() to add AWS SDK client instrumentation to TracerProvider. The below example is for an ASP.NET Core application.

Copy
using OpenTelemetry;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Trace;
using OpenTelemetry.Trace;

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
  
    tracerProvider.AddAWSInstrumentation() // for tracing calls to AWS services via AWS SDK for .NET
                  .AddAspNetCoreInstrumentation()
                  .AddOtlpExporter()
                  .Build();
}

Custom Instrumentation

Creating Custom Spans

In .NET, you can use the activity API to create custom spans to monitor the performance of internal activities that are not captured by instrumentation libraries.

Copy
using System.Diagnostics;

ActivitySource activitySource = new ActivitySource("ActivitySourceName", "ActivitySourceVersion");
var activity = activitySource.StartActivity("ActivityName", ActivityKind.Server); // this will be translated to a X-Ray Segment
var internalActivity = activitySource.StartActivity("ActivityName", ActivityKind.Internal); // this will be translated to an X-Ray Subsegment

Note that only spans of kind Server are converted into X-Ray segments, all other kind of spans (Internal, Client, etc.) are converted into X-Ray subsegments. For more on segments and subsegments, see the AWS X-Ray developer guide.

Adding Custom Attributes

You can also add custom key-value pairs as attributes onto your spans.

Copy
activity.SetTag("http.method", "GET");
activity.SetTag("http.url", "http://www.mywebsite.com");

Attributes are converted to metadata by default. If you configure your collector, you can convert some or all of the attributes to annotations. To read more about X-Ray annotations and metadata see the AWS X-Ray developer guide.

For more information about the activity API, see the OpenTelemetry .NET SDK's developer guide.

Creating Metrics

Similarly to Traces, you can create custom metrics in your application using the OpenTelemetry API and SDK.

In the following example application we demonstrate how to use metric instruments to record metrics with a Counter.

Copy
using System.Threading.Tasks;

Meter meter = new Meter("example_meter", "1.0"); 

totalTimeSentObserver = meter.CreateCounter<int>("time_alive",
                "ms",
                "Measures the total time the application has been alive");

while (true) {
    var delayTask = Task.Delay(1000);
    await Task.Run(() => totalTimeSentObserver.Add(1, new KeyValuePair<string, object>("attribute", "sample")));
    await delayTask;
}

Sample Application

Take a reference to the sample application that is instrumented by ADOT and OpenTelemetry .NET SDK.