How to pass X-Amzn-Trace-Id with the SDK?

[nmoutschen]

Prerequisites

• I've searched for previous similar issues or discussions and didn't find any solution.
• I searched the Developer Guide for an answer to my question.

Question Description

To propagate X-Ray trace information through AWS services that supports it, we need to add an X-Amzn-Trace-Id header to the underlying API call. For example, here's the utility for the Python X-Ray SDK that patches boto3.

How can I manually add this header to the requests made by the AWS SDK for Rust?

Platform/OS/Device

Ubuntu 20.04/Windows 11/macOS Catalina

Language Version

1.57.0

[mbergkvist]

@nmoutschen, did you find any solution to this?

[Chandrian]

Based on what I can see from the go sdk, I'm at the same point before having to add some specific code in the handler. It looks like for GO, you need to use the X-Ray sdk to register the downstream client xray.AWS(s3.Client) and then use svc.ListBucketsWithContext(ctx aws.Context, input *ListBucketsInput) method. Anything equivalent needs to be done in the Rust SDK?

[mbergkvist]

I have about the same findings as you. E.g.

in the lambda environment I have the env var
_X_AMZN_TRACE_ID: Root=1-626634a7-4bb9605739f955ce260994a6;Parent=7a3f99dc6a97a134;Sampled=1

in the request to EventBridge, I can see the header
x-amzn-trace-id: Root=1-626634a7-4bb9605739f955ce260994a6;Parent=7a3f99dc6a97a134;Sampled=1

and in the event payload, I can see
TraceHeader: Root=1-626634a7-4bb9605739f955ce260994a6;Parent=7a3f99dc6a97a134;Sampled=1

but I still can't see the trace to the EvenBridge or to the SQS.

It doesn't look like we need to instrument the clients manually, but I can't say why it isn't working...

[rafalwrzeszcz]

The problem here is that it's also required to send segments to XRay and currently I found no bindings for that in Rust SDK.

[Velfi]

@rafalwrzeszcz I'm not familiar with XRay but does aws_sdk_xray::client::fluent_builders::PutTraceSegments do what you want?

[mbergkvist]

The problem here is that it's also required to send segments to XRay and currently I found no bindings for that in Rust SDK.

Do you need to send segments even when calling services such as S3, DDB, SQS etc are sending segments themselves?

[sarmst22]

@mbergkvist I've looked into this, and based on the AWS X-Ray documentation about SQS and DDB we'll need to send subsegment data about those services ourselves. These are called Inferred Segments from what I read.

I've encountered this same problem when working with the AWS ADO Collector distro for Opentelemetry where I wasn't finding SQS nodes on Servicelens even though I was including the X-Ray Trace Id in the System Message Attribute Value fields for it in the SQS Message. This was because SQS doesn't actually send data to X-Ray, so I have to capture the SDK client response times, queue ARN, and response code to allow it to be inferred in the X-Ray timelines.

[sarmst22]

See my response to my original issue thread here for additional info about my current OTEL progress #194

[Velfi]

We will possibly have a much nicer way of doing this soon

[jnicholls]

For modifying a service request on a request-by-request basis, the above solution is adequate. However, I think a better customer experience is to allow service Client's to be configured with an X-ray Trace ID, such that all requests sent by the Client specify the X-Aws-Trace-Id header. Any consideration to bake X-Ray deeper into the SDK like it is in other language SDKs (Python, Node.js, Java, etc.)?

[jnicholls]

I've lost track of this issue and am curious if the smithy http client or aws-sdk-rust have automatic X-Ray Trace ID propagation baked in now (looks for the env var in a lambda context, and propagates it with every outbound request to AWS services)? I'm not exactly sure where this issue is being tracked at this point, or what the recommended approach is for propagating traces in the Rust SDK. /cc @Velfi

Thanks in advance, and Happy Thanksgiving!

[rcoh]

We will automatically propagate those environment variables in lambda environments: https://github.com/smithy-lang/smithy-rs/blob/main/aws/rust-runtime/aws-runtime/src/recursion_detection.rs#L58-L66

If you wanted to do so generally, you could adapt that interceptor then register it during client creation.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.

[ZoltanLaszlo]

Hi,

I follow the approach for aws-sdk-dynamodb specified here: We will possibly have a much nicer way of doing this soon.
This works fine in most cases, but not for DynamoDB query paginators. As far as I see, I cannot mutate the underlying HTTP requests if I want to use a query paginator.

I try to achieve something like this:

let mut paginator = self.dynamo_db_client.query()
    .table_name(self.table_name.clone())
    .key_condition_expression("partitionKey = :partitionKey")
    .expression_attribute_values(
        ":partitionKey",
        aws_sdk_dynamodb::types::AttributeValue::S(partition_key),
    )
    .into_paginator()
    // .customize()
    // .mutate_request(move |req| {
    //     req.headers_mut().insert(
    //         TRACING_HTTP_HEADER.to_string(),
    //         trace_id.clone(),
    //     );
    // })
    .send();

The .into_paginator() function returns a QueryPaginator struct that cannot be customized this way, while the .customize() method returns a CustomizableOperation struct that does not support query functions such as pagination.

Maybe I'm just trying to do this the wrong way. Can anybody help me out? How can I add a tracing header to the HTTP requests of a DynamoDB query paginator?