
Trying Out OpenTelemetry Support for AWS Lambda Durable Functions
This page has been translated by machine translation. View original
Introduction
On November 19, 2026, OpenTelemetry support for AWS Lambda durable functions was announced.
With the OpenTelemetry plugin, the entire durable execution, including steps, waits, callbacks, and chained invokes, is output as a single trace. In this article, I use the Python SDK to check the structure of the spans that appear in aws/spans in CloudWatch Logs. I also compare the execution view and the invocation view.
Test Environment
- Region: Tokyo (ap-northeast-1)
- Runtime: Python 3.13 (x86_64)
- aws-durable-execution-sdk-python: 3.0.2
- aws-durable-execution-sdk-python-otel: 2.1.0
- ADOT layer: AWSOpenTelemetryDistroPython 30
- Tracing: Active tracing
- Transaction Search: Enabled (from before the test)
- Destination of X-Ray trace segments: CloudWatchLogs
- X-Ray indexing rule: Default (1% sampling)
- AWS CLI: 2.41.2
- SAM CLI: 1.174.0
Plugin 2.1.0 requires >=3.0.0,<4 of the core SDK. When I did not pin the core SDK version, pip installed 3.1.0. In this article, I pin it to 3.0.2.
Configuration
The specification of the OpenTelemetry plugin is in the following documentation.
Dependency Packages
In requirements.txt, list the plugin alongside the core SDK.
aws-durable-execution-sdk-python==3.0.2
aws-durable-execution-sdk-python-otel==2.1.0
Template
For a configuration that outputs to X-Ray via ADOT, the documentation shows enabling Active tracing and attaching AWSXRayDaemonWriteAccess to the execution role. The template.yaml is as follows.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Lambda durable functions OpenTelemetry plugin verification
Globals:
Function:
Runtime: python3.13
Architectures:
- x86_64
MemorySize: 512
Timeout: 60
Tracing: Active
Layers:
- arn:aws:lambda:ap-northeast-1:615299751070:layer:AWSOpenTelemetryDistroPython:30
Environment:
Variables:
AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
Resources:
ChildFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: otel-demo-child
CodeUri: src/
Handler: child.handler
AutoPublishAlias: live
DurableConfig:
ExecutionTimeout: 900
RetentionPeriodInDays: 1
Policies:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
MainFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: otel-demo-main
CodeUri: src/
Handler: main.handler
AutoPublishAlias: live
DurableConfig:
ExecutionTimeout: 900
RetentionPeriodInDays: 1
Environment:
Variables:
CHILD_FUNCTION: !Ref ChildFunction.Alias
Policies:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicDurableExecutionRolePolicy
- arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess
- LambdaInvokePolicy:
FunctionName: !Ref ChildFunction
The ADOT layer version differs by region. The example in the AWS Compute Blog uses version 33 in us-east-1, while Tokyo is 30. You can check the ARN for your own region in the following list. The 615299751070 in the ARN is the account ID of the provider of the public ADOT layer, not the user's account ID.
The AWS Compute Blog article is below.
After sam build, deploy with the following command.
sam deploy --stack-name otel-demo --region ap-northeast-1 --resolve-s3 --capabilities CAPABILITY_IAM
Function Code
The parent function's main.py lines up step, retry, wait, callback, and chained invoke in a single flow. The plugin is registered to the handler with plugins=[create_otel_plugin_factory()]. The span name of each operation uses the value passed to name= in the code. The name of the per-attempt span (Attempt span) gets a suffix such as attempt 1.
import json
import os
from aws_durable_execution_sdk_python import (
DurableContext,
durable_execution,
durable_step,
)
from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python.types import WaitForCallbackContext
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory
@durable_step
def extract_text(step_context, doc_id):
step_context.logger.info(f"Extracting text from document: {doc_id}")
return {"doc_id": doc_id, "text": "extracted content"}
@durable_step
def summarize(step_context, extracted):
# Fail only the first attempt to check how a retry appears as an Attempt span
step_context.logger.info(f"Summarizing document (attempt {step_context.attempt})")
if step_context.attempt == 1:
raise RuntimeError("temporary failure on first attempt")
return {"summary": "Document summary here"}
@durable_step
def publish(step_context, summary, approval, child_result):
step_context.logger.info(f"Publishing with approval from: {approval.get('reviewer')}")
return {"status": "published", "child": child_result}
@durable_execution(plugins=[create_otel_plugin_factory()])
def handler(event: dict, context: DurableContext) -> dict:
extracted = context.step(extract_text(event["doc_id"]), name="extract-text")
summary = context.step(summarize(extracted), name="summarize")
context.wait(Duration.from_seconds(10), name="cooldown")
def submit_for_review(callback_id: str, ctx: WaitForCallbackContext) -> None:
ctx.logger.info(f"CALLBACK_ID={callback_id}")
approval = context.wait_for_callback(submitter=submit_for_review, name="human-review")
if isinstance(approval, str):
approval = json.loads(approval)
child_result = context.invoke(
os.environ["CHILD_FUNCTION"], {"doc_id": event["doc_id"]}, name="call-child"
)
return context.step(publish(summary, approval, child_result), name="publish")
I registered the plugin in the same way in child.py, another durable function called with context.invoke.
from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory
@durable_execution(plugins=[create_otel_plugin_factory()])
def handler(event: dict, context: DurableContext) -> dict:
return context.step(lambda ctx: {"child": "done", "doc_id": event["doc_id"]}, name="child-work")
Retrieving the Trace
Execution
Because this function uses wait_for_callback, I start it asynchronously. The DurableExecutionArn in the response is the key used later to look up the execution's trace ID.
aws lambda invoke --function-name otel-demo-main --qualifier live --invocation-type Event --region ap-northeast-1 --cli-binary-format raw-in-base64-out --payload '{"doc_id": "test-doc-001"}' /tmp/otel-resp.json
{
"StatusCode": 202,
"DurableExecutionArn": "arn:aws:lambda:ap-northeast-1:123456789012:function:otel-demo-main:1/durable-execution/<実行名>/<実行ID>"
}
The execution waits until it receives the callback for human-review. The callback ID appears in the log message in the form CALLBACK_ID=<value> about 18 seconds after the start (after the step retry wait and the 10-second wait).
aws logs filter-log-events --log-group-name /aws/lambda/otel-demo-main --region ap-northeast-1 --filter-pattern CALLBACK_ID --start-time $(( ($(date +%s) - 600) * 1000 )) --query "events[].message" --output text
Using the obtained callback ID, send the result before the execution timeout (900 seconds in this test). The callback ID is a Base64 string containing /. Do not split at the position of /; pass the whole thing as is.
aws lambda send-durable-execution-callback-success --cli-binary-format raw-in-base64-out --callback-id <callback-id> --region ap-northeast-1 --result '{"approved": true, "reviewer": "alice"}'
Retrieval from aws/spans
The execution's TraceHeader can be obtained with get-durable-execution.
aws lambda get-durable-execution --durable-execution-arn <DurableExecutionArn> --region ap-northeast-1 --output json
"TraceHeader": {
"XAmznTraceId": "Root=1-<8桁>-<24桁>;Parent=<親ID>;Sampled=1"
}
Remove the leading 1- from the Root value, and the 32-digit value formed by joining the remaining 8 digits and 24 digits becomes the traceId in aws/spans. In this test as well, the two matched.
Spans can be extracted with filter-log-events.
aws logs filter-log-events --log-group-name aws/spans --region ap-northeast-1 \
--filter-pattern '"<traceId>"' --start-time <開始のエポックミリ秒> --output json
In Logs Insights, I lined up the span type, attempt, outcome, and duration (in milliseconds) with the following query. It is narrowed to spans whose execution ARN contains otel-demo-main.
aws logs start-query --log-group-name aws/spans --region ap-northeast-1 \
--start-time <開始のエポック秒> --end-time <終了のエポック秒> \
--query-string "fields name, attributes.durable.operation.type as type, attributes.durable.attempt.number as attempt, attributes.durable.attempt.outcome as outcome, durationNano/1000000 as ms | filter traceId = '<traceId>' and attributes.durable.execution.arn like /otel-demo-main/ | sort startTimeUnixNano asc | limit 50" \
--output json
Pass the queryId returned by start-query to get-query-results. The results are complete once the status becomes Complete.
aws logs get-query-results --query-id <queryId> --region ap-northeast-1 --output json
The Retrieved Trace
Scope Included in a Single Trace
The retrieved spans include, besides the durable spans, spans such as otel-demo-main/LambdaService that Lambda emits with Active tracing, and the main.handler span that ADOT emits. To make parent-child relationships easier to follow, I prepared a script that displays spans as a tree. With --durable-only, the spans emitted by Lambda and ADOT are excluded and only durable spans are displayed. Each line shows, in order: span name, operation type, attempt and outcome, status, start time from the first span of the trace (+), duration, and function name.
span_tree.py
# 使い方: python3 span_tree.py spans.json --durable-only
import json
import sys
from collections import defaultdict
path = sys.argv[1]
durable_only = "--durable-only" in sys.argv
text = open(path).read()
events = json.loads(text[text.index("{"): text.rindex("}") + 1])["events"]
spans = {}
for e in events:
m = json.loads(e["message"])
key = m["spanId"]
# 同じ spanId が部分的に2回出力されるものは、所要時間が長いほうを採用する
if key not in spans or m.get("durationNano", 0) > spans[key].get("durationNano", 0):
spans[key] = m
t0 = min(m["startTimeUnixNano"] for m in spans.values())
children = defaultdict(list)
roots = []
for m in spans.values():
p = m.get("parentSpanId")
if p in spans:
children[p].append(m)
else:
roots.append(m)
def label(m):
a = m.get("attributes", {})
parts = [m["name"]]
if "durable.operation.type" in a:
parts.append(f"[{a['durable.operation.type']}]")
if "durable.attempt.number" in a:
parts.append(f"attempt={a['durable.attempt.number']} outcome={a.get('durable.attempt.outcome')}")
if "durable.operation.status" in a:
parts.append(f"status={a['durable.operation.status']}")
if "durable.execution.status" in a:
parts.append(f"execution.status={a['durable.execution.status']}")
if "durable.invocation.status" in a:
parts.append(f"invocation.status={a['durable.invocation.status']}")
if "durable.invocation.first" in a:
parts.append(f"first={a['durable.invocation.first']}")
if m.get("status", {}).get("code") == "ERROR":
parts.append("ERROR")
return " ".join(parts)
def is_durable(m):
return any(k.startswith("durable.") for k in m.get("attributes", {})) or m["name"] in ("Workflow", "Invocation")
def show(m, depth):
start = (m["startTimeUnixNano"] - t0) / 1e9
dur = m.get("durationNano", 0) / 1e9
if not durable_only or is_durable(m):
print(f"{' ' * depth}{label(m)} +{start:.3f}s {dur:.3f}s ({m.get('resource', {}).get('attributes', {}).get('service.name', '-')})")
depth += 1
for c in sorted(children[m["spanId"]], key=lambda x: x["startTimeUnixNano"]):
show(c, depth)
for r in sorted(roots, key=lambda x: x["startTimeUnixNano"]):
show(r, 0)
print()
print(f"spans={len(spans)} trace_ids={sorted({m['traceId'] for m in spans.values()})}")
Save the output of filter-log-events to spans.json and run it.
python3 span_tree.py spans.json --durable-only
Invocation invocation.status=PENDING first=True +2.648s 0.271s (otel-demo-main)
Invocation invocation.status=PENDING first=False +7.883s 0.214s (otel-demo-main)
Invocation invocation.status=PENDING first=False +18.050s 0.187s (otel-demo-main)
Invocation invocation.status=PENDING first=False +153.129s 0.275s (otel-demo-main)
Workflow execution.status=SUCCEEDED +0.089s 156.202s (otel-demo-main)
extract-text [STEP] attempt=1 outcome=None status=SUCCEEDED +2.649s 0.096s (otel-demo-main)
extract-text attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +2.649s 0.001s (otel-demo-main)
summarize [STEP] attempt=2 outcome=None status=SUCCEEDED +2.806s 5.125s (otel-demo-main)
summarize attempt 1 [STEP] attempt=1 outcome=FAILED ERROR +2.771s 0.001s (otel-demo-main)
summarize attempt 2 [STEP] attempt=2 outcome=SUCCEEDED +7.884s 0.000s (otel-demo-main)
cooldown [WAIT] status=SUCCEEDED +7.984s 10.000s (otel-demo-main)
human-review [CONTEXT] status=SUCCEEDED +18.075s 135.080s (otel-demo-main)
human-review create callback id [CALLBACK] status=SUCCEEDED +18.075s 134.970s (otel-demo-main)
human-review submitter [STEP] attempt=1 outcome=None status=SUCCEEDED +18.089s 0.028s (otel-demo-main)
human-review submitter attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +18.089s 0.001s (otel-demo-main)
call-child [CHAINED_INVOKE] status=SUCCEEDED +153.266s 2.810s (otel-demo-main)
Workflow execution.status=SUCCEEDED +153.304s 2.751s (otel-demo-child)
child-work [STEP] attempt=1 outcome=None status=SUCCEEDED +155.825s 0.110s (otel-demo-child)
child-work attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +155.825s 0.000s (otel-demo-child)
Invocation invocation.status=SUCCEEDED first=True +155.824s 0.230s (otel-demo-child)
publish [STEP] attempt=1 outcome=None status=SUCCEEDED +156.148s 0.021s (otel-demo-main)
publish attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +156.148s 0.000s (otel-demo-main)
Invocation invocation.status=SUCCEEDED first=False +156.147s 0.144s (otel-demo-main)
A single run of the parent function was split into 5 Lambda invocations, and the spans from all 5 were included in one trace ID. Invocations are split at four points: the step retry wait, the 10-second wait, the callback wait, and the wait for the chained invoke to complete. The Workflow, Invocation, and step spans of the child function child.py are also included in the same trace ID, and appear under the parent's call-child span.
In the default execution view, the operation spans are children of the Workflow. Invocation spans are not children of the Workflow; they appear under the main.handler span that ADOT emits. The Workflow appears under otel-demo-main/LambdaExecutionEnvironment of the last invocation. Each operation span has a link to the span of the invocation that exported that operation.
You can also trace from logs to the trace. The JSON logs contain traceId, spanId, and otelTraceSampled, and the traceId matched the trace ID in aws/spans.
Retry and Duration
The first attempt of summarize fails with an exception in the code. The Attempt span summarize attempt 1 has durable.attempt.outcome=FAILED, and its OTel status is ERROR. The exception event of this span contains the exception type and message.
The summarize step span in this execution is 5.125 seconds, covering the interval from right after the first failure to the second success. Because no retry strategy was specified for the step, the retry uses the default behavior. In this execution, the wait before the retry was about 5 seconds. The duration of the whole step can be read from the step span, and the length of each individual attempt can be read from the Attempt spans.
Both the first invocation, which ended by waiting for the step retry, and the invocations that ended by waiting for the wait, callback, and chained invoke had durable.invocation.status of PENDING and an OTel status of OK.
Registration via Environment Variable and Views
Registration Settings
Instead of writing plugins= in the code, the plugin can also be registered with the environment variable DURABLE_EXECUTION_PLUGINS. A value of otel gives the execution view, and otel-invocation gives the invocation view.
To compare the views, I added two functions: EnvExecFunction (otel-demo-env-exec) and EnvInvFunction (otel-demo-env-inv). Both use the same CodeUri: src/ and Globals as the parent function. They differ from the parent function in three points: the function name, the environment variable, and Handler being minimal.handler.
Set the following environment variable on EnvExecFunction.
Environment:
Variables:
DURABLE_EXECUTION_PLUGINS: otel
For EnvInvFunction, set the value to otel-invocation.
Environment:
Variables:
DURABLE_EXECUTION_PLUGINS: otel-invocation
The handler is minimal.py, with only @durable_execution attached.
from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python.config import Duration
# Do not write the plugin in the code. Register it with the environment variable DURABLE_EXECUTION_PLUGINS.
@durable_execution
def handler(event: dict, context: DurableContext) -> str:
data = context.step(lambda ctx: event["id"], name="fetch-data")
context.wait(Duration.from_seconds(5), name="cooldown")
return context.step(lambda ctx: f"processed {data}", name="process")
For both functions, pass JSON containing id as the payload. The following is an example of invoking EnvExecFunction.
aws lambda invoke --function-name otel-demo-env-exec --qualifier live --invocation-type Event --region ap-northeast-1 --cli-binary-format raw-in-base64-out --payload '{"id": "abc"}' /tmp/otel-resp-exec.json
Trace by View
I ran the same minimal.py in the two views and compared the span trees. The execution view (otel) is as follows.
Invocation invocation.status=PENDING first=True +2.744s 0.238s (otel-demo-env-exec)
Workflow execution.status=SUCCEEDED +0.082s 8.001s (otel-demo-env-exec)
fetch-data [STEP] attempt=1 outcome=None status=SUCCEEDED +2.745s 0.090s (otel-demo-env-exec)
fetch-data attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +2.745s 0.000s (otel-demo-env-exec)
cooldown [WAIT] status=SUCCEEDED +2.871s 5.000s (otel-demo-env-exec)
process [STEP] attempt=1 outcome=None status=SUCCEEDED +7.929s 0.036s (otel-demo-env-exec)
process attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +7.929s 0.000s (otel-demo-env-exec)
Invocation invocation.status=SUCCEEDED first=False +7.928s 0.155s (otel-demo-env-exec)
The invocation view (otel-invocation) is as follows.
Invocation invocation.status=PENDING first=True +2.609s 0.276s (otel-demo-env-inv)
fetch-data [STEP] attempt=1 outcome=None status=SUCCEEDED +2.610s 0.103s (otel-demo-env-inv)
fetch-data attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +2.610s 0.000s (otel-demo-env-inv)
cooldown [WAIT] status=STARTED +2.739s 0.146s (otel-demo-env-inv)
Workflow execution.status=SUCCEEDED +0.078s 7.959s (otel-demo-env-inv)
Invocation invocation.status=SUCCEEDED first=False +7.840s 0.196s (otel-demo-env-inv)
cooldown [WAIT] status=SUCCEEDED +7.841s 0.000s (otel-demo-env-inv)
process [STEP] attempt=1 outcome=None status=SUCCEEDED +7.841s 0.074s (otel-demo-env-inv)
process attempt 1 [STEP] attempt=1 outcome=SUCCEEDED +7.841s 0.000s (otel-demo-env-inv)
The cooldown span is a single 5-second span in the execution view. In the invocation view, it is split into two: a STARTED span under the first Invocation span, and a SUCCEEDED span under the second Invocation span. Also, in the invocation view, the parent of an operation span is not the Workflow but the Invocation that executed the operation.
Guidance on which view to choose is in the "Choosing a view" section of the documentation.
Summary
Using the OpenTelemetry plugin for Lambda durable functions, an execution split across multiple invocations was consolidated under a single trace ID.
Because step durations and retries can be read from this trace, it may help in identifying bottlenecks and the causes of failures. I recommend evaluating this update.
