I tried setting up correlation between traces and logs in CloudWatch Application Signals
This page has been translated by machine translation. View original
I want to easily check logs related to traces
Hello, this is nonPi (@non____97).
Have you ever wanted to easily check logs related to traces when using CloudWatch Application Signals? I have.
In Application Signals, when you click on a metrics graph, up to 10 spans are displayed from that point in time.
By clicking on the trace ID of these spans, you can check the trace details.
Being able to see logs related to traces allows you to check detailed error messages and stack traces immediately without having to search for logs separately, reducing investigation time.
The implementation method is described in the following AWS official documentation.
You can enable trace for log correlation in Application Signals. This automatically inserts trace IDs and span IDs into related application logs. Then, when you open a trace detail page in the Application Signals console, if there are log entries related to the current trace, they're automatically displayed at the bottom of the page.
For example, you might notice a spike in your latency graph. You can select that point on the graph to load diagnostic information for that time point. Then, you can select a related trace to see details. When you view the trace information, you can scroll down to see logs that are associated with that trace. Examining these logs might help you pinpoint an issue that's causing the latency spike, revealing patterns or error codes.
I tried it out.
Quick summary
- To enable log correlation, implement the following steps:
- Embed trace information in logs
- Be careful whether the package and version you're using is supported by the zero-code instrumentation library
- Specify the log output destination
- Specify using
aws.log.group.namesinOTEL_RESOURCE_ATTRIBUTES
- Specify using
- Embed trace information in logs
- CloudWatch Logs Insights works behind the scenes for log correlation
- It automatically queries log groups starting with
/ecs/
- It automatically queries log groups starting with
Testing it out
Test environment
Here's my test environment.
For logs, I'm using AWS FireLens (AWS for Fluent Bit), with error logs going to CloudWatch Logs and all logs to an S3 bucket via Data Firehose.
This is based on the following article.
All resources are deployed using AWS CDK. The code used is as follows.
Once deployed, the Open Telemetry and Application Signals related environment variables are as follows.
| Key | Value |
|---|---|
| OTEL_AWS_APPLICATION_SIGNALS_ENABLED | false |
| OTEL_EXPORTER_OTLP_ENDPOINT | http://localhost:4318 |
| OTEL_EXPORTER_OTLP_PROTOCOL | http/protobuf |
| OTEL_LOGS_EXPORTER | none |
| OTEL_METRICS_EXPORTER | none |
| OTEL_PROPAGATORS | tracecontext,baggage,xray |
| OTEL_RESOURCE_ATTRIBUTES | service.name=ecs-express-app,deployment.environment=ecs:EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-4JfsTzadcWYq,aws.log.group.names=EcsNativeBlueGreenStack-FirelensConstructFirelensLogGroupD186C82F-Oieo4YPQifeI |
| OTEL_TRACES_EXPORTER | otlp |
| OTEL_TRACES_SAMPLER | xray |
| OTEL_TRACES_SAMPLER_ARG | endpoint=http://localhost:2000,polling_interval=300 |
Embedding trace information in logs
First, we need to embed trace information in logs.
The AWS official documentation recommends using zero-code instrumentation libraries for each logger for Node.js.
Node.js
For details on how to enable trace context injection in Node.js to work with Node.js-compatible logging libraries, see documentation about using NPM for auto-instrumentation of either Pino, Winston, or Bunyan in Node.js.
Since I'm using Pino, I need to use @opentelemetry/instrumentation-pino.
In this case, since I'm using public.ecr.aws/aws-observability/adot-autoinstrumentation-node:v0.8.1 in the init container, I don't need to install @opentelemetry/instrumentation-pino separately.
Checking the releases of aws-observability/aws-otel-js-instrumentation, we can see that this container image includes aws-otel-js-instrumentation:
See ADOT node auto-instrumentation Docker image v0.8.1 in our public ECR repository:
https://gallery.ecr.aws/aws-observability/adot-autoinstrumentation-node
And by checking the package-lock.json of aws-otel-js-instrumentation, we can see that it includes the Pino zero-code instrumentation library version ^0.46.0.
The Pino version is important to note.
For zero-code instrumentation library version 0.46.0, supported Pino versions are >=5.14.0 <10.
pino versions >=5.14.0 <10
The "log sending" feature is only supported in pino v7 and later.
The latest version of Pino is 10.3.1. Therefore, if you don't specify a version, you'll end up using a Pino version that the zero-code instrumentation library doesn't support.
When Pino's zero-code instrumentation is working properly, trace_id and span_id are added as follows:
{
"level": "error",
"time": "2026-02-11T00:42:49.466Z",
"pid": 6,
"hostname": "ip-10-10-8-112.ec2.internal",
"req": {
"id": 36,
"method": "GET",
"url": "/error/",
"query": {},
"params": {},
"headers": {
"host": "ecsnat-albco-qzyosxmqsim4-1634003559.us-east-1.elb.amazonaws.com",
"x-real-ip": "10.10.8.10",
"x-forwarded-for": "<source IP address>, 10.10.8.10",
"x-forwarded-proto": "http",
"connection": "close",
"x-forwarded-port": "80",
"x-amzn-trace-id": "Root=1-698bd089-3e85c9ad79fddc8706f7ad7f",
"cookie": "connect.sid=s%3Aa-rXSzyN-2POI_b5dkH4LkP6ijl68rve.zWuJDO786nBLrcXV3U70DxhwxfrFLqXFjPWwB9ATd%2FQ",
"user-agent": "curl/8.7.1",
"accept": "*/*"
},
"remoteAddress": "127.0.0.1",
"remotePort": 57812
},
"trace_id": "698bd089673affe7e1c8a2add24be6a9",
"span_id": "448e1d51d070a746",
"trace_flags": "01",
"res": {
"statusCode": 500,
"headers": {
"x-powered-by": "Express",
"content-type": "application/json; charset=utf-8",
"content-length": "33",
"etag": "W/\"21-Fau8GdrOCOyGNNH/IiTxy2DuMu0\""
}
},
"err": {
"type": "Error",
"message": "failed with status code 500",
"stack": "Error: failed with status code 500\n at onResFinished (/app/node_modules/.pnpm/pino-http@10.5.0/node_modules/pino-http/logger.js:115:39)\n at ServerResponse.onResponseComplete (/app/node_modules/.pnpm/pino-http@10.5.0/node_modules/pino-http/logger.js:178:14)\n at /otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55\n at AsyncLocalStorage.run (node:internal/async_local_storage/async_context_frame:59:14)\n at AsyncLocalStorageContextManager.with (/otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n at ServerResponse.contextWrapper (/otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)\n at ServerResponse.emit (node:events:520:35)\n at onFinish (node:_http_outgoing:1026:10)\n at callback (node:internal/streams/writable:764:21)\n at afterWrite (node:internal/streams/writable:708:5)"
},
"responseTime": 5,
"msg": "request errored",
"container_name": "app",
"source": "stderr",
"container_id": "6b8cc0c5e5f94ebe888e690dbb051168-0527074092",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/6b8cc0c5e5f94ebe888e690dbb051168",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:95"
}
On the other hand, if Pino version is 10 or higher, trace information is not recorded:
{
"level": "error",
"time": "2026-02-04T07:54:00.739Z",
"pid": 6,
"hostname": "ip-10-10-8-103.ec2.internal",
"req": {
"id": 63,
"method": "GET",
"url": "/error/",
"query": {},
"params": {},
"headers": {
"host": "ecsnat-albco-fgdpbvnooxwj-1973419334.us-east-1.elb.amazonaws.com",
"x-real-ip": "10.10.8.57",
"x-forwarded-for": "<source IP address>, 10.10.8.57",
"x-forwarded-proto": "http",
"connection": "close",
"x-forwarded-port": "80",
"x-amzn-trace-id": "Root=1-6982fb18-3fb552d8541fd9151d48406f",
"accept": "*/*",
"user-agent": "curl/8.7.1"
},
"remoteAddress": "127.0.0.1",
"remotePort": 53378
},
"res": {
"statusCode": 500,
"headers": {
"x-powered-by": "Express",
"content-type": "application/json; charset=utf-8",
"content-length": "33",
"etag": "W/\"21-Fau8GdrOCOyGNNH/IiTxy2DuMu0\"",
"set-cookie": [
"connect.sid=s%3An3YhhvIDKp6XWpVpDbx0U9F0GeNEHJxg.1HNNiMD4%2BhQEmjNdBO9NbQsADN67pKUccIeBYe%2Fp%2FVc; Path=/; Expires=Thu, 05 Feb 2026 07:54:00 GMT; HttpOnly"
]
}
},
"err": {
"type": "Error",
"message": "failed with status code 500",
"stack": "Error: failed with status code 500\n at onResFinished (/app/node_modules/.pnpm/pino-http@11.0.0/node_modules/pino-http/logger.js:115:39)\n at ServerResponse.onResponseComplete (/app/node_modules/.pnpm/pino-http@11.0.0/node_modules/pino-http/logger.js:178:14)\n at /otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55\n at AsyncLocalStorage.run (node:internal/async_local_storage/async_context_frame:59:14)\n at AsyncLocalStorageContextManager.with (/otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n at ServerResponse.contextWrapper (/otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)\n at ServerResponse.emit (node:events:520:35)\n at onFinish (node:_http_outgoing:1026:10)\n at callback (node:internal/streams/writable:764:21)\n at afterWrite (node:internal/streams/writable:708:5)"
},
"responseTime": 3,
"msg": "request errored",
"container_name": "app",
"source": "stderr",
"container_id": "0dfd495fff0048de92bd87f68daa8de1-0527074092",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-s5tytpIiJ9VQ",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-s5tytpIiJ9VQ/0dfd495fff0048de92bd87f68daa8de1",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:93"
}
Be careful with Express and other packages as well, making sure they're at versions supported by zero-code instrumentation. The documentation and source code contain supported versions, like this:
Specifying the log output destination
To correlate traces and logs, you need to specify the log output destination.
Specifically, add aws.log.group.names=<target log group name> to OTEL_RESOURCE_ATTRIBUTES.
(Optional) To enable Application Signals log correlation, additionally set the environment variable aws.log.group.names to the log group name of your application logs. This enables correlation of traces and metrics generated by your application with corresponding log entries in this log group. In this variable, replace $YOUR_APPLICATION_LOG_GROUP with the name of your application log group. If you have multiple log groups, you can use an ampersand (&) to separate them as in this example: aws.log.group.names=log-group-1&log-group-2. Setting this current environment variable is sufficient to enable log correlation for metrics. For more information, see Enable log correlation for metrics. To enable log correlation for traces, you also need to change your application's logging configuration. For more information, see Enable trace for log correlation.
This will include the specified log group name in the cloudwatch_logs property of the span's aws property.
{
"ecs.task.arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/015048a8181a46e4bbc15539e33568f7",
"ecs.task.family": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2",
"ecs.cluster.arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:cluster/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g",
"ecs": {
"container": "ip-10-10-8-74.ec2.internal",
"container_id": "46e4bbc15539e33568f7/015048a8181a46e4bbc15539e33568f7-0527074092"
},
"ecs.container.arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:container/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/015048a8181a46e4bbc15539e33568f7/8059f724-8aeb-4554-afd9-bddf9f15b80e",
"ecs.launchtype": "fargate",
"span.kind": "CLIENT",
"cloudwatch_logs": [
{
"log_group": "EcsNativeBlueGreenStack-FirelensConstructFirelensLogGroupD186C82F-wTQ4890Mp8e9"
}
],
"ecs.task.revision": "100"
},
Note that log correlation won't work if you include resourcedetection with ECS Detectors in the trace pipeline of the Open Telemetry Collector as shown below:
processors:
.
.
(omitted)
.
.
resourcedetection:
detectors: [env, ecs]
timeout: 5s
override: false
.
.
(omitted)
.
.
service:
extensions: [sigv4auth, awsproxy]
pipelines:
traces:
receivers: [otlp]
processors: [filter/exclude_health, resourcedetection, batch]
exporters: [otlphttp]
Specifically, you'll see an error message like "An error occurred while retrieving data":
Note: The same result appears whether from "CloudWatch > Transaction Search" or "CloudWatch > Traces"
As mentioned later, CloudWatch Logs Insights runs behind the scenes for log correlation, but in this case, there was no evidence that Logs Insights was executed.
By the way, in this case, the aws property of the span contained information about the log group and log stream of the container with the awslogs log driver in cloudwatch_logs, log.stream.arns, and log.stream.names:
{
"ecs.task.arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/2a9e9b028b6b4bd0b4edf85023bd9c3b",
"ecs.task.family": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2",
"ecs.cluster.arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:cluster/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g",
"ecs": {
"container": "ip-10-10-8-125.ec2.internal",
"container_id": "4bd0b4edf85023bd9c3b/2a9e9b028b6b4bd0b4edf85023bd9c3b-0527074092"
},
"ecs.container.arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:container/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/2a9e9b028b6b4bd0b4edf85023bd9c3b/e8984edf-97c0-4e46-a8ea-23018b9537c6",
"ecs.launchtype": "fargate",
"span.kind": "CLIENT",
"cloudwatch_logs": [
{
"log_group": "EcsNativeBlueGreenStack-FirelensConstructFirelensLogGroupD186C82F-wTQ4890Mp8e9"
},
{
"log_group": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:EcsNativeBlueGreenStack-EcsConstructTaskDefinitionAdotAutoInstrumentationInitContainerLogGroup8060A51D-TTDVP1HlaLC8"
},
{
"log_group": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:EcsNativeBlueGreenStack-EcsConstructTaskDefinitionlogRouterLogGroup27EC9B3C-OxmB1VpOf8SS"
}
],
"ecs.task.revision": "100",
"log.stream.arns": [
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:EcsNativeBlueGreenStack-EcsConstructTaskDefinitionAdotAutoInstrumentationInitContainerLogGroup8060A51D-TTDVP1HlaLC8:log-stream:init/init/2a9e9b028b6b4bd0b4edf85023bd9c3b",
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:EcsNativeBlueGreenStack-EcsConstructTaskDefinitionlogRouterLogGroup27EC9B3C-OxmB1VpOf8SS:log-stream:firelens/EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2/logRouter/2a9e9b028b6b4bd0b4edf85023bd9c3b"
],
"log.stream.names": [
"init/init/2a9e9b028b6b4bd0b4edf85023bd9c3b",
"firelens/EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2/logRouter/2a9e9b028b6b4bd0b4edf85023bd9c3b"
],
"ecs.task.id": "2a9e9b028b6b4bd0b4edf85023bd9c3b"
},
Since the only other addition is ecs.task.id, avoid inserting this in your pipeline.
Verification of Operation
I will verify the operation.
From the Application Signals service, I click on a data point in the metrics graph to display the correlation span.
I click on the trace ID within the correlation span.
As a result, I can also see logs associated with the trace as follows:
{
"level": "error",
"time": "2026-02-11T05:39:13.085Z",
"pid": 8,
"hostname": "ip-10-10-8-89.ec2.internal",
"req": {
"id": 479,
"method": "GET",
"url": "/crash/",
"query": {},
"params": {},
"headers": {
"host": "ecsnat-albco-qzyosxmqsim4-1634003559.us-east-1.elb.amazonaws.com",
"x-real-ip": "10.10.8.40",
"x-forwarded-for": "<source IP address>, 10.10.8.40",
"x-forwarded-proto": "http",
"connection": "close",
"x-forwarded-port": "80",
"x-amzn-trace-id": "Root=1-698c1601-5ec60d115f6377e539ba7666",
"cookie": "connect.sid=s%3AkdwCTdPnOMbapxgvwkYpR3T2Y1Ykw_eW.cef%2Bw5X7jVYHTqDmz0m8XtFUePmZOE225Vy9JSE%2B4uQ",
"user-agent": "curl/8.7.1",
"accept": "*/*"
},
"remoteAddress": "127.0.0.1",
"remotePort": 33256
},
"trace_id": "698c160176b2820f3142596543b58e08",
"span_id": "84537a0cf5432a39",
"trace_flags": "01",
"msg": "Crash endpoint triggered - throwing exception",
"container_id": "26510d3c39984a7cb58a1bf43dda452b-0527074092",
"container_name": "app",
"source": "stderr",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/26510d3c39984a7cb58a1bf43dda452b",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:98"
}
{
"level": "error",
"time": "2026-02-11T05:39:13.086Z",
"pid": 8,
"hostname": "ip-10-10-8-89.ec2.internal",
"req": {
"id": 479,
"method": "GET",
"url": "/crash/",
"query": {},
"params": {},
"headers": {
"host": "ecsnat-albco-qzyosxmqsim4-1634003559.us-east-1.elb.amazonaws.com",
"x-real-ip": "10.10.8.40",
"x-forwarded-for": "<source IP address>, 10.10.8.40",
"x-forwarded-proto": "http",
"connection": "close",
"x-forwarded-port": "80",
"x-amzn-trace-id": "Root=1-698c1601-5ec60d115f6377e539ba7666",
"cookie": "connect.sid=s%3AkdwCTdPnOMbapxgvwkYpR3T2Y1Ykw_eW.cef%2Bw5X7jVYHTqDmz0m8XtFUePmZOE225Vy9JSE%2B4uQ",
"user-agent": "curl/8.7.1",
"accept": "*/*"
},
"remoteAddress": "127.0.0.1",
"remotePort": 33256
},
"trace_id": "698c160176b2820f3142596543b58e08",
"span_id": "84537a0cf5432a39",
"trace_flags": "01",
"err": {
"type": "Error",
"message": "Application crash test - intentional exception",
"stack": "Error: Application crash test - intentional exception\n at /app/dist/router.js:218:11\n at Layer.handle [as handle_request] (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/layer.js:95:5)\n at next (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/route.js:149:13)\n at Route.dispatch (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/route.js:119:3)\n at patched (/otel-auto-instrumentation/node_modules/@opentelemetry/instrumentation-express/build/src/instrumentation.js:210:37)\n at Layer.handle [as handle_request] (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/layer.js:95:5)\n at /app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/index.js:284:15\n at router.process_params (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/index.js:346:12)\n at next (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/index.js:280:10)\n at router.handle (/app/node_modules/.pnpm/express@4.22.1/node_modules/express/lib/router/index.js:175:3)"
},
"msg": "Error occurred",
"container_id": "26510d3c39984a7cb58a1bf43dda452b-0527074092",
"container_name": "app",
"source": "stderr",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/26510d3c39984a7cb58a1bf43dda452b",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:98"
}
{
"level": "error",
"time": "2026-02-11T05:39:13.088Z",
"pid": 8,
"hostname": "ip-10-10-8-89.ec2.internal",
"req": {
"id": 479,
"method": "GET",
"url": "/crash/",
"query": {},
"params": {},
"headers": {
"host": "ecsnat-albco-qzyosxmqsim4-1634003559.us-east-1.elb.amazonaws.com",
"x-real-ip": "10.10.8.40",
"x-forwarded-for": "<source IP address>, 10.10.8.40",
"x-forwarded-proto": "http",
"connection": "close",
"x-forwarded-port": "80",
"x-amzn-trace-id": "Root=1-698c1601-5ec60d115f6377e539ba7666",
"cookie": "connect.sid=s%3AkdwCTdPnOMbapxgvwkYpR3T2Y1Ykw_eW.cef%2Bw5X7jVYHTqDmz0m8XtFUePmZOE225Vy9JSE%2B4uQ",
"user-agent": "curl/8.7.1",
"accept": "*/*"
},
"remoteAddress": "127.0.0.1",
"remotePort": 33256
},
"trace_id": "698c160176b2820f3142596543b58e08",
"span_id": "84537a0cf5432a39",
"trace_flags": "01",
"res": {
"statusCode": 500,
"headers": {
"x-powered-by": "Express",
"content-type": "application/json; charset=utf-8",
"content-length": "33",
"etag": "W/\"21-Fau8GdrOCOyGNNH/IiTxy2DuMu0\""
}
},
"err": {
"type": "Error",
"message": "failed with status code 500",
"stack": "Error: failed with status code 500\n at onResFinished (/app/node_modules/.pnpm/pino-http@10.5.0/node_modules/pino-http/logger.js:115:39)\n at ServerResponse.onResponseComplete (/app/node_modules/.pnpm/pino-http@10.5.0/node_modules/pino-http/logger.js:178:14)\n at /otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55\n at AsyncLocalStorage.run (node:internal/async_local_storage/async_context_frame:59:14)\n at AsyncLocalStorageContextManager.with (/otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n at ServerResponse.contextWrapper (/otel-auto-instrumentation/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)\n at ServerResponse.emit (node:events:520:35)\n at onFinish (node:_http_outgoing:1026:10)\n at callback (node:internal/streams/writable:764:21)\n at afterWrite (node:internal/streams/writable:708:5)"
},
"responseTime": 5,
"msg": "request errored",
"container_id": "26510d3c39984a7cb58a1bf43dda452b-0527074092",
"container_name": "app",
"source": "stderr",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-zJnVnD9x7R6g/26510d3c39984a7cb58a1bf43dda452b",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:98"
}
I can also see when dependency availability is low.
The logs at this time are as follows:
{
"level": "error",
"time": "2026-02-12T09:08:27.268Z",
"pid": 7,
"hostname": "ip-10-10-8-82.ec2.internal",
"req": {
"id": 15,
"method": "GET",
"url": "/health",
"query": {},
"params": {},
"headers": {
"host": "localhost:3000",
"user-agent": "Wget",
"accept": "*/*",
"connection": "close"
},
"remoteAddress": "127.0.0.1",
"remotePort": 57732
},
"trace_id": "698d988b0f2d820050ca5552c933f32d",
"span_id": "f9ff425ef6931901",
"trace_flags": "01",
"err": {
"type": "ReplyError",
"message": "ERR syntax error",
"stack": "ReplyError: ERR syntax error\n at parseError (/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:179:12)\n at parseType (/app/node_modules/.pnpm/redis-parser@3.0.0/node_modules/redis-parser/lib/parser.js:302:14)",
"command": {
"name": "set",
"args": [
"sess:st2zrsEzLyFlzb5i4XocReBSUhmyjgRz",
"{\"cookie\":{\"originalMaxAge\":86400000,\"expires\":\"2026-02-13T09:08:27.265Z\",\"secure\":false,\"httpOnly\":true,\"path\":\"/\"}}",
"[object Object]"
]
}
},
"msg": "Error occurred",
"container_id": "3d5c2659c6ad4d0587a43191759bd918-0527074092",
"container_name": "app",
"source": "stderr",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-4JfsTzadcWYq",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS Account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-4JfsTzadcWYq/3d5c2659c6ad4d0587a43191759bd918",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:103"
}
When I check the Logs Insights execution history at the time of clicking on the trace ID to view trace details, the following queries were executed:
fields @message, @log, traceId, spanId | filter traceId = "698c160176b2820f3142596543b58e08" | sort @timestamp desc | limit 10000
fields @message, @log, traceId, spanId, jsonparse(@message) as json_message | unnest json_message.links into link | filter link.traceId = "698c160176b2820f3142596543b58e08" | sort @timestamp desc | limit 10000
fields @message, @log, traceId, spanId | filter traceId = "698c160176b2820f3142596543b58e08" | sort @timestamp desc | limit 10000
fields @message, @log, traceId, spanId, jsonparse(@message) as json_message | unnest json_message.links into link | filter link.traceId = "698c160176b2820f3142596543b58e08" | sort @timestamp desc | limit 10000
fields @log, @timestamp, @message
| filter @message like "698c160176b2820f3142596543b58e08" or @message like "8c1601762820f3142596543b58e08"
| sort @timestamp, @message desc
fields @log, @timestamp, @message
| filter @message like "698c160176b2820f3142596543b58e08" or @message like "8c1601762820f3142596543b58e08"
| sort @timestamp, @message desc
Essentially, three types of queries are being executed twice each.
Judging from the log group names in the queries, the log correlation is likely happening in the last two queries mentioned above.
The CloudTrail event at this time is as follows:
{
"eventVersion": "1.11",
"userIdentity": {
"type": "AssumedRole",
"principalId": "<Principal ID>",
"arn": "arn:aws:sts::<AWS Account ID>:assumed-role/<IAM Role Name>/<Session Name>",
"accountId": "<AWS Account ID>",
"accessKeyId": "<Access Key>",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "<Principal ID>",
"arn": "arn:aws:iam::<AWS Account ID>:role/<IAM Role Name>",
"accountId": "<AWS Account ID>",
"userName": "<IAM User Name>"
},
"attributes": {
"creationDate": "2026-02-11T05:38:53Z",
"mfaAuthenticated": "true"
}
}
},
"eventTime": "2026-02-11T05:45:59Z",
"eventSource": "logs.amazonaws.com",
"eventName": "StartQuery",
"awsRegion": "us-east-1",
"sourceIPAddress": "<Source IP Address>",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36",
"requestParameters": {
"logGroupNames": [
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/",
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/default-nginx-5d29",
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/ecs-aws-firelens-sidecar-container",
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/ecs-aws-otel-sidecar-collector",
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/nginx",
"arn:aws:logs:us-east-1:<AWS Account ID>:log-group:EcsNativeBlueGreenStack-FirelensConstructFirelensLogGroupD186C82F-wTQ4890Mp8e9"
],
"startTime": 1770788338,
"endTime": 1770788383,
"queryString": "fields @log, @timestamp, @message\n| filter @message like \"698c160176b2820f3142596543b58e08\" or @message like \"8c1601762820f3142596543b58e08\"\n| sort @timestamp, @message desc",
"dryRun": false
},
"responseElements": null,
"additionalEventData": {
"queryId": "5f79c759-30fa-45e5-adff-e68542d98ab0"
},
"requestID": "3f4c9697-9f6f-4213-891d-3f72959a2916",
"eventID": "cc9165a8-ecda-4992-9498-dbe332ebce96",
"readOnly": true,
"resources": [
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/default-nginx-5d29"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/ecs-aws-firelens-sidecar-container"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/ecs-aws-otel-sidecar-collector"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/nginx"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:EcsNativeBlueGreenStack-FirelensConstructFirelensLogGroupD186C82F-wTQ4890Mp8e9"
}
],
"eventType": "AwsApiCall",
"apiVersion": "20140328",
"managementEvent": true,
"recipientAccountId": "<AWS Account ID>",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.3",
"cipherSuite": "TLS_AES_128_GCM_SHA256",
"clientProvidedHostHeader": "logs.us-east-1.amazonaws.com"
},
"sessionCredentialFromConsole": "true"
}
From "startTime": 1770788338 and "endTime": 1770788383, we can see that it's searching for logs within a 45-second period across the specified log groups.
What's concerning is that log groups starting with /ecs/ such as /ecs/ and /ecs/default-nginx-5d29 are also being included as query targets. I didn't specify these conditions.
Looking at CloudTrail, when log correlation was performed, it was searching for log groups starting with /ecs/:
{
"eventVersion": "1.11",
"userIdentity": {
"type": "AssumedRole",
"principalId": "<Principal ID>",
"arn": "arn:aws:sts::<AWS Account ID>:assumed-role/<IAM Role Name>/<Session Name>",
"accountId": "<AWS Account ID>",
"accessKeyId": "<Access Key>",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "<Principal ID>",
"arn": "arn:aws:iam::<AWS Account ID>:role/<IAM Role Name>",
"accountId": "<AWS Account ID>",
"userName": "<IAM User Name>"
},
"attributes": {
"creationDate": "2026-02-11T05:38:53Z",
"mfaAuthenticated": "true"
}
}
},
"eventTime": "2026-02-11T06:02:54Z",
"eventSource": "logs.amazonaws.com",
"eventName": "DescribeLogGroups",
"awsRegion": "us-east-1",
"sourceIPAddress": "<Source IP Address>",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36",
"requestParameters": {
"logGroupNamePrefix": "/ecs/",
"includeLinkedAccounts": true
},
"responseElements": null,
"requestID": "c59cb761-d8cf-4c71-aa7e-24e7ca4a9cb7",
"eventID": "46c30084-8d1d-4484-a167-0b4821b413fc",
"readOnly": true,
"resources": [
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/default-nginx-5d29"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/ecs-aws-firelens-sidecar-container"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/ecs-aws-otel-sidecar-collector"
},
{
"accountId": "<AWS Account ID>",
"type": "AWS::Logs::LogGroup",
"ARN": "arn:aws:logs:us-east-1:<AWS Account ID>:log-group:/ecs/nginx"
}
],
"eventType": "AwsApiCall",
"apiVersion": "20140328",
"managementEvent": true,
"recipientAccountId": "<AWS Account ID>",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.3",
"cipherSuite": "TLS_AES_128_GCM_SHA256",
"clientProvidedHostHeader": "logs.us-east-1.amazonaws.com"
},
"sessionCredentialFromConsole": "true"
}
Personally, I'd like to disable this feature.
I'm concerned about Logs Insights billing when there's a large volume of logs flowing through log groups starting with /ecs/. Since it's using filter @message like "<trace ID>" to search the entire log message body, field indexing won't be effective.
If anyone knows how to opt out of including log groups starting with /ecs/ in log correlation, I'd appreciate the information.
Personally, I'm considering avoiding creating log groups that start with /ecs/ as a workaround.
Log Correlation is Easy to Set Up
I tried trace and log correlation with CloudWatch Application Signals.
It was easier than I expected because you just need to use a supported logger and specify the log group name with an environment variable.
By the way, since I'm only sending error logs to CloudWatch Logs this time, log correlation isn't possible when the HTTP status code 200 is returned.
All logs are output to an S3 bucket via Data Firehose. If you want to find logs with HTTP status code 200, search using the trace ID. Logs are recorded as follows:
{
"level": "info",
"time": "2026-02-12T01:48:36.868Z",
"pid": 7,
"hostname": "ip-10-10-8-104.ec2.internal",
"req": {
"id": 55,
"method": "GET",
"url": "/",
"query": {},
"params": {},
"headers": {
"host": "ecsnat-albco-shaasnih0xar-1652870936.us-east-1.elb.amazonaws.com",
"x-real-ip": "10.10.8.43",
"x-forwarded-for": "<source IP address>, 10.10.8.43",
"x-forwarded-proto": "http",
"connection": "close",
"x-forwarded-port": "80",
"x-amzn-trace-id": "Root=1-698d3174-5876cff8075b183978fe1d65",
"accept": "*/*",
"user-agent": "curl/8.7.1"
},
"remoteAddress": "127.0.0.1",
"remotePort": 35448
},
"trace_id": "698d3174fa6d5a5d85e08988b96c738d",
"span_id": "6f704133bb94ada4",
"trace_flags": "01",
"res": {
"statusCode": 200,
"headers": {
"x-powered-by": "Express",
"content-type": "application/json; charset=utf-8",
"content-length": "427",
"etag": "W/\"1ab-kBv3WE96jXCcAIdDtz3e81LLMik\"",
"set-cookie": [
"connect.sid=s%3A_ykD64hKoji4n7SB71zDtgljHN5UDJBx.7y5uIGbxZsGgM4knIOw1Pt7BAOhIo%2FA6NLGKotG6hoU; Path=/; Expires=Fri, 13 Feb 2026 01:48:36 GMT; HttpOnly"
]
}
},
"responseTime": 292,
"msg": "request completed",
"container_id": "c9979e7d4b08471b8285c6b3ecfd5cfb-0527074092",
"container_name": "app",
"source": "stdout",
"ecs_cluster": "EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-VoUD00RfbaSW",
"ecs_task_arn": "arn:aws:ecs:us-east-1:<AWS account ID>:task/EcsNativeBlueGreenStack-EcsConstructCluster14AE103B-VoUD00RfbaSW/c9979e7d4b08471b8285c6b3ecfd5cfb",
"ecs_task_definition": "EcsNativeBlueGreenStackEcsConstructTaskDefinitionF683F4B2:102",
"datetime": "2026-02-12T01:48:36.869Z"
}
I hope this article helps someone.
That's all from NonPi (@non____97) of the Consulting Department, Cloud Business Division!