CloudWatch Agent Now Supports journald, So I Tried It Out on Amazon Linux 2023
This page has been translated by machine translation. View original
Introduction
On August 28, 2026, the CloudWatch agent added support for collecting logs from the systemd journal (journald).
| Item | Before | Now |
|---|---|---|
| How journald logs are collected | Additional configuration to export journald to files was required | The agent reads journald directly |
| Offloading to files for the CloudWatch agent | The journal had to be written out to files | Not required |
| Structured metadata | Could be lost when going through files | Sent with unit name, priority, and process information preserved |
| Filtering | None | Unit, priority, field match, and regular expression filters |
The standard Amazon Linux 2023 configuration has no /var/log/messages and does not include rsyslog. Because the logs emitted by the OS are gathered in journald, I configured the agent with only the journald section, without using files, which specifies files. I collected four kinds of logs (systemd units, kernel messages, audit events, and Docker) into separate log groups. I verified the format of the events that arrived in CloudWatch Logs and even queries in Logs Insights.
What I Verified
Test Environment
| Item | Value |
|---|---|
| OS | Amazon Linux 2023.12.20260817 |
| Kernel | 6.18.41-94.142.amzn2023.x86_64 |
| systemd | 252 |
| Instance | t3.micro |
| Region | ap-northeast-1 |
| CloudWatch agent | 1.300072.0b1766 |
I did not open any inbound ports, and all operations were performed through SSM. The subnet is a public subnet, the instance was assigned a public IP, and outbound traffic is allowed. Connectivity to SSM, S3, the Amazon Linux repositories, and CloudWatch Logs is required.
journald support is a feature of agent version 1.300070.0 and later. As of August 29, 2026, the version available from the amazonlinux repository on Amazon Linux 2023 was 1.300069.1. Therefore I switched the source to the latest RPM on S3 (https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm).
Least Privilege and Deployment
I created the EC2 instance and IAM role using CloudFormation Express mode. In addition to the managed policy for SSM, the instance role limits CloudWatch Logs permissions to just four actions. Because the configuration file specifies retention_in_days, I also included logs:PutRetentionPolicy. The retention period is set to 1 day for this test.
The instance role part of the template is as follows.
InstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Policies:
- PolicyName: journald-logs-minimal
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- logs:PutRetentionPolicy
Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:journald-*
aws cloudformation create-stack \
--stack-name journald-demo \
--template-body file://journald-demo.yaml \
--capabilities CAPABILITY_IAM \
--deployment-config Mode=EXPRESS \
--parameters ParameterKey=VpcId,ParameterValue=vpc-xxxxxxxxxxxxxxxxx \
ParameterKey=SubnetId,ParameterValue=subnet-xxxxxxxxxxxxxxxxx
journald Configuration and Delivery Format
There are four entries to collect, each sent to a separate log group.
{
"agent": {
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"journald": {
"collect_list": [
{
"log_group_name": "journald-units",
"log_stream_name": "{instance_id}",
"units": ["sshd", "systemd-logind", "chronyd", "amazon-ssm-agent"],
"retention_in_days": 1
},
{
"log_group_name": "journald-kernel",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "kernel"}],
"retention_in_days": 1
},
{
"log_group_name": "journald-audit",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "audit"}],
"priority": "debug",
"retention_in_days": 1
},
{
"log_group_name": "journald-docker",
"log_stream_name": "{instance_id}",
"units": ["docker"],
"retention_in_days": 1
}
]
}
}
}
}
I applied it with the following command.
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json
I retrieved one event of the systemd unit logs that arrived in CloudWatch Logs.
{
"events": [
{
"timestamp": 1787982970748,
"message": "{\"body\":{\"MESSAGE\":\"pam_unix(sudo:session): session closed for user root\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"10\",\"SYSLOG_IDENTIFIER\":\"sudo\",\"_CMDLINE\":\"sudo docker run --rm public.ecr.aws/docker/library/alpine:3.20 true\",\"_COMM\":\"sudo\",\"_EXE\":\"/usr/bin/sudo\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"29152\",\"_SYSTEMD_UNIT\":\"amazon-ssm-agent.service\",\"_TRANSPORT\":\"syslog\",\"_UID\":\"0\",\"__CURSOR\":\"s=xxxxxxxx;i=e09;b=xxxxxxxx;m=2155d6a2;t=65a2939d5ff89;x=xxxxxxxx\"}}",
"ingestionTime": 1787982974133
}
]
}
Each event arrives as structured JSON with all of the journald fields placed under body. It is not plain text containing only the MESSAGE body. The above is an excerpt with some fields omitted (the same applies to the event JSON shown below).
I also checked the body of a successful sshd login event.
{
"body": {
"MESSAGE": "Accepted publickey for ec2-user from 127.0.0.1 port 54478 ssh2: ED25519 SHA256:xxxxxxxx",
"PRIORITY": "6",
"SYSLOG_FACILITY": "10",
"SYSLOG_IDENTIFIER": "sshd-session",
"SYSLOG_PID": "29125",
"_COMM": "sshd-session",
"_EXE": "/usr/libexec/openssh/sshd-session",
"_HOSTNAME": "ip-10-0-x-x.ap-northeast-1.compute.internal",
"_PID": "29125",
"_SELINUX_CONTEXT": "system_u:system_r:sshd_t:s0-s0:c0.c1023",
"_SYSTEMD_CGROUP": "/system.slice/sshd.service",
"_SYSTEMD_UNIT": "sshd.service",
"_TRANSPORT": "syslog",
"_UID": "0"
}
}
What I specified in the configuration was sshd, but SYSLOG_IDENTIFIER was sshd-session and _SYSTEMD_UNIT was sshd.service. Specification by unit works on the unit name, not the identifier.
The source being 127.0.0.1 is because no sshd logs were produced in the Session Manager shell session I used this time. I generated logs by running ssh ec2-user@localhost with a throwaway key from inside the instance.
Note that fetch-config deletes etc/amazon-cloudwatch-agent.json when applied and also replaces existing files under amazon-cloudwatch-agent.d. Therefore, be aware that the procedure shown here overwrites any existing agent configuration.
journalctl that the agent launches internally
journalctl --utc --output=json --follow --priority debug _TRANSPORT=audit
journalctl --utc --output=json --follow --priority info _TRANSPORT=kernel --after-cursor s=xxxxxxxx;i=9be;...
journalctl --utc --output=json --follow --unit docker --priority info --after-cursor s=xxxxxxxx;i=9b2;...
journalctl --utc --output=json --follow --unit sshd --unit systemd-logind --unit chronyd --unit amazon-ssm-agent --priority info --after-cursor s=xxxxxxxx;i=cb2;...
One entry in collect_list corresponds to one journalctl process. The unit names, field specifications, and priority written in the configuration became command-line arguments as they were.
--after-cursor is attached to collect_list entries that already have a cursor, so past logs are not collected again.
Kernel and Audit Logs
I collected kernel messages by specifying _TRANSPORT as kernel in matches.
{
"events": [
{
"timestamp": 1787982734525,
"message": "{\"body\":{\"MESSAGE\":\"veth97da191 (unregistering): left allmulticast mode\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"0\",\"SYSLOG_IDENTIFIER\":\"kernel\",\"_BOOT_ID\":\"xxxxxxxx\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_TRANSPORT\":\"kernel\",\"__CURSOR\":\"s=xxxxxxxx;i=9bc;b=xxxxxxxx;m=13415a86;t=65a292bc1836e;x=xxxxxxxx\"}}",
"ingestionTime": 1787982739614
}
]
}
Kernel messages have no unit name, so they cannot be specified with units. journalctl --unit kernel over the 20 minutes during which logs were collected in the test returned "-- No entries --". On the other hand, journalctl _TRANSPORT=kernel returned 639 lines.
On Amazon Linux 2023, auditd and systemd-journald-audit.socket are both active by default, so audit events also reach journald. During the 20 minutes of log collection, 911 entries were in scope. However, of those 911, 0 had a PRIORITY field. The documentation states that the default value of priority is info (debug is excluded). Since the agent launches with journalctl --priority info, audit entries are excluded from collection by default.
With the default priority, audit events could not be collected, and the journald-audit log group itself was not even created. The other three log groups were created. When I specified debug for priority, all 911 entries were in scope, and the agent also created the log group.
{
"events": [
{
"timestamp": 1787983006400,
"message": "{\"body\":{\"AUDIT_FIELD_EXE\":\"/usr/lib/systemd/systemd\",\"AUDIT_FIELD_RES\":\"success\",\"AUDIT_FIELD_UNIT\":\"user@0\",\"MESSAGE\":\"SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user@0 comm=\\\"systemd\\\" exe=\\\"/usr/lib/systemd/systemd\\\" hostname=? addr=? terminal=? res=success'\",\"SYSLOG_FACILITY\":\"4\",\"SYSLOG_IDENTIFIER\":\"audit\",\"_AUDIT_ID\":\"1433\",\"_AUDIT_TYPE\":\"1131\",\"_AUDIT_TYPE_NAME\":\"SERVICE_STOP\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"1\",\"_TRANSPORT\":\"audit\",\"_UID\":\"0\"}}",
"ingestionTime": 1787983009138
}
]
}
Audit events arrived with _AUDIT_TYPE_NAME and AUDIT_FIELD_* preserved.
Docker Logs
As an example of an application managed by systemd, I separated Docker logs into a dedicated log group.
{
"events": [
{
"timestamp": 1787982718503,
"message": "{\"body\":{\"MESSAGE\":\"time=\\\"2026-08-29T05:51:58.502767587Z\\\" level=info msg=\\\"API listen on /run/docker.sock\\\"\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"3\",\"SYSLOG_IDENTIFIER\":\"dockerd\",\"_CMDLINE\":\"/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536\",\"_COMM\":\"dockerd\",\"_EXE\":\"/usr/bin/dockerd\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"7416\",\"_SYSTEMD_UNIT\":\"docker.service\",\"_TRANSPORT\":\"stdout\",\"_UID\":\"0\"}}",
"ingestionTime": 1787982724803
}
]
}
When docker was specified as the unit, dockerd logs were collected. The same log group also contains docker.service-related messages emitted by systemd itself (entries whose CODE_FILE is src/core/job.c). The scope is the same as journalctl -u docker.
Logs Insights Queries
Because the delivery format is JSON, the fields under body became discovered fields as they were, and no parse was needed. All of the following queries target the last hour. First, in the systemd unit log group, count by unit (recordsMatched 111).
stats count(*) as cnt by body._SYSTEMD_UNIT | sort cnt desc
| body._SYSTEMD_UNIT | cnt |
|---|---|
| amazon-ssm-agent.service | 95 |
| sshd.service | 10 |
| systemd-logind.service | 6 |
Next, across the four log groups, break down by priority (logGroupsScanned 4, recordsMatched 366).
stats count(*) as cnt by @log, body.PRIORITY | sort @log, body.PRIORITY
| @log | body.PRIORITY | cnt |
|---|---|---|
| 123456789012:journald-audit | (empty) | 186 |
| 123456789012:journald-docker | 6 | 2 |
| 123456789012:journald-kernel | 5 | 5 |
| 123456789012:journald-kernel | 6 | 62 |
| 123456789012:journald-units | 5 | 34 |
| 123456789012:journald-units | 6 | 77 |
Count audit events by type (recordsMatched 176).
stats count(*) as cnt by body._AUDIT_TYPE_NAME | sort cnt desc | limit 8
| body._AUDIT_TYPE_NAME | cnt |
|---|---|
| BPF | 74 |
| USER_START | 12 |
| USER_ACCT | 11 |
| CRYPTO_KEY_USER | 10 |
| CRED_DISP | 10 |
| SERVICE_STOP | 9 |
| USER_END | 9 |
| SERVICE_START | 7 |
Extract only sshd connections by invalid users (recordsMatched 2).
filter body._SYSTEMD_UNIT = "sshd.service" and body.MESSAGE like /Invalid user/
| fields @timestamp, body.MESSAGE | sort @timestamp desc
| @timestamp | body.MESSAGE |
|---|---|
| 2026-08-29 05:56:09.986 | Invalid user nosuchuser from 127.0.0.1 port 54494 |
| 2026-08-29 05:52:10.066 | Invalid user nosuchuser from 127.0.0.1 port 47170 |
Summary
With the CloudWatch agent's journald support, I was able to collect systemd unit, kernel, audit, and Docker logs on Amazon Linux 2023 into CloudWatch Logs. No configuration to write the journal out to files was needed. Each event is sent as structured JSON that preserves the journald fields. Using unit names and audit event types, you can aggregate and search directly from Logs Insights.
Setups that add rsyslog solely to feed the CloudWatch agent, or that offload the journal to files and tail them, can be simplified with the journald section. In particular, if you want to collect logs of the OS and of services managed by systemd all together, consider migrating.
Appendix: UserData
This is UserData that builds the same configuration without post-configuring via SSM. Everything from installing the agent to applying the journald configuration completes at boot.
#!/bin/bash
set -xeuo pipefail
# systemd 管理下のアプリ例として Docker を導入
dnf install -y docker
systemctl enable --now docker
# AL2023 の dnf リポジトリ版は journald 非対応バージョンのため S3 の最新 RPM を使う
cd /tmp
curl -sS -O https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
dnf install -y ./amazon-cloudwatch-agent.rpm
cat > /opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json <<'CONFIG'
{
"agent": { "run_as_user": "root" },
"logs": {
"logs_collected": {
"journald": {
"collect_list": [
{
"log_group_name": "journald-userdata-units",
"log_stream_name": "{instance_id}",
"units": ["sshd", "systemd-logind", "chronyd", "amazon-ssm-agent"],
"retention_in_days": 1
},
{
"log_group_name": "journald-userdata-kernel",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "kernel"}],
"retention_in_days": 1
},
{
"log_group_name": "journald-userdata-audit",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "audit"}],
"priority": "debug",
"retention_in_days": 1
},
{
"log_group_name": "journald-userdata-docker",
"log_stream_name": "{instance_id}",
"units": ["docker"],
"retention_in_days": 1
}
]
}
}
}
}
CONFIG
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config -m ec2 -s \
-c file:/opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json
