I tried out CloudWatch Agent's journald support on Amazon Linux 2023
This page has been translated by machine translation. View original
Introduction
On August 28, 2026, the CloudWatch Agent added support for collecting logs from the systemd journal (journald).
| Item | Before | Now |
|---|---|---|
| How to collect journald logs | Additional configuration required to export journald to a file | Agent reads journald directly |
| File offloading for CloudWatch Agent | Required to write journal to a file | Not required |
| Structured metadata | May be lost when going through a file | Sent with unit name, priority, and process information preserved |
| Filtering | None | Unit, priority, field match, and regex filters |
The default configuration of Amazon Linux 2023 does not include /var/log/messages and does not have rsyslog installed. Since logs generated by the OS are collected by journald, the agent configuration was set up using only the journald section rather than specifying files with files. Four types of logs — systemd units, kernel messages, audit events, and Docker — were collected into separate log groups. The format of events delivered to CloudWatch Logs and querying with Logs Insights were also verified.
Verification Details
Verification Environment
| Item | Value |
|---|---|
| OS | Amazon Linux 2023.12.20260817 |
| Kernel | 6.18.41-94.142.amzn2023.x86_64 |
| systemd | 252 |
| Instance | t3.micro |
| Region | ap-northeast-1 |
| CloudWatch Agent | 1.300072.0b1766 |
No inbound ports were opened, and all operations were performed via SSM. The subnet is a public subnet, the instance was assigned a public IP, and outbound traffic is allowed. Communication with SSM, S3, the Amazon Linux repository, and CloudWatch Logs is required.
journald support is a feature available from agent version 1.300070.0 onwards. As of August 29, 2026, the version available from the Amazon Linux 2023 amazonlinux repository was 1.300069.1. Therefore, the source was switched to the latest RPM on S3 (https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm).
Least Privilege and Deployment
The EC2 instance and IAM role were created using CloudFormation in Express mode. The instance role, in addition to the managed policy for SSM, restricts CloudWatch Logs permissions to only 4 actions. Since retention_in_days is specified in the configuration file, logs:PutRetentionPolicy is also included. The retention period is set to 1 day for verification purposes.
The instance role portion of the template is as follows.
InstanceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Policies:
- PolicyName: journald-logs-minimal
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- logs:PutRetentionPolicy
Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:journald-*
aws cloudformation create-stack \
--stack-name journald-demo \
--template-body file://journald-demo.yaml \
--capabilities CAPABILITY_IAM \
--deployment-config Mode=EXPRESS \
--parameters ParameterKey=VpcId,ParameterValue=vpc-xxxxxxxxxxxxxxxxx \
ParameterKey=SubnetId,ParameterValue=subnet-xxxxxxxxxxxxxxxxx
journald Configuration and Delivery Format
The collection targets consist of the following 4 entries, each sent to a separate log group.
{
"agent": {
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"journald": {
"collect_list": [
{
"log_group_name": "journald-units",
"log_stream_name": "{instance_id}",
"units": ["sshd", "systemd-logind", "chronyd", "amazon-ssm-agent"],
"retention_in_days": 1
},
{
"log_group_name": "journald-kernel",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "kernel"}],
"retention_in_days": 1
},
{
"log_group_name": "journald-audit",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "audit"}],
"priority": "debug",
"retention_in_days": 1
},
{
"log_group_name": "journald-docker",
"log_stream_name": "{instance_id}",
"units": ["docker"],
"retention_in_days": 1
}
]
}
}
}
}
The configuration was applied with the following command.
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json
One event of systemd unit logs delivered to CloudWatch Logs was retrieved.
{
"events": [
{
"timestamp": 1787982970748,
"message": "{\"body\":{\"MESSAGE\":\"pam_unix(sudo:session): session closed for user root\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"10\",\"SYSLOG_IDENTIFIER\":\"sudo\",\"_CMDLINE\":\"sudo docker run --rm public.ecr.aws/docker/library/alpine:3.20 true\",\"_COMM\":\"sudo\",\"_EXE\":\"/usr/bin/sudo\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"29152\",\"_SYSTEMD_UNIT\":\"amazon-ssm-agent.service\",\"_TRANSPORT\":\"syslog\",\"_UID\":\"0\",\"__CURSOR\":\"s=xxxxxxxx;i=e09;b=xxxxxxxx;m=2155d6a2;t=65a2939d5ff89;x=xxxxxxxx\"}}",
"ingestionTime": 1787982974133
}
]
}
Each event is delivered as structured JSON with all journald fields contained under body. It is not plain text with only the MESSAGE content. The above is an excerpt with some fields omitted (the same applies to the event JSONs shown below).
The body of an sshd successful login event was also confirmed.
{
"body": {
"MESSAGE": "Accepted publickey for ec2-user from 127.0.0.1 port 54478 ssh2: ED25519 SHA256:xxxxxxxx",
"PRIORITY": "6",
"SYSLOG_FACILITY": "10",
"SYSLOG_IDENTIFIER": "sshd-session",
"SYSLOG_PID": "29125",
"_COMM": "sshd-session",
"_EXE": "/usr/libexec/openssh/sshd-session",
"_HOSTNAME": "ip-10-0-x-x.ap-northeast-1.compute.internal",
"_PID": "29125",
"_SELINUX_CONTEXT": "system_u:system_r:sshd_t:s0-s0:c0.c1023",
"_SYSTEMD_CGROUP": "/system.slice/sshd.service",
"_SYSTEMD_UNIT": "sshd.service",
"_TRANSPORT": "syslog",
"_UID": "0"
}
}
sshd was specified in the configuration, but the SYSLOG_IDENTIFIER was sshd-session and _SYSTEMD_UNIT was sshd.service. Unit-level specification works by unit name, not by identifier.
The reason the source address is 127.0.0.1 is that no sshd logs were generated in the Session Manager shell session used this time. To generate logs, ssh ec2-user@localhost was executed from within the instance using a disposable key.
Note that fetch-config deletes etc/amazon-cloudwatch-agent.json when applied and also replaces existing files under amazon-cloudwatch-agent.d. Therefore, be aware that the procedure shown here will overwrite existing agent configurations.
journalctl launched internally by the agent
journalctl --utc --output=json --follow --priority debug _TRANSPORT=audit
journalctl --utc --output=json --follow --priority info _TRANSPORT=kernel --after-cursor s=xxxxxxxx;i=9be;...
journalctl --utc --output=json --follow --unit docker --priority info --after-cursor s=xxxxxxxx;i=9b2;...
journalctl --utc --output=json --follow --unit sshd --unit systemd-logind --unit chronyd --unit amazon-ssm-agent --priority info --after-cursor s=xxxxxxxx;i=cb2;...
One entry in collect_list corresponds to one journalctl process. The unit names, field specifications, and priorities written in the configuration were directly reflected as command-line arguments.
For collect_list entries that already have a cursor, --after-cursor is added, so past logs are not re-collected.
Kernel and Audit Logs
Kernel messages were collected by specifying _TRANSPORT as kernel in matches.
{
"events": [
{
"timestamp": 1787982734525,
"message": "{\"body\":{\"MESSAGE\":\"veth97da191 (unregistering): left allmulticast mode\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"0\",\"SYSLOG_IDENTIFIER\":\"kernel\",\"_BOOT_ID\":\"xxxxxxxx\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_TRANSPORT\":\"kernel\",\"__CURSOR\":\"s=xxxxxxxx;i=9bc;b=xxxxxxxx;m=13415a86;t=65a292bc1836e;x=xxxxxxxx\"}}",
"ingestionTime": 1787982739614
}
]
}
Since kernel messages do not have a unit name attached, they cannot be specified with units. Running journalctl --unit kernel for the 20-minute verification period returned "-- No entries --". On the other hand, journalctl _TRANSPORT=kernel returned 639 lines.
On Amazon Linux 2023, both auditd and systemd-journald-audit.socket are active by default, so audit events also reach journald. During the 20-minute log collection period, 911 entries were targeted. However, of these 911 entries, none had a PRIORITY field. The documentation states that the default value of priority is info (excluding debug). Since the agent starts with journalctl --priority info, audit entries are excluded from collection by default.
With the default priority, audit events could not be collected, and the journald-audit log group itself was not created. The other three log groups were created. When debug was specified as the priority, all 911 entries became targets and the agent also created the log group.
{
"events": [
{
"timestamp": 1787983006400,
"message": "{\"body\":{\"AUDIT_FIELD_EXE\":\"/usr/lib/systemd/systemd\",\"AUDIT_FIELD_RES\":\"success\",\"AUDIT_FIELD_UNIT\":\"user@0\",\"MESSAGE\":\"SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user@0 comm=\\\"systemd\\\" exe=\\\"/usr/lib/systemd/systemd\\\" hostname=? addr=? terminal=? res=success'\",\"SYSLOG_FACILITY\":\"4\",\"SYSLOG_IDENTIFIER\":\"audit\",\"_AUDIT_ID\":\"1433\",\"_AUDIT_TYPE\":\"1131\",\"_AUDIT_TYPE_NAME\":\"SERVICE_STOP\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"1\",\"_TRANSPORT\":\"audit\",\"_UID\":\"0\"}}",
"ingestionTime": 1787983009138
}
]
}
Audit events were delivered with _AUDIT_TYPE_NAME and AUDIT_FIELD_* preserved.
Docker Logs
As an example of an application managed under systemd, Docker logs were separated into a dedicated log group.
{
"events": [
{
"timestamp": 1787982718503,
"message": "{\"body\":{\"MESSAGE\":\"time=\\\"2026-08-29T05:51:58.502767587Z\\\" level=info msg=\\\"API listen on /run/docker.sock\\\"\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"3\",\"SYSLOG_IDENTIFIER\":\"dockerd\",\"_CMDLINE\":\"/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536\",\"_COMM\":\"dockerd\",\"_EXE\":\"/usr/bin/dockerd\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"7416\",\"_SYSTEMD_UNIT\":\"docker.service\",\"_TRANSPORT\":\"stdout\",\"_UID\":\"0\"}}",
"ingestionTime": 1787982724803
}
]
}
By specifying docker as the unit, dockerd logs were collected. The same log group also includes messages related to docker.service emitted by systemd itself (entries where CODE_FILE is src/core/job.c). The same scope as journalctl -u docker is targeted.
Logs Insights Queries
Since the delivery format is JSON, fields under body become discovered fields as-is, and no parsing is required. All of the following queries target the last 1 hour. First, counting by unit in the systemd units log group (recordsMatched 111).
stats count(*) as cnt by body._SYSTEMD_UNIT | sort cnt desc
| body._SYSTEMD_UNIT | cnt |
|---|---|
| amazon-ssm-agent.service | 95 |
| sshd.service | 10 |
| systemd-logind.service | 6 |
Next, breaking down by priority across all 4 log groups (logGroupsScanned 4, recordsMatched 366).
stats count(*) as cnt by @log, body.PRIORITY | sort @log, body.PRIORITY
| @log | body.PRIORITY | cnt |
|---|---|---|
| 123456789012:journald-audit | (empty) | 186 |
| 123456789012:journald-docker | 6 | 2 |
| 123456789012:journald-kernel | 5 | 5 |
| 123456789012:journald-kernel | 6 | 62 |
| 123456789012:journald-units | 5 | 34 |
| 123456789012:journald-units | 6 | 77 |
Counting audit events by type (recordsMatched 176).
stats count(*) as cnt by body._AUDIT_TYPE_NAME | sort cnt desc | limit 8
| body._AUDIT_TYPE_NAME | cnt |
|---|---|
| BPF | 74 |
| USER_START | 12 |
| USER_ACCT | 11 |
| CRYPTO_KEY_USER | 10 |
| CRED_DISP | 10 |
| SERVICE_STOP | 9 |
| USER_END | 9 |
| SERVICE_START | 7 |
Extracting only sshd invalid user connections (recordsMatched 2).
filter body._SYSTEMD_UNIT = "sshd.service" and body.MESSAGE like /Invalid user/
| fields @timestamp, body.MESSAGE | sort @timestamp desc
| @timestamp | body.MESSAGE |
|---|---|
| 2026-08-29 05:56:09.986 | Invalid user nosuchuser from 127.0.0.1 port 54494 |
| 2026-08-29 05:52:10.066 | Invalid user nosuchuser from 127.0.0.1 port 47170 |
Summary
With the CloudWatch Agent's journald support, logs from Amazon Linux 2023's systemd units, kernel, audit, and Docker were able to be collected into CloudWatch Logs. No configuration to write the journal to a file was required. Each event is sent as structured JSON with journald fields preserved. Unit names and audit event types can be used directly for aggregation and searching from Logs Insights.
Configurations that added rsyslog solely for passing data to the CloudWatch Agent, or that tailed journals after offloading them to files, can be simplified using the journald section. In particular, if you want to collect logs from the OS and services managed under systemd together, consider migrating.
Appendix: UserData
This is UserData that completes the same configuration without post-configuration via SSM. From agent installation to journald configuration application, everything completes at startup.
#!/bin/bash
set -xeuo pipefail
# Install Docker as an example of an application managed under systemd
dnf install -y docker
systemctl enable --now docker
# The dnf repository version for AL2023 is a non-journald-compatible version, so use the latest RPM from S3
cd /tmp
curl -sS -O https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
dnf install -y ./amazon-cloudwatch-agent.rpm
cat > /opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json <<'CONFIG'
{
"agent": { "run_as_user": "root" },
"logs": {
"logs_collected": {
"journald": {
"collect_list": [
{
"log_group_name": "journald-userdata-units",
"log_stream_name": "{instance_id}",
"units": ["sshd", "systemd-logind", "chronyd", "amazon-ssm-agent"],
"retention_in_days": 1
},
{
"log_group_name": "journald-userdata-kernel",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "kernel"}],
"retention_in_days": 1
},
{
"log_group_name": "journald-userdata-audit",
"log_stream_name": "{instance_id}",
"matches": [{"_TRANSPORT": "audit"}],
"priority": "debug",
"retention_in_days": 1
},
{
"log_group_name": "journald-userdata-docker",
"log_stream_name": "{instance_id}",
"units": ["docker"],
"retention_in_days": 1
}
]
}
}
}
}
CONFIG
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config -m ec2 -s \
-c file:/opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json
