CloudWatch Agent Now Supports journald, So I Tried It Out on Amazon Linux 2023

CloudWatch Agent Now Supports journald, So I Tried It Out on Amazon Linux 2023

The CloudWatch agent can now collect systemd journal logs directly. On Amazon Linux 2023, which has no /var/log/messages, I sent systemd unit, kernel, audit and Docker logs to CloudWatch Logs and queried them with Logs Insights.
2026.08.29

This page has been translated by machine translation. View original

Introduction

On August 28, 2026, the CloudWatch agent added support for collecting logs from the systemd journal (journald).

https://aws.amazon.com/about-aws/whats-new/2026/08/amazon-cloudwatch-agent-journald/

Item Before Now
How journald logs are collected Additional configuration to export journald to files was required The agent reads journald directly
Offloading to files for the CloudWatch agent The journal had to be written out to files Not required
Structured metadata Could be lost when going through files Sent with unit name, priority, and process information preserved
Filtering None Unit, priority, field match, and regular expression filters

The standard Amazon Linux 2023 configuration has no /var/log/messages and does not include rsyslog. Because the logs emitted by the OS are gathered in journald, I configured the agent with only the journald section, without using files, which specifies files. I collected four kinds of logs (systemd units, kernel messages, audit events, and Docker) into separate log groups. I verified the format of the events that arrived in CloudWatch Logs and even queries in Logs Insights.

What I Verified

Test Environment

Item Value
OS Amazon Linux 2023.12.20260817
Kernel 6.18.41-94.142.amzn2023.x86_64
systemd 252
Instance t3.micro
Region ap-northeast-1
CloudWatch agent 1.300072.0b1766

I did not open any inbound ports, and all operations were performed through SSM. The subnet is a public subnet, the instance was assigned a public IP, and outbound traffic is allowed. Connectivity to SSM, S3, the Amazon Linux repositories, and CloudWatch Logs is required.

journald support is a feature of agent version 1.300070.0 and later. As of August 29, 2026, the version available from the amazonlinux repository on Amazon Linux 2023 was 1.300069.1. Therefore I switched the source to the latest RPM on S3 (https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm).

https://github.com/aws/amazon-cloudwatch-agent/blob/main/RELEASE_NOTES

Least Privilege and Deployment

I created the EC2 instance and IAM role using CloudFormation Express mode. In addition to the managed policy for SSM, the instance role limits CloudWatch Logs permissions to just four actions. Because the configuration file specifies retention_in_days, I also included logs:PutRetentionPolicy. The retention period is set to 1 day for this test.

The instance role part of the template is as follows.

  InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
      Policies:
        - PolicyName: journald-logs-minimal
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - logs:PutRetentionPolicy
                Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:journald-*
aws cloudformation create-stack \
  --stack-name journald-demo \
  --template-body file://journald-demo.yaml \
  --capabilities CAPABILITY_IAM \
  --deployment-config Mode=EXPRESS \
  --parameters ParameterKey=VpcId,ParameterValue=vpc-xxxxxxxxxxxxxxxxx \
               ParameterKey=SubnetId,ParameterValue=subnet-xxxxxxxxxxxxxxxxx

journald Configuration and Delivery Format

There are four entries to collect, each sent to a separate log group.

{
  "agent": {
    "run_as_user": "root"
  },
  "logs": {
    "logs_collected": {
      "journald": {
        "collect_list": [
          {
            "log_group_name": "journald-units",
            "log_stream_name": "{instance_id}",
            "units": ["sshd", "systemd-logind", "chronyd", "amazon-ssm-agent"],
            "retention_in_days": 1
          },
          {
            "log_group_name": "journald-kernel",
            "log_stream_name": "{instance_id}",
            "matches": [{"_TRANSPORT": "kernel"}],
            "retention_in_days": 1
          },
          {
            "log_group_name": "journald-audit",
            "log_stream_name": "{instance_id}",
            "matches": [{"_TRANSPORT": "audit"}],
            "priority": "debug",
            "retention_in_days": 1
          },
          {
            "log_group_name": "journald-docker",
            "log_stream_name": "{instance_id}",
            "units": ["docker"],
            "retention_in_days": 1
          }
        ]
      }
    }
  }
}

I applied it with the following command.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json

I retrieved one event of the systemd unit logs that arrived in CloudWatch Logs.

{
    "events": [
        {
            "timestamp": 1787982970748,
            "message": "{\"body\":{\"MESSAGE\":\"pam_unix(sudo:session): session closed for user root\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"10\",\"SYSLOG_IDENTIFIER\":\"sudo\",\"_CMDLINE\":\"sudo docker run --rm public.ecr.aws/docker/library/alpine:3.20 true\",\"_COMM\":\"sudo\",\"_EXE\":\"/usr/bin/sudo\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"29152\",\"_SYSTEMD_UNIT\":\"amazon-ssm-agent.service\",\"_TRANSPORT\":\"syslog\",\"_UID\":\"0\",\"__CURSOR\":\"s=xxxxxxxx;i=e09;b=xxxxxxxx;m=2155d6a2;t=65a2939d5ff89;x=xxxxxxxx\"}}",
            "ingestionTime": 1787982974133
        }
    ]
}

Each event arrives as structured JSON with all of the journald fields placed under body. It is not plain text containing only the MESSAGE body. The above is an excerpt with some fields omitted (the same applies to the event JSON shown below).

I also checked the body of a successful sshd login event.

{
 "body": {
  "MESSAGE": "Accepted publickey for ec2-user from 127.0.0.1 port 54478 ssh2: ED25519 SHA256:xxxxxxxx",
  "PRIORITY": "6",
  "SYSLOG_FACILITY": "10",
  "SYSLOG_IDENTIFIER": "sshd-session",
  "SYSLOG_PID": "29125",
  "_COMM": "sshd-session",
  "_EXE": "/usr/libexec/openssh/sshd-session",
  "_HOSTNAME": "ip-10-0-x-x.ap-northeast-1.compute.internal",
  "_PID": "29125",
  "_SELINUX_CONTEXT": "system_u:system_r:sshd_t:s0-s0:c0.c1023",
  "_SYSTEMD_CGROUP": "/system.slice/sshd.service",
  "_SYSTEMD_UNIT": "sshd.service",
  "_TRANSPORT": "syslog",
  "_UID": "0"
 }
}

What I specified in the configuration was sshd, but SYSLOG_IDENTIFIER was sshd-session and _SYSTEMD_UNIT was sshd.service. Specification by unit works on the unit name, not the identifier.

The source being 127.0.0.1 is because no sshd logs were produced in the Session Manager shell session I used this time. I generated logs by running ssh ec2-user@localhost with a throwaway key from inside the instance.

Note that fetch-config deletes etc/amazon-cloudwatch-agent.json when applied and also replaces existing files under amazon-cloudwatch-agent.d. Therefore, be aware that the procedure shown here overwrites any existing agent configuration.

journalctl that the agent launches internally
journalctl --utc --output=json --follow --priority debug _TRANSPORT=audit
journalctl --utc --output=json --follow --priority info _TRANSPORT=kernel --after-cursor s=xxxxxxxx;i=9be;...
journalctl --utc --output=json --follow --unit docker --priority info --after-cursor s=xxxxxxxx;i=9b2;...
journalctl --utc --output=json --follow --unit sshd --unit systemd-logind --unit chronyd --unit amazon-ssm-agent --priority info --after-cursor s=xxxxxxxx;i=cb2;...

One entry in collect_list corresponds to one journalctl process. The unit names, field specifications, and priority written in the configuration became command-line arguments as they were.

--after-cursor is attached to collect_list entries that already have a cursor, so past logs are not collected again.

Kernel and Audit Logs

I collected kernel messages by specifying _TRANSPORT as kernel in matches.

{
    "events": [
        {
            "timestamp": 1787982734525,
            "message": "{\"body\":{\"MESSAGE\":\"veth97da191 (unregistering): left allmulticast mode\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"0\",\"SYSLOG_IDENTIFIER\":\"kernel\",\"_BOOT_ID\":\"xxxxxxxx\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_TRANSPORT\":\"kernel\",\"__CURSOR\":\"s=xxxxxxxx;i=9bc;b=xxxxxxxx;m=13415a86;t=65a292bc1836e;x=xxxxxxxx\"}}",
            "ingestionTime": 1787982739614
        }
    ]
}

Kernel messages have no unit name, so they cannot be specified with units. journalctl --unit kernel over the 20 minutes during which logs were collected in the test returned "-- No entries --". On the other hand, journalctl _TRANSPORT=kernel returned 639 lines.

On Amazon Linux 2023, auditd and systemd-journald-audit.socket are both active by default, so audit events also reach journald. During the 20 minutes of log collection, 911 entries were in scope. However, of those 911, 0 had a PRIORITY field. The documentation states that the default value of priority is info (debug is excluded). Since the agent launches with journalctl --priority info, audit entries are excluded from collection by default.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html

With the default priority, audit events could not be collected, and the journald-audit log group itself was not even created. The other three log groups were created. When I specified debug for priority, all 911 entries were in scope, and the agent also created the log group.

{
    "events": [
        {
            "timestamp": 1787983006400,
            "message": "{\"body\":{\"AUDIT_FIELD_EXE\":\"/usr/lib/systemd/systemd\",\"AUDIT_FIELD_RES\":\"success\",\"AUDIT_FIELD_UNIT\":\"user@0\",\"MESSAGE\":\"SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user@0 comm=\\\"systemd\\\" exe=\\\"/usr/lib/systemd/systemd\\\" hostname=? addr=? terminal=? res=success'\",\"SYSLOG_FACILITY\":\"4\",\"SYSLOG_IDENTIFIER\":\"audit\",\"_AUDIT_ID\":\"1433\",\"_AUDIT_TYPE\":\"1131\",\"_AUDIT_TYPE_NAME\":\"SERVICE_STOP\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"1\",\"_TRANSPORT\":\"audit\",\"_UID\":\"0\"}}",
            "ingestionTime": 1787983009138
        }
    ]
}

Audit events arrived with _AUDIT_TYPE_NAME and AUDIT_FIELD_* preserved.

Docker Logs

As an example of an application managed by systemd, I separated Docker logs into a dedicated log group.

{
    "events": [
        {
            "timestamp": 1787982718503,
            "message": "{\"body\":{\"MESSAGE\":\"time=\\\"2026-08-29T05:51:58.502767587Z\\\" level=info msg=\\\"API listen on /run/docker.sock\\\"\",\"PRIORITY\":\"6\",\"SYSLOG_FACILITY\":\"3\",\"SYSLOG_IDENTIFIER\":\"dockerd\",\"_CMDLINE\":\"/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536\",\"_COMM\":\"dockerd\",\"_EXE\":\"/usr/bin/dockerd\",\"_HOSTNAME\":\"ip-10-0-x-x.ap-northeast-1.compute.internal\",\"_PID\":\"7416\",\"_SYSTEMD_UNIT\":\"docker.service\",\"_TRANSPORT\":\"stdout\",\"_UID\":\"0\"}}",
            "ingestionTime": 1787982724803
        }
    ]
}

When docker was specified as the unit, dockerd logs were collected. The same log group also contains docker.service-related messages emitted by systemd itself (entries whose CODE_FILE is src/core/job.c). The scope is the same as journalctl -u docker.

Logs Insights Queries

Because the delivery format is JSON, the fields under body became discovered fields as they were, and no parse was needed. All of the following queries target the last hour. First, in the systemd unit log group, count by unit (recordsMatched 111).

stats count(*) as cnt by body._SYSTEMD_UNIT | sort cnt desc
body._SYSTEMD_UNIT cnt
amazon-ssm-agent.service 95
sshd.service 10
systemd-logind.service 6

Next, across the four log groups, break down by priority (logGroupsScanned 4, recordsMatched 366).

stats count(*) as cnt by @log, body.PRIORITY | sort @log, body.PRIORITY
@log body.PRIORITY cnt
123456789012:journald-audit (empty) 186
123456789012:journald-docker 6 2
123456789012:journald-kernel 5 5
123456789012:journald-kernel 6 62
123456789012:journald-units 5 34
123456789012:journald-units 6 77

Count audit events by type (recordsMatched 176).

stats count(*) as cnt by body._AUDIT_TYPE_NAME | sort cnt desc | limit 8
body._AUDIT_TYPE_NAME cnt
BPF 74
USER_START 12
USER_ACCT 11
CRYPTO_KEY_USER 10
CRED_DISP 10
SERVICE_STOP 9
USER_END 9
SERVICE_START 7

Extract only sshd connections by invalid users (recordsMatched 2).

filter body._SYSTEMD_UNIT = "sshd.service" and body.MESSAGE like /Invalid user/
| fields @timestamp, body.MESSAGE | sort @timestamp desc
@timestamp body.MESSAGE
2026-08-29 05:56:09.986 Invalid user nosuchuser from 127.0.0.1 port 54494
2026-08-29 05:52:10.066 Invalid user nosuchuser from 127.0.0.1 port 47170

Summary

With the CloudWatch agent's journald support, I was able to collect systemd unit, kernel, audit, and Docker logs on Amazon Linux 2023 into CloudWatch Logs. No configuration to write the journal out to files was needed. Each event is sent as structured JSON that preserves the journald fields. Using unit names and audit event types, you can aggregate and search directly from Logs Insights.

Setups that add rsyslog solely to feed the CloudWatch agent, or that offload the journal to files and tail them, can be simplified with the journald section. In particular, if you want to collect logs of the OS and of services managed by systemd all together, consider migrating.

Appendix: UserData

This is UserData that builds the same configuration without post-configuring via SSM. Everything from installing the agent to applying the journald configuration completes at boot.

#!/bin/bash
set -xeuo pipefail

# systemd 管理下のアプリ例として Docker を導入
dnf install -y docker
systemctl enable --now docker

# AL2023 の dnf リポジトリ版は journald 非対応バージョンのため S3 の最新 RPM を使う
cd /tmp
curl -sS -O https://amazoncloudwatch-agent.s3.amazonaws.com/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
dnf install -y ./amazon-cloudwatch-agent.rpm

cat > /opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json <<'CONFIG'
{
  "agent": { "run_as_user": "root" },
  "logs": {
    "logs_collected": {
      "journald": {
        "collect_list": [
          {
            "log_group_name": "journald-userdata-units",
            "log_stream_name": "{instance_id}",
            "units": ["sshd", "systemd-logind", "chronyd", "amazon-ssm-agent"],
            "retention_in_days": 1
          },
          {
            "log_group_name": "journald-userdata-kernel",
            "log_stream_name": "{instance_id}",
            "matches": [{"_TRANSPORT": "kernel"}],
            "retention_in_days": 1
          },
          {
            "log_group_name": "journald-userdata-audit",
            "log_stream_name": "{instance_id}",
            "matches": [{"_TRANSPORT": "audit"}],
            "priority": "debug",
            "retention_in_days": 1
          },
          {
            "log_group_name": "journald-userdata-docker",
            "log_stream_name": "{instance_id}",
            "units": ["docker"],
            "retention_in_days": 1
          }
        ]
      }
    }
  }
}
CONFIG

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/cw-journald.json

Share this article

AWSのお困り事はクラスメソッドへ