How to set up custom metrics of multiple EC2 instances (Amazon Linux) with Run Command

In this post, we will introduce how to set up custom metrics of disk usage and memory usage on multiple Amazon Linux using Run Command.

Please see blog below for how to set up Windows Server.

How to set up custom metrics of multiple EC2 instances (Windows Server) with Run Command

Environment

• Amazon Linux 2 AMI (amzn2-ami-hvm-2.0.20190228-x86_64-gp2 (ami-095cd038eef3e5074))

How to Setup

Step1. Create Tags for EC2 instances

When you want to manage with tags, we create Tags for EC2 instances. This example adds the tag custom-metrics=enable

Step2. Create IAM role

From the IAM management console, create IAM role.

We will create an IAM role that trusts EC2 service and attache AmazonEC2RoleforSSM SSM policy.

• Roles → Create role
• Select type of trusted entity : AWS service
• Choose the service that will use this role : EC2

• Click Next Permissions
• Attach permissions policies : AmazonEC2RoleforSSM

• Click Next Tags.
• Add tags (optional) is skip and click Next: Reviews.
• Role name* : custom-metrics-role

• Click Create role.

Step3. Attach IAM Role for EC2 instances

From the EC2 management console, attach the IAM role custom-metrics-role to the EC2 instance.

• Select your EC2 instance → Actions → Instance Settings → Attach/Replace IAM Role

• Select IAM role custom-metrics-role

• Click Apply

Step4. Execute Run Command

From the AWS Systems Manager management console, AWS Systems Manager → Run Command → Run a command

• Command document : Document name prefix : Equal : AWS-RunShellScrip

• Command parameters :

• Sample Script

# Create directory
mkdir /opt/cw-monitoring

# Create user
useradd cw-mon

# Memory usage rate Custom metrics send script creation
tee /opt/cw-monitoring/memoryUsage.sh << "EOF" > /dev/null
#!/bin/bash

log() {
echo $1 | logger -t cw-monitoring
}

aws=$(which aws)

# Amazon Linux 2
usage=$(free | head -n 2 | tail -n 1 | awk '{a = $3/($3 + $4)*100; print a}')

# If you use Amazon Linux
# usage=$(free | head -n 3 | tail -n 1 | awk '{a = $3/($3 + $4)*100; print a}')
instanceId=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
region=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/.$//')

cw_opts="--namespace CustomMetrics/EC2"
cw_opts=${cw_opts}" --metric-name MemoryUtilization"
cw_opts=${cw_opts}" --dimensions InstanceId=${instanceId}"
cw_opts=${cw_opts}" --unit Percent"
cw_opts=${cw_opts}" --region ${region}"
cw_opts=${cw_opts}" --value ${usage}"

# wait for 0 - 15 seconds to disperse API request.
sleep $(($RANDOM % 15))

counter=0
MAX_RETRY=3

while :; do
# Error Retry
${aws} cloudwatch put-metric-data ${cw_opts}
if [ $? -ne 0 ]; then
if [ "${counter}" -ge "${MAX_RETRY}" ]; then
log "failed to put metrics of memory usage."
exit 1
fi
else
break
fi

counter=$((counter + 1))
sleep 10
done

exit 0
EOF

# Disk utilization rate Custom metrics transmission script creation
tee /opt/cw-monitoring/diskUsage.sh << "EOF" >> /dev/null
#!/bin/bash

log() {
echo $1 | logger -t cw-monitoring
}

aws=$(which aws)
usage=$(df | sed -e 1d -e 's/%//' | awk '{print $5}' | sort -nr | head -n 1)
instanceId=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
region=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/.$//')

cw_opts="--namespace CustomMetrics/EC2"
cw_opts=${cw_opts}" --metric-name DiskUsage"
cw_opts=${cw_opts}" --dimensions InstanceId=${instanceId}"
cw_opts=${cw_opts}" --unit Percent"
cw_opts=${cw_opts}" --region ${region}"
cw_opts=${cw_opts}" --value ${usage}"

# wait for 0 - 15 seconds to disperse API request.
sleep $(($RANDOM % 15))

counter=0
MAX_RETRY=3

while :; do
# Error Retry
${aws} cloudwatch put-metric-data ${cw_opts}
if [ $? -ne 0 ]; then
if [ "${counter}" -ge "${MAX_RETRY}" ]; then
log "failed to put metrics of disk usage."
exit 1
fi
else
break
fi

counter=$((counter + 1))
sleep 10
done

exit 0
EOF

# Owners, groups, permission settings
chown -R cw-mon:cw-mon /opt/cw-monitoring
chmod 744 /opt/cw-monitoring/*Usage.sh

# Setup cron
tee /var/spool/cron/cw-mon << EOF > /dev/null
*/5 * * * * /opt/cw-monitoring/memoryUsage.sh
*/5 * * * * /opt/cw-monitoring/diskUsage.sh
EOF

note: Since the result of free command is different between amazon linux and amazon linux 2, please select line 17 or 20.

Amazon Linux free command

$ free
total used free shared buffers cached
Mem: 1009432 343908 665524 60 12056 261276
-/+ buffers/cache: 70576 938856
Swap: 0 0 0

Amazon Linux 2 free command

$ free
total used free shared buff/cache available
Mem: 1009384 75924 510504 528 422956 765392
Swap: 0 0 0

• Targets : Specifying a tag
• Tag key :custom-metrics
• Tag value : enable
• Click Add

OR

• Targets : Manually selecting instances

• If you want to write command output to an Amazon S3 bucket, you select S3 bucket name

• Click Run

Step5. Check the Custom Metrics

From the CloudWatch console