AWS Parallel Computing Service Now Supports Node Lifecycle Actions, So I Tried It Out
This page has been translated by machine translation. View original
Introduction
AWS Parallel Computing Service (AWS PCS) is a managed service for Slurm HPC clusters. The July 2026 update added node lifecycle actions. Node lifecycle actions let you run scripts stored in places like S3 while a compute node is starting up. For ParallelCluster users, this means custom bootstrap actions are now available.
What's good about it
In AWS PCS, EC2 instances with the same configuration are grouped as a compute node group and associated with a queue to accept jobs. The PCS agent runs on each node and joins the node to the cluster. Until now, when you wanted to add configuration or install applications at node startup, you used the user data of the launch template.
A feature like ParallelCluster's bootstrap scripts
Lifecycle actions correspond to the custom bootstrap actions (CustomActions) of AWS ParallelCluster. To explain it for users who have used ParallelCluster since v2, they are pre_install and post_install.
The official documentation included the following correspondence table.
| AWS ParallelCluster | AWS PCS |
|---|---|
OnNodeStart |
nodeBootstrapped |
OnNodeConfigured |
nodeReady |
OnNodeUpdated |
No equivalent |
Most existing bootstrap scripts are said to work as they are. However, the default execution timing differs. ParallelCluster runs them on every boot, but PCS runs them only on the first boot. To get the same behavior as ParallelCluster, specify EVERY_BOOT for executionPolicy.
No need to touch the launch template
User data is merged with PCS's own user data. For that reason, you have to write it in MIME multipart format. If you pass it from the CLI or an SDK, base64 encoding is also required.
Choose reboot and error handling per script
With user data, you cannot specify per script whether to rerun on reboot or what to do on failure. With lifecycle actions, you can specify both per script.
The official documentation also recommends lifecycle actions for most configuration. User data should be used only for processing that must happen before the PCS agent starts.
For most node configuration tasks, we recommend node lifecycle actions. They provide reusable, API-native configuration with per-script error handling and reboot control. Use launch template user data for tasks that must run before the AWS PCS agent starts.
Source: https://docs.aws.amazon.com/pcs/latest/userguide/working-with_ec2-user-data.html
How lifecycle actions work
Lifecycle actions are configured per compute node group. Scripts run as root, and you choose the stage to run in from two options.
| Stage | Execution timing | Slurm commands |
|---|---|---|
nodeBootstrapped |
After PCS node configuration, before slurmd starts | Not available |
nodeReady |
After slurmd registers with the controller | Available |
Because nodeReady runs after slurmd starts, jobs can be assigned even before the script completes.
The two main settings specified per script are as follows.
| Setting | Value | Behavior | When to use |
|---|---|---|---|
executionPolicy |
FIRST_BOOT_ONLY (default) |
Runs only on the first boot | One-time setup such as installing packages |
EVERY_BOOT |
Runs on every reboot | Setup that must be redone after a reboot | |
onError |
TERMINATE (default) |
Treats the node as failed and terminates it | Required setup such as mounting shared storage |
STOP_SEQUENCE |
Does not run subsequent scripts in the same stage and keeps the node | Debugging scripts | |
CONTINUE |
Records the failure and moves on to the next script | Optional processing that may fail |
The documentation describes which to choose in which case, so please check the latest information.
Test environment
We start from a state where the cluster and a compute node group for login have already been created. The instance role has s3:GetObject for the S3 bucket where the scripts are placed. Lifecycle actions require PCS agent 1.5.0-1 or later. I will skip the cluster build steps themselves. For how to create a PCS cluster, refer to the following article.
| Item | Value |
|---|---|
| Region | ap-northeast-1 |
| Slurm | 26.05.4 |
| PCS agent | 1.5.1-1 |
| AMI | PCS sample AMI (aws-pcs-sample_ami-al2023-x86_64-2026-10-14) |
| Instance type | c7i.large |
| AWS CLI | 2.37.3 |
Scripts used for testing
I placed three scripts under scripts/ in the S3 bucket.
record-boot.sh uses its first argument as a label and appends one line together with the values of environment variables. The destination is /var/lib/lca-boot-record.log. This file is for counting how many times the script ran before and after a reboot.
#!/bin/bash
set -euo pipefail
LABEL="${1:-unknown}"
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) label=${LABEL} first_boot=${PCS_IS_FIRST_BOOT:-unset} node=${PCS_NODE_ID:-unset} cng=${PCS_COMPUTE_NODE_GROUP_NAME:-unset}" >> /var/lib/lca-boot-record.log
echo "recorded ${LABEL}"
ready-check.sh is used in nodeReady and outputs the slurmd state and the result of scontrol show node. The Slurm node name (lca-static-1) differs from the hostname (ip-10-0-1-82). For that reason, the node is specified with PCS_NODE_ID.
#!/bin/bash
set -uo pipefail
echo "slurmd: $(systemctl is-active slurmd 2>/dev/null || echo unknown)"
SLURM_BIN=$(ls -d /opt/aws/pcs/scheduler/slurm-*/bin 2>/dev/null | sort | tail -1)
"${SLURM_BIN}/scontrol" show node "${PCS_NODE_ID:-$(hostname -s)}" 2>&1 | head -5 || true
fail.sh always fails with exit code 1. It is used to check onError.
#!/bin/bash
echo "intentional failure for verification" >&2
exit 1
Creating a compute node group with lifecycle actions configured
For nodeBootstrapped, I registered record-boot.sh twice with different arguments. One is FIRST_BOOT_ONLY and the other is EVERY_BOOT. For ready-check in nodeReady, I omitted executionPolicy and set onError to CONTINUE so that the node is kept even on failure.
{
"stages": {
"nodeBootstrapped": [
{
"name": "record-first-boot",
"scriptSource": { "scriptLocation": "s3://<バケット名>/scripts/record-boot.sh" },
"arguments": ["bootstrapped-first-boot-only"],
"executionPolicy": "FIRST_BOOT_ONLY"
},
{
"name": "record-every-boot",
"scriptSource": { "scriptLocation": "s3://<バケット名>/scripts/record-boot.sh" },
"arguments": ["bootstrapped-every-boot"],
"executionPolicy": "EVERY_BOOT"
}
],
"nodeReady": [
{
"name": "ready-check",
"scriptSource": { "scriptLocation": "s3://<バケット名>/scripts/ready-check.sh" },
"onError": "CONTINUE"
}
]
}
}
Pass this JSON to --node-lifecycle-actions using file://.
aws pcs create-compute-node-group --region ap-northeast-1 \
--cluster-identifier <クラスター ID> \
--compute-node-group-name lca-static \
--subnet-ids <サブネット ID> \
--custom-launch-template id=<起動テンプレート ID>,version=1 \
--iam-instance-profile-arn <インスタンスプロファイル ARN> \
--ami-id <サンプル AMI の ID> \
--scaling-configuration minInstanceCount=1,maxInstanceCount=1 \
--instance-configs instanceType=c7i.large \
--node-lifecycle-actions file://<JSON ファイルのパス>
It became ACTIVE in about 3 minutes, so I created the queue lca-queue and associated it.
In the console, the registered scripts are displayed per stage. The omitted error handling and execution policy were shown as "-".
From "Add script" in the console, you can choose any of the script library, S3, or an HTTPS URL to add a script.
Logs from the first boot are saved in per-stage directories
After the node became idle, I checked the logs on the node using Systems Manager Run Command. Inside the directories named after the stages, log files named after the scripts had been created.
/var/log/amazon/pcs/lifecycle/actions/:
-rw-r--r-- 1 root root 4334 2026-10-21 00:48:00.140915114 +0000 executor.log
drwxr-x--- 2 root root 64 2026-10-21 00:47:39.549764603 +0000 nodeBootstrapped
drwxr-x--- 2 root root 29 2026-10-21 00:48:00.070915040 +0000 nodeReady
/var/log/amazon/pcs/lifecycle/actions/nodeBootstrapped/:
-rw-r----- 1 root root 149 2026-10-21 00:47:39.551852990 +0000 record-every-boot.log
-rw-r----- 1 root root 159 2026-10-21 00:47:39.549764603 +0000 record-first-boot.log
/var/log/amazon/pcs/lifecycle/actions/nodeReady/:
-rw-r----- 1 root root 122 2026-10-21 00:48:00.140915114 +0000 ready-check.log
On reboot, only the EVERY_BOOT script was rerun from the cache
I rebooted the instance without replacing it using scontrol reboot.
Because I ran it on the login node via Run Command, I specified the Slurm command with its full path.
#!/bin/bash
/opt/aws/pcs/scheduler/slurm-26.05/bin/scontrol reboot ASAP nextstate=RESUME lca-static-1
About 45 seconds after execution, the EVERY_BOOT script was rerun. The instance ID did not change. Only one line, bootstrapped-every-boot, was added to lca-boot-record.log, and first_boot was 0.
2026-10-21T00:47:39Z label=bootstrapped-first-boot-only first_boot=1 node=lca-static-1 cng=lca-static
2026-10-21T00:47:39Z label=bootstrapped-every-boot first_boot=1 node=lca-static-1 cng=lca-static
+2026-10-21T00:49:33Z label=bootstrapped-every-boot first_boot=0 node=lca-static-1 cng=lca-static
The script's log had two runs appended. Reusing cached script (CACHE_ONCE) in executor.log shows that the cache was used instead of fetching again from S3.
== /var/log/amazon/pcs/lifecycle/actions/nodeBootstrapped/record-every-boot.log
[start record-every-boot args=["bootstrapped-every-boot"] @ 1792543659]
recorded bootstrapped-every-boot
[end record-every-boot exit=0 @ 1792543659]
[start record-every-boot args=["bootstrapped-every-boot"] @ 1792543773]
recorded bootstrapped-every-boot
[end record-every-boot exit=0 @ 1792543773]
== executor.log (after 00:48:30)
{"timestamp":"2026-10-21T00:49:33.129571Z","level":"INFO","message":"Reusing cached script (CACHE_ONCE)","script":"record-every-boot","stage":"nodeBootstrapped",...}
{"timestamp":"2026-10-21T00:49:33.148413Z","level":"INFO","message":"script completed","stage":"nodeBootstrapped","script":"record-every-boot",...}
The times in the script's log are UNIX seconds, and the second one, 1792543773, is the same time as the line in executor.log. The cache was placed under /etc/amazon/pcs/lifecycle/actions/config/scripts/.
ready-check, for which executionPolicy was omitted, was not rerun, and the log's modification time remained as it was before the reboot.
/var/log/amazon/pcs/lifecycle/actions/nodeReady/:
-rw-r----- 1 root root 122 2026-10-21 00:48:00.140915114 +0000 ready-check.log
When omitted, it is treated as FIRST_BOOT_ONLY, so scripts you want to run on every reboot need EVERY_BOOT specified explicitly.
A node with onError set to STOP_SEQUENCE joined Slurm and ran jobs
At the top of nodeBootstrapped, I configured fail.sh, which always fails, with STOP_SEQUENCE. record-boot.sh is configured after it.
{
"stages": {
"nodeBootstrapped": [
{
"name": "fail-stop-sequence",
"scriptSource": { "scriptLocation": "s3://<バケット名>/scripts/fail.sh" },
"onError": "STOP_SEQUENCE"
},
{
"name": "record-after-fail",
"scriptSource": { "scriptLocation": "s3://<バケット名>/scripts/record-boot.sh" },
"arguments": ["after-fail"]
}
],
"nodeReady": [
{
"name": "ready-check",
"scriptSource": { "scriptLocation": "s3://<バケット名>/scripts/ready-check.sh" },
"onError": "CONTINUE"
}
]
}
}
The update command is as follows.
aws pcs update-compute-node-group --region ap-northeast-1 \
--cluster-identifier <クラスター ID> \
--compute-node-group-identifier lca-static \
--node-lifecycle-actions file://<JSON ファイルのパス>
The existing node was drained, and about 7 minutes after the update, the new instance became idle. On the new instance, fail-stop-sequence.log ended with exit=1. lca-boot-record.log, which the subsequent script writes, had not been created. ready-check in nodeReady was executed, and slurmd was active.
ls: cannot access '/var/lib/lca-boot-record.log': No such file or directory
/var/log/amazon/pcs/lifecycle/actions/nodeBootstrapped/:
-rw-r----- 1 root root 130 2026-10-21 00:57:27.862315978 +0000 fail-stop-sequence.log
/var/log/amazon/pcs/lifecycle/actions/nodeReady/:
-rw-r----- 1 root root 269 2026-10-21 00:57:54.192609490 +0000 ready-check.log
== fail-stop-sequence.log
[start fail-stop-sequence args=[] @ 1792544247]
intentional failure for verification
+[end fail-stop-sequence exit=1 @ 1792544247]
== ready-check.log
[start ready-check args=[] @ 1792544274]
+slurmd: active
NodeName=lca-static-1 Arch=x86_64 CoresPerSocket=1
[end ready-check exit=0 @ 1792544274]
== executor.log
{"timestamp":"2026-10-21T00:57:27.868622Z","level":"ERROR","message":"lifecycle script failed; stopping remaining scripts","stage":"nodeBootstrapped","script":"fail-stop-sequence","on_error":"StopSequence","error":"script 'fail-stop-sequence' failed with exit status ExitStatus(unix_wait_status(256))",...}
== bootstrap.log
[2026-10-21T00:57:27.869+00:00] - /opt/aws/pcs/bin/pcs_bootstrap_finalize.sh: WARN: Lifecycle execute completed with exit 3 for stage nodeBootstrapped
Only the subsequent scripts in the same stage were not executed; the scripts in the next stage, nodeReady, were executed.
I also checked whether jobs run on this node where the script failed. Running srun -p lca-queue hostname from the login node completed with exit code 0. The scontrol show node state was also IDLE+CLOUD, a state in which it accepts jobs.
State=IDLE+CLOUD ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
ip-10-0-1-118.ap-northeast-1.compute.internal
srun exit=0
In PCS's scaling log as well, this node was recorded as a success. The scaling log is a feature that delivers node state transitions to CloudWatch Logs and similar destinations.
Extracting the main items, BOOTSTRAP_SUCCESS was recorded, and there was no failure event.
00:55:45 lca-static-1 LAUNCHED MAINTAIN_MIN_CAPACITY i-046cb3897e5a0a6a3
00:57:03 lca-static-1 REGISTERED REGISTER_SUCCESS i-046cb3897e5a0a6a3
00:57:52 lca-static-1 ACTIVE BOOTSTRAP_SUCCESS i-046cb3897e5a0a6a3
The failure did not appear in the scaling log, the Slurm node state, or the compute node group state. The failure was recorded only in fail-stop-sequence.log, executor.log, and bootstrap.log on the node. STOP_SEQUENCE can be used when you want to keep the node and investigate. The documentation also says its use case is debugging, so that was indeed the case.
Trying onError as TERMINATE
In the JSON from the previous section, I changed onError to TERMINATE, renamed the script to fail-terminate, and updated. In the scaling log, the replacement instances were terminated with BOOTSTRAP_FAILED two times in a row. While the third was starting, I reverted to a configuration without the failing script, so the third was replaced by that update.
| Instance | Launched | Registered | Terminated |
|---|---|---|---|
| 1st | 01:04:45 | 01:06:03 | 01:06:42 BOOTSTRAP_FAILED |
| 2nd | 01:09:44 | 01:11:17 | 01:12:42 BOOTSTRAP_FAILED |
| 3rd | 01:15:45 | None | 01:18:26 REPLACEMENT_REQUESTED |
During this time, the node never became a plain idle without a symbol (the state in which it accepts jobs). In sinfo, it repeated idle# (starting up), idle% (powering down), and idle~ (powered down), and the failed instance ID was shown in Reason.
01:04:45 SINFO=lca-static-1 lca-queue idle# none
01:06:50 SINFO=lca-static-1 lca-queue idle# none
01:07:08 SINFO=lca-static-1 lca-queue idle% Backing Instance Failed To Bootstrap: i-059b4b99b4f2e9739 : Not responding
01:08:19 SINFO=lca-static-1 lca-queue idle~ Backing Instance Failed To Bootstrap: i-059b4b99b4f2e9739 : Not responding
01:09:12 SINFO=lca-static-1 lca-queue idle# none
01:13:04 SINFO=lca-static-1 lca-queue idle% Backing Instance Failed To Bootstrap: i-06644f6e5a2b001c9
01:14:15 SINFO=lca-static-1 lca-queue idle~ Backing Instance Failed To Bootstrap: i-06644f6e5a2b001c9
01:15:08 SINFO=lca-static-1 lca-queue idle# none
One cycle took about 5 to 6 minutes. TERMINATE is the default value of onError, and replacement continued until I reverted the configuration. The logs on the node also disappear along with the instance.
TERMINATE suits configuration where the node is unusable if it fails, such as mounting shared storage. Unlike STOP_SEQUENCE in the previous section, a failed node does not enter a state where it accepts jobs. However, if a script keeps failing, replacement never stops, so during trial and error while writing scripts, it seems better to try with STOP_SEQUENCE. The official documentation also lists temporarily switching to STOP_SEQUENCE while debugging, and forwarding logs to CloudWatch Logs.
About 7 minutes after reverting the configuration, the node returned to idle.
Summary
With node lifecycle actions, you can now run scripts stored in places like S3 while a compute node is starting up. You no longer need to write user data in MIME multipart format, and you can choose rerun on reboot and the behavior on failure per script. You can also bring over your ParallelCluster bootstrap scripts. The official documentation also recommends lifecycle actions for most configuration.
Closing
As a ParallelCluster user myself, this is a familiar approach, and I'm very grateful for it.
