Introduction

Hello everyone, I'm Akaike.

I recently had the opportunity to work with ParallelCluster, and in an environment using Permissions Boundary as a countermeasure against IAM privilege escalation, I encountered a situation where the CloudFormation for cluster creation failed with Permission Deny.

So this time, I actually tried using pcluster's "PermissionsBoundary mode" and summarized my experience, including the pitfalls I encountered.

Prerequisites

• AWS ParallelCluster v3 (version 3.15.1 at the time of verification)
• Scheduler is Slurm
• Assuming Permissions Boundary is attached to the IAM principal used for running pcluster
  • In this article, I'll verify using a role that has a policy set as a boundary to "prohibit IAM role creation without Permissions Boundary" as a countermeasure against privilege escalation
• Existing VPC and subnets for placing HeadNode/Compute are already prepared

How to Use Permissions Boundary with ParallelCluster

ParallelCluster automatically creates IAM roles for HeadNode and Compute nodes using iam:CreateRole when creating a cluster.
On the other hand, in environments using Permissions Boundary, it is common to enforce a control that "prohibits creating IAM roles without attaching a Boundary."

Therefore, if you don't take Permissions Boundary into account when creating a cluster, the IAM roles created by pcluster will be blocked by the Boundary's Deny, causing cluster creation to fail.

ParallelCluster officially provides a "PermissionsBoundary mode" to work around this, so let's walk through how to use it to resolve the issue.

ParallelCluster IAM Management Modes and PermissionsBoundary Mode

ParallelCluster v3 has three modes for handling IAM permissions of the pcluster executor.

https://docs.aws.amazon.com/ja_jp/parallelcluster/latest/ug/iam-roles-in-parallelcluster-v3.html#iam-roles-in-parallelcluster-v3-user-policy-manage-iam

The third option, PermissionsBoundary mode, fits most cleanly into environments that use Permissions Boundary as a countermeasure against privilege escalation.

In this mode, when you specify a policy ARN in Iam.PermissionsBoundary in the cluster configuration file, ParallelCluster will attach that policy as a Boundary to all IAM roles it automatically creates under the parallelcluster/* path.

The Permissions Boundary Used This Time

For verification, I attached the following Permissions Boundary to the role used for running pcluster.
Specifically, the condition is: "Deny if pcluster-dev-boundary is not specified as iam:PermissionsBoundary in the iam:CreateRole request."

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAll",
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    },
    {
      "Sid": "DenyRoleCreationWithoutBoundary",
      "Effect": "Deny",
      "Action": "iam:CreateRole",
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "iam:PermissionsBoundary": "arn:aws:iam::*:policy/pcluster-dev-boundary"
        }
      }
    }
  ]
}

Creating a Cluster

①: Creating Without Specifying Permissions Boundary

First, let's create without specifying Permissions Boundary and confirm that an error occurs.
If you create a cluster without writing Iam.PermissionsBoundary in the cluster configuration file, pcluster will try to create roles without a Boundary, which will be blocked by the Boundary's deny.

Below is the cluster configuration file without the Iam section.

Region: ap-northeast-1
Image:
  Os: alinux2023
HeadNode:
  InstanceType: t3.medium
  Networking:
    SubnetId: subnet-xxxxxxxx
  Iam:
    AdditionalIamPolicies:
      - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: compute
      ComputeResources:
        - Name: cr-compute
          InstanceType: t3.micro
          MinCount: 0
          MaxCount: 1
      Networking:
        SubnetIds:
          - subnet-xxxxxxxx
# Intentionally not writing Iam.PermissionsBoundary

Creating with this configuration.

pcluster create-cluster \
  --cluster-name pc-ng \
  --cluster-configuration cluster-config.yaml

The creation starts with CREATE_IN_PROGRESS, but after a while it becomes CREATE_FAILED.
Looking at the CloudFormation events, you can clearly see that iam:CreateRole is being blocked by the Boundary's Deny.

User: arn:aws:sts::123456789012:assumed-role/.../pcluster-test-user
is not authorized to perform: iam:CreateRole
on resource: arn:aws:iam::123456789012:role/parallelcluster/pc-deny-repro/pc-deny-repro-RoleHeadNode-xxxx
with an explicit deny in a permissions boundary:
arn:aws:iam::123456789012:policy/pcluster-dev-boundary

Since it says with an explicit deny in a permissions boundary, we can identify that the Boundary is the cause.

②: Creating With Permissions Boundary Specified

The error in ① can be resolved simply by writing the Boundary ARN in Iam.PermissionsBoundary in the cluster configuration file.
With this, pcluster calls iam:CreateRole with the Boundary attached to the automatically created roles, satisfying the deny condition and allowing it to pass.

Region: ap-northeast-1
Image:
  Os: alinux2023
HeadNode:
  InstanceType: t3.medium
  Networking:
    SubnetId: subnet-xxxxxxxx
  Iam:
    AdditionalIamPolicies:
      - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: compute
      ComputeResources:
        - Name: cr-compute
          InstanceType: t3.micro
          MinCount: 0
          MaxCount: 2
      Networking:
        SubnetIds:
          - subnet-xxxxxxxx
# This is the key point this time
Iam:
  PermissionsBoundary: arn:aws:iam::123456789012:policy/pcluster-dev-boundary

Creating with this configuration.

pcluster create-cluster \
  --cluster-name pc-ok \
  --cluster-configuration cluster-config.yaml

By writing Iam.PermissionsBoundary, iam:CreateRole passed successfully and CREATE_COMPLETE was achieved.

Pitfalls

Here I'll share the issues I got stuck on during the actual process.
Many are environmental factors not directly related to Permissions Boundary, but since others are likely to get stuck in the same places, I'll summarize them here.

Python Too New for pcluster to Work

At first, create-cluster failed with the following error during configuration validation.

{
  "message": "Invalid cluster configuration: There is no current event loop in thread 'MainThread'."
}

This was caused by the Python version being too new.
Due to changes in event loop handling in Python 3.12, internal pcluster processing fails in 3.13/3.14.

My environment was Python 3.14, but downgrading to 3.12 resolved it.
Since ParallelCluster 3.14.x/3.15.x supports Python 3.12, using 3.12 seems to be the safe choice.

https://github.com/aws/aws-parallelcluster/issues/7230

Required to Install botocore[crt]

When I ran pcluster with aws login credentials, I was prompted to install an additional dependency package as follows.

{
  "message": "Missing Dependency: Using the login credential provider requires an additional dependency. You will need to pip install \"botocore[crt]\" before proceeding."
}

As the message says, installing it additionally within pcluster's venv resolves this.

pip install "botocore[crt]"

Operation Verification

Since it was my first time working with ParallelCluster, I also verified that it could be used properly.

Shared Storage (/home) Verification

I verified that files written on HeadNode can be read from Compute nodes.
ParallelCluster shares HeadNode's /home to Compute via NFS by default.

$ echo "written on headnode at $(date)" > /home/ec2-user/shared_test.txt
$ srun -N1 bash -c 'hostname; cat /home/ec2-user/shared_test.txt'
compute-dy-cr-compute-1
written on headnode at Fri Aug 14 17:54:37 UTC 2026

Since the content written by HeadNode can be read from the Compute node side, shared storage is working without issues.

Multi-node MPI Job

Next, I submitted an MPI job spanning 2 nodes to verify that inter-node communication works.
I prepared a simple MPI program, built it, and created a job script with --nodes=2.

module load openmpi
cat > ~/hello.c <<'EOF'
#include <mpi.h>
#include <stdio.h>
int main(int argc,char**argv){
  MPI_Init(&argc,&argv);
  int rank,size,len; char name[MPI_MAX_PROCESSOR_NAME];
  MPI_Comm_rank(MPI_COMM_WORLD,&rank);
  MPI_Comm_size(MPI_COMM_WORLD,&size);
  MPI_Get_processor_name(name,&len);
  printf("rank %d / %d on %s\n",rank,size,name);
  MPI_Finalize(); return 0;
}
EOF
mpicc ~/hello.c -o ~/hello

cat > ~/mpi.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=mpihello
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --output=/home/ec2-user/mpi-%j.out
module load openmpi
srun --mpi=pmix /home/ec2-user/hello
EOF

Submitting this job allocates 2 Compute instances.

$ sbatch ~/mpi.sbatch
Submitted batch job 3
$ squeue
   JOBID PARTITION     NAME     USER ST  TIME  NODES NODELIST(REASON)
       3   compute mpihello ec2-user CF  0:01      2 compute-dy-cr-compute-[1-2]

NODES is 2 and NODELIST is compute-dy-cr-compute-[1-2], confirming that the job was scheduled across 2 nodes.

Auto-scaling (Scale Up/Down)

I also confirmed the behavior of scaling up to MaxCount when multiple jobs are submitted at once, and scaling down after jobs complete.

$ for i in $(seq 1 4); do sbatch --wrap="sleep 120" -N1; done
$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      2 alloc~ compute-dy-cr-compute-[1-2]   # Starting up
...
$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      2  alloc compute-dy-cr-compute-[1-2]   # Job running
...
$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      2   idle compute-dy-cr-compute-[1-2]   # Job completed

It transitions from alloc~ (starting up) to alloc (running), and then to idle after job completion.
After this, if ScaledownIdletime (default 10 minutes) of idle time continues, the Compute nodes will automatically terminate.

Can Cluster Updates Be Done Without Issues

Since IAM role creation and modification can occur during updates via update-cluster, I also checked whether "Boundary passes without issues not only during creation but also during updates."

I performed an update to change MaxCount from 1 to 2 after stopping the compute fleet.

# Changing MaxCount requires stopping the fleet first
pcluster update-compute-fleet --cluster-name pc-ok --status STOP_REQUESTED

# After changing MaxCount from 1 -> 2 in cluster-config.yaml
pcluster update-cluster \
  --cluster-name pc-ok \
  --cluster-configuration cluster-config.yaml

# Wait for update completion (UPDATE_COMPLETE) then restart the fleet
pcluster update-compute-fleet --cluster-name pc-ok --status START_REQUESTED

The update reflected the MaxCount change in the changeSet and completed without being blocked by the Boundary.
In other words, it was confirmed that by explicitly specifying the Boundary in the cluster configuration file, not only cluster creation but also cluster updates work without issues.

Note that the fleet cannot be started while an update is UPDATE_IN_PROGRESS.
I made the mistake of running START_REQUESTED without waiting and got the following error.
This is a sequencing issue rather than an actual error, so just re-run it after UPDATE_COMPLETE.

{
  "message": "Bad Request: Failed when starting compute fleet with error: Cannot start/enable compute fleet while stack is in UPDATE_IN_PROGRESS status."
}

Conclusion

That concludes my experience using Permissions Boundary with AWS ParallelCluster.

The fix for the Permission Deny was as simple as writing one line of Iam.PermissionsBoundary, and pcluster worked without issues even in a Boundary environment with privilege escalation countermeasures.
I hope this serves as a helpful reference for those trying to use pcluster in a similar environment.