Amazon SageMaker HyperPod のワークショップから学ぶ環境構築のハマりどころ

[cloudshell-user@ip-10-134-45-116 automate-smhp-slurm]$ ./automate-cluster-creation.sh

=================================================

==== 🚀 Welcome to the SageMaker HyperPod Cluster Creation Script! 🚀 ====

=================================================
Before running this script, please ensure the following:

1. 🔑 IAM Credentials:
   You have Administrator Access Credentials in IAM.
   This is crucial as we'll be using CloudFormation to create IAM roles and policies.
   Run 'aws configure' to set up your credentials.

2. 🌐 VPC Stack:
   Deploy the sagemaker-hyperpod VPC stack using:
   https://catalog.workshops.aws/sagemaker-hyperpod/en-US/00-setup/02-own-account
   This creates essential resources: VPC, subnets, FSx Lustre volumes,
   S3 bucket, and IAM role for your SageMaker HyperPod cluster.

3. 📊 Observability Stack:
   It's highly recommended to deploy the observability stack as well.
   Navigate to https://catalog.workshops.aws/sagemaker-hyperpod/en-US/00-setup/02-own-account#2.-deploy-cluster-observability-stack-(recommended) to deploy the stack

4. 💻 Development Environment:
   Ensure you have a Linux-based development environment (macOS works great too).

5. 🔧 Packages required for this script to run:
   Ensure you install the following: pip, jq, boto3, and jsonschema

Ready to proceed? Press Enter to continue or Ctrl+C to exit...

🔍 AWS Account Verification
Your AWS Account ID is: 123456789012
Press Enter to confirm ✅ or Ctrl+C to exit❌...

📦 1a: AWS CLI Installation and Verification
=== Checking AWS CLI Installation ===
✅ AWS CLI found. Checking version...
Current version: 2.22.18
Min. required version: 2.17.1
✅ AWS CLI version 2.22.18 is up to date.
=== AWS CLI Check Complete ===

🌎 AWS Region Configuration
Please confirm that your AWS region is us-east-1 (default).
If not, enter the AWS region where you want to set up your cluster (e.g., us-west-2):
> us-east-1
✅ Region updated to: us-east-1

Your region is set to: us-east-1
Ensure your chosen region supports SageMaker HyperPod.
You can check out https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod.html#sagemaker-hyperpod-available-regions to learn about supported regions.
Press Enter to continue...

🔧 Setting Up Lifecycle Scripts
Cloning awsome-distributed-training
⚠️  The directory 'awsome-distributed-training' already exists.
Do you want to remove it and clone again? (yes/no):
yes
Removing existing directory...
Cloning repository...
Cloning into 'awsome-distributed-training'...
remote: Enumerating objects: 687, done.
remote: Counting objects: 100% (687/687), done.
remote: Compressing objects: 100% (559/559), done.
remote: Total 687 (delta 101), reused 475 (delta 57), pack-reused 0 (from 0)
Receiving objects: 100% (687/687), 27.63 MiB | 25.88 MiB/s, done.
Resolving deltas: 100% (101/101), done.
✅ Repository cloned successfully
Enter the name of the SageMaker VPC CloudFormation stack that was deployed as a prerequisite (default: sagemaker-hyperpod):
ohmura-sagemaker-hyperpod
✅ Configuration script updated with stack name: ohmura-sagemaker-hyperpod
Generating new environment variables...
✅ New environment variables generated and sourced

=== Environment Variables Summary ===
Note: You may ignore the INSTANCES parameter for now
Current environment variables:
export AWS_REGION=us-east-1
export INSTANCES=g5.12xlarge
export VPC_ID=vpc-08a6dd95e3c6d8b80
export SUBNET_ID=subnet-0f86c5bd4038e1a6e
export PUBLIC_SUBNET_ID=subnet-020fb679044617906
export FSX_ID=fs-0a3d78c27db03d58e
export FSX_MOUNTNAME=ccuitb4v
export SECURITY_GROUP=sg-028d6a11916c85acf
export ROLE=arn:aws:iam::123456789012:role/ohmura-sagemaker-hyperpod-AmazonSagemakerClusterExe-HPprMfnS4bYv
export ROLENAME=ohmura-sagemaker-hyperpod-AmazonSagemakerClusterExe-HPprMfnS4bYv
export BUCKET=sagemaker-lifecycle-4681d320

=== Environment Setup Complete ===
=== Setting Up Lifecycle Scripts ===
Did you deploy the optional hyperpod-observability CloudFormation stack? (yes/no)
Observability not enabled. Continuing with default configuration
Uploading your lifecycle scripts to S3 bucket sagemaker-lifecycle-4681d320
✅ Lifecycle scripts uploaded successfully

=== Lifecycle Scripts Setup Complete ===
✅ Lifecycle scripts setup completed

🚀 Creating the Cluster
1c. Generating cluster configuration...

=== Lifecycle Scripts Setup Complete ===
Enter the name for the controller instance group [controller-machine]:
Enter the instance type for the controller [ml.m5.12xlarge]: ml.c5.large
Do you want to add a login group? (yes/no):
yes
Enter the instance type for the login group [ml.m5.4xlarge]: ml.c5.large
✅ Login Group added

=== Worker Group Configuration ===
Do you want to add a worker instance group? (yes/no): [yes]:
Configuring Worker Group 1
Enter the instance type for worker group 1 [ml.c5.4xlarge]: ml.c5.xlarge
Enter the instance count for worker group 1 [4]: 10
Are you using training plans? (yes/no):
no
✅ Worker Group 1 added
Do you want to add another worker instance group? (yes/no): [no]: no
What would you like to name your cluster? (default: ml-cluster):
✅ cluster-config.json created successfully

Creating provisioning_parameters.json...
✅ provisioning_parameters.json created successfully

Copying configuration to S3 bucket...
✅ Provisioning Parameters uploaded successfully

=== Cluster Configuration Complete ===
✅ Cluster configuration created successfully
ℹ️  Validating the generated configuration before proceeding
✅ Cluster configuration validated!
ℹ️  For your viewing, here's the cluster configuration generated. Please make sure it looks right before proceeding. Press enter to continue, or Ctrl+C to exit and make changes
{
  "ClusterName": "ml-cluster",
  "InstanceGroups": [
    {
      "InstanceGroupName": "login-group",
      "InstanceType": "ml.c5.large",
      "InstanceStorageConfigs": [
        {
          "EbsVolumeConfig": {
            "VolumeSizeInGB": 500
          }
        }
      ],
      "InstanceCount": 1,
      "LifeCycleConfig": {
        "SourceS3Uri": "s3://sagemaker-lifecycle-4681d320/src",
        "OnCreate": "on_create.sh"
      },
      "ExecutionRole": "arn:aws:iam::123456789012:role/ohmura-sagemaker-hyperpod-AmazonSagemakerClusterExe-HPprMfnS4bYv",
      "ThreadsPerCore": 2
    },
    {
      "InstanceGroupName": "controller-machine",
      "InstanceType": "ml.c5.large",
      "InstanceStorageConfigs": [
        {
          "EbsVolumeConfig": {
            "VolumeSizeInGB": 500
          }
        }
      ],
      "InstanceCount": 1,
      "LifeCycleConfig": {
        "SourceS3Uri": "s3://sagemaker-lifecycle-4681d320/src",
        "OnCreate": "on_create.sh"
      },
      "ExecutionRole": "arn:aws:iam::123456789012:role/ohmura-sagemaker-hyperpod-AmazonSagemakerClusterExe-HPprMfnS4bYv",
      "ThreadsPerCore": 2
    },
    {
      "InstanceGroupName": "worker-group-1",
      "InstanceType": "ml.c5.xlarge",
      "InstanceCount": 10,
      "InstanceStorageConfigs": [
        {
          "EbsVolumeConfig": {
            "VolumeSizeInGB": 500
          }
        }
      ],
      "LifeCycleConfig": {
        "SourceS3Uri": "s3://sagemaker-lifecycle-4681d320/src",
        "OnCreate": "on_create.sh"
      },
      "ExecutionRole": "arn:aws:iam::123456789012:role/ohmura-sagemaker-hyperpod-AmazonSagemakerClusterExe-HPprMfnS4bYv",
      "ThreadsPerCore": 1
    }
  ],
  "VpcConfig": {
    "SecurityGroupIds": [
      "sg-028d6a11916c85acf"
    ],
    "Subnets": [
      "subnet-0f86c5bd4038e1a6e"
    ]
  }
}

=================================================

==== 🎉 Cluster Creation Script Completed! 🎉 ====

=================================================
Congratulations! You've completed all the preparatory steps.
Next Steps:
Do you want the script to create the cluster for you now? (yes/no): [yes]:
⚠️  Please note:
   - Cluster creation may take some time (~15-20 min)
   - This operation may incur costs on your AWS account
   - Ensure you understand the implications before proceeding

✅ Creating cluster for you!
✅ Cluster creation request submitted successfully. To monitor the progress of cluster creation, you can either check the SageMaker console, or you can run:.
watch -n 1 aws sagemaker list-clusters --output table
Thank you for using the SageMaker HyperPod Cluster Creation Script!
For any issues or questions, please refer to the AWS documentation.
https://docs.aws.amazon.com/sagemaker/latest/dg/smcluster-getting-started.html

Exiting script. Good luck with your SageMaker HyperPod journey! 👋

[cloudshell-user@ip-10-134-45-116 automate-smhp-slurm]$