Slurm Metric을 Amazon CloudWatch로 수집하고 Grafana 대시보드로 시각화해 봤습니다.

[PCS 클러스터]
  Slurm Controller (OpenMetrics 엔드포인트, 포트 6817)
        │
        │  (Prometheus 스크래핑 형식으로 노출)
        ▼
[CloudWatch Agent]  ← PCS 로그인 노드(또는 접근 가능한 EC2)에 설치
  - Prometheus 리시버로 스크래핑
  - EMF(Embedded Metric Format)로 변환
        │
        ▼
[Amazon CloudWatch]
  - Logs: 로그 그룹에 EMF 로그 적재
  - Metrics: 커스텀 네임스페이스(CWAgent/Prometheus)로 메트릭 저장
        │
        ▼
[Grafana] (별도 EC2에 구축)
  - CloudWatch 데이터소스로 연결
  - 대시보드에서 시각화

# 컨트롤러 IP는 PCS 콘솔의 클러스터 정보에서 확인
curl -s http://<컨트롤러_IP>:6817/metrics

slurmctld index of metrics endpoints:
  '/metrics/jobs': get job metrics
  '/metrics/nodes': get node metrics
  '/metrics/partitions': get partition metrics
  '/metrics/jobs-users-accts': get user and account jobs metrics
  '/metrics/scheduler': get scheduler metrics

curl -s http://<컨트롤러_IP>:6817/metrics/jobs        # 잡 상태별 개수 (running, pending, completed 등)
curl -s http://<컨트롤러_IP>:6817/metrics/nodes       # 노드별 CPU/상태 (node 라벨 포함)
curl -s http://<컨트롤러_IP>:6817/metrics/scheduler   # 스케줄러 내부 통계
curl -s http://<컨트롤러_IP>:6817/metrics/partitions  # 파티션별 잡 상태 (partition 라벨 포함)

# 배포판 확인
cat /etc/os-release

# Amazon Linux 계열
sudo yum install -y amazon-cloudwatch-agent

sudo mkdir -p /opt/aws/amazon-cloudwatch-agent/etc

sudo tee /opt/aws/amazon-cloudwatch-agent/etc/prometheus.yaml > /dev/null << 'EOF'
global:
  scrape_interval: 30s

scrape_configs:
  - job_name: 'pcs-slurm-jobs'
    metrics_path: /metrics/jobs
    static_configs:
      - targets: ['<컨트롤러_IP>:6817']

  - job_name: 'pcs-slurm-nodes'
    metrics_path: /metrics/nodes
    static_configs:
      - targets: ['<컨트롤러_IP>:6817']

  - job_name: 'pcs-slurm-scheduler'
    metrics_path: /metrics/scheduler
    static_configs:
      - targets: ['<컨트롤러_IP>:6817']

  - job_name: 'pcs-slurm-partitions'
    metrics_path: /metrics/partitions
    static_configs:
      - targets: ['<컨트롤러_IP>:6817']
EOF

sudo tee /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json > /dev/null << 'EOF'
{
  "agent": {
    "region": "<YOUR_REGION>"
  },
  "logs": {
    "metrics_collected": {
      "prometheus": {
        "log_group_name": "pcs-slurm-metrics",
        "prometheus_config_path": "/opt/aws/amazon-cloudwatch-agent/etc/prometheus.yaml",
        "emf_processor": {
          "metric_declaration": [
            {
              "source_labels": ["job"],
              "label_matcher": "^pcs-slurm-jobs$",
              "dimensions": [["job"]],
              "metric_selectors": ["^slurm_.*"]
            },
            {
              "source_labels": ["job"],
              "label_matcher": "^pcs-slurm-nodes$",
              "dimensions": [["job", "node"]],
              "metric_selectors": ["^slurm_.*"]
            },
            {
              "source_labels": ["job"],
              "label_matcher": "^pcs-slurm-scheduler$",
              "dimensions": [["job"]],
              "metric_selectors": ["^slurm_.*"]
            },
            {
              "source_labels": ["job"],
              "label_matcher": "^pcs-slurm-partitions$",
              "dimensions": [["job", "partition"]],
              "metric_selectors": ["^slurm_.*"]
            }
          ]
        }
      }
    }
  }
}
EOF

Dropped metric: no metric declaration matched metric name Metric name: slurm_nodes_down

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

# 상태 확인
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

# 로그로 드롭되는 메트릭이 있는지 확인
sudo tail -50 /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log | grep -i "drop\|unsupported\|error"

AccessDenied: User: .../GrafanaXXXQueryPolicy/... is not authorized to perform: cloudwatch:ListMetrics

export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH

cat > ~/test_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=test_job
#SBATCH --partition=<파티션명>
#SBATCH --ntasks=1
#SBATCH --time=00:03:00
sleep 90
EOF

sbatch ~/test_job.sh
squeue