hyperpod-slurm-debugger
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on "drain before reboot", "diagnose a Slurm node", "investigate stuck jobs."
Install
With your agent
Install OMGSkills skill "awslabs/agent-plugins:plugins/sagemaker-ai/skills/hyperpod-slurm-debugger" into this agent's normal skills directory: use the OMGSkills MCP get_skill tool, install only the pinned_install commit and path, verify SKILL.md against pinned_install.skill_md_sha, and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/awslabs/agent-plugins /tmp/agent-plugins && ln -s /tmp/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-slurm-debugger ~/.claude/skills/hyperpod-slurm-debuggerFrom README
HyperPod Slurm Debugger Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command. For remediation, link to the official AWS or Slurm documentation. When to invoke Invoke when the user reports any of the symptoms in the decision table. When NOT to invoke Cluster has Orchestrator.Eks — invoke hyperpod-node-debugger or hyperpod-nccl. Single-node hardware fault with healthy Slurm scheduler — invoke hyperpod-node-debugger. NCCL training-hang investigation — invoke hyperpod-nccl. Node unreachable via SSM — invoke hyperpod-ssm. Constraints Read-only. Do not run, recommend, or print state-mutating commands. For any remediation, link to AWS or Slurm docs. The user authorizes and executes. IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation drifts the live state from the IaC plan.
More from this repo
aws-lambda-microvms
Build, run, debug, and operate applications on AWS Lambda MicroVMs — Firecracker-isolated, snapshot-resumable...
dataset-evaluation
Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR).
directory-management
Manages project directory setup and artifact organization.