gpu cluster scheduler

    1

    Design, optimize, and troubleshoot GPU cluster scheduling across Slurm, Kubernetes, and Ray.

    $5

    /mo

    Subscription · cancel anytime

    Secure checkout via Stripe

    • Always the latest version
    • Delivered live to your agent via MCP
    • Cancel anytime, access ends at period end
    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    gpu cluster scheduler

    Example session with this skill installed

    I need to submit a 2-node training job on Slurm using 8 A100s per node. Ensure tasks are bound to GPUs and memory is allocated correctly to prevent starvation.

    • Read your context and instructions
    • Compiled the gpu cluster scheduler

    I have generated the optimized Slurm batch script for your 2-node multi-GPU training job. This configuration enforces GPU binding to prevent oversubscription and sets specific memory limits per GPU to ensure stability.

    Slurm Batch Configuration

    #!/bin/bash
    #SBATCH --job-name=multi_node_a100_train
    #SBATCH --partition=gpu_a100
    #SBATCH --nodes=2
    #SBATCH --ntasks-per-node=8
    #SBATCH --gpus-per-node=8
    #SBATCH --cpus-per-task=12
    #SBATCH --mem-per-gpu=40G
    #SBATCH --gres-flags=enforce-binding
    #SBATCH --output=logs/%j.out
    #SBATCH --error=logs/%j.err
    
    # Load required modules or activate environment
    module load nvidia/cuda-12.1
    conda activate ml-env
    
    # Export NCCL variables for multi-node communication
    export NCCL_DEBUG=INFO
    export NCCL_IB_DISABLE=0
    
    # Execute training with srun
    srun python train.py --batch_size 32 --distributed
    

    Next steps

    • Verify node availability using sinfo -p gpu_a100 before submission.
    • Check GPU topology with nvidia-smi topo -m to confirm NVLink health.
    • Monitor real-time utilization with sstat -j <jobid> --format=AveCPU,AveRSS,MaxRSS.
    $5/mo

    Subscription · always the latest version

    • Secure checkout via Stripe
    • Cancel anytime
    • Delivered live via MCP

    About this skill

    High-performance AI workloads fail when GPU resources are mismanaged or underutilized. This skill provides the engineering expertise to design, configure, and optimize GPU clusters across Slurm, Kubernetes, and Ray environments.

    What it does

    • Multi-scheduler orchestration handles job submission and queue management for Slurm, Kubernetes, and Ray.
    • Resource partitioning configures MIG (Multi-Instance GPU) and time-slicing for multi-tenant efficiency.
    • Performance optimization tunes GPU topology, NVLink/InfiniBand bindings, and data loading to maximize throughput.
    • Utilization analytics monitors SM engines, HBM bandwidth, and power draw using DCGM and Prometheus.
    • Advanced troubleshooting resolves OOM errors, scheduling stalls, and interconnect bottlenecks.

    How it works

    1. Analyze cluster state by auditing node inventory, GPU topology, and existing queue policies.
    2. Define resource requirements using GRES flags for Slurm or resource limits for Kubernetes YAML manifests.
    3. Execute scheduling logic to deploy batch training jobs, distributed RL tasks, or inference services.
    4. Monitor and tune live workloads using nvidia-smi queries and DCGM metrics to identify bottlenecks.

    Frameworks & tools

    NVIDIA GPU Operator, DCGM, Slurm (HPC), Kubernetes (Volcano, Kueue), Ray Train, NCCL, InfiniBand, and RoCE.

    Why this beats prompting it yourself

    Generic AI lacks the specific syntax for complex GRES bindings and often misses critical topology considerations like NVLink domains. This skill applies battle-tested HPC patterns for preemption and fair-share that prevent cluster starvation.

    Use cases

    • Configuring a multi-tenant A100 cluster using MIG for isolated inference workloads.
    • Optimizing multi-node LLM training jobs to use InfiniBand instead of falling back to TCP.
    • Implementing Volcano batch scheduling on Kubernetes for gang-scheduled AI workloads.

    Known limitations

    Does not support proprietary cloud-only schedulers like AWS SageMaker or Google Vertex AI. Requires cluster-level permissions to modify configurations.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for DevOps & Cloud.

    Requires root/admin access for scheduler config. Excludes proprietary cloud-native schedulers. Limited to Slurm, K8s, and Ray.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 22 days ago

    Permissions

    Terminal / Shell
    Listed22 days ago

    What's inside

    Frequently Asked Questions

    This skill automates the complex configuration of GPU resources, handling everything from NVLink topology mapping to multi-tenant partitioning (MIG), which prevents common issues like resource starvation and underutilization.

    The skill is designed to work with any AI agent capable of executing terminal commands or YAML/config generation, specifically those used in MLOps and infrastructure automation workflows.

    Setup requires providing the agent with cluster-level access to your environment; the skill then performs an initial audit of your Slurm, Kubernetes, or Ray configuration to establish a performance baseline.

    Yes, the skill includes optimized configurations for multi-node training using NCCL over InfiniBand or RoCE, ensuring your distributed workloads avoid the latency bottlenecks of standard TCP networking.

    You receive the core scheduling logic, pre-configured templates for Slurm GRES and Kubernetes Volcano/Kueue manifests, and a suite of DCGM-based monitoring profiles for real-time analytics.

    While the skill excels at managing hardware and software scheduling layers like Slurm and Kubernetes, it requires cluster-level permissions and does not support proprietary, managed services like AWS SageMaker.

    New in DevOps & Cloud