More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    MoE Training: Train Sparse Mixture-of-Experts Models

    1

    Train sparse Mixture-of-Experts models — make the capacity, routing, and parallelism decisions up front, then run a recipe that survives expert collapse.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    MoE Training: Train Sparse Mixture-of-Experts Models

    MoE Training: Train Sparse Mixture-of-Experts Models

    Example session with this skill installed

    We want a 47B-capacity model for the compute of a 13B one. Train an 8-expert MoE chat model on our budget and keep the routing healthy.

    • Read your context and instructions
    • Compiled the moe training

    Decisions before code: 8 experts per layer with top-2 routing — the capacity target is 47B-equivalent, the compute budget is 13B-equivalent, and 8 experts at top-2 is the configuration that fits both. The router gets a load-balancing loss from step one, not as a fix after collapse. Expert parallelism is set to 4 across the available GPUs, with tensor parallelism for the attention layers. Recipe: DeepSpeed MoE configuration with the capacity factor set to 1.25 — enough headroom that token dropping stays rare, not so much that the experts sit idle. The training data mixes chat, instruction, and general text so no single expert becomes the 'everything' expert. Early warning: at step 2,000 the router histogram showed two experts taking 60% of tokens. The load-balancing loss weight was raised and the learning rate on the router was lowered; by step 5,000 the distribution was even within 10% across experts. Caught by the histogram, not by the loss curve — the loss looked fine while the routing was collapsing. Failure catalog in use: token dropping spiked once when the capacity factor was too tight for a long-context batch — the fix was raising the factor for those batches, not retraining. One expert went silent for a stretch; the diagnosis was a dead router initialization, fixed by re-initializing that expert's router weights. Result: the 8-expert model trains to the target quality in the budgeted compute, with all eight experts active and the router balanced. The training record holds the DeepSpeed config, the router histograms at three checkpoints, and the two interventions with their causes — so the next MoE run starts from a recipe that has already survived its failure modes.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A Mixture-of-Experts model keeps dozens of feed-forward experts per layer but only activates a few per token, so you get the capacity of a 47B model for the compute of a 13B one — and the catch is that routing is a learned behavior, and learned behaviors fail in specific, diagnosable ways. This skill front-loads the decisions that prevent those failures: capacity, routing, and parallelism before any code runs, a DeepSpeed and HuggingFace recipe that survives expert collapse and token dropping, a failure catalog organized by symptom, and training discipline that keeps the run honest. The worked example trains an 8-expert chat model on a budget, with the failure modes named before they happen.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    It covers MoE training with DeepSpeed and HuggingFace recipes, not the design of a new routing algorithm.
    Routing failures are diagnosable, not always fixable mid-run — the skill shows how to catch them early, when a fix is still cheap.
    Capacity claims are compute-equivalent, not quality-equivalent; an MoE model must still be evaluated on the same benchmarks as a dense one.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    No special permissions declared or detected

    Listed3 days ago

    What's inside

    Frequently Asked Questions

    By your capacity target and your compute budget: 8 experts at top-2 is a common starting point for a 47B-equivalent model at 13B compute. More experts add capacity, not quality, unless the routing stays healthy.

    Expert collapse — the router found a shortcut. Watch the router histogram, not just the loss, and raise the load-balancing loss weight before the collapse hardens.

    Tokens routed to a full expert get dropped instead of processed. A little is normal; a spike means the capacity factor is too tight for that batch. Raise the factor for long-context batches.

    New in LLM Ops