- Home
- Skills
- Education & Learning
- OpenRLHF: RLHF Training at Scale
More screenshots
Works with the AI Agents you already use
OpenRLHF: RLHF Training at Scale
Pick the alignment method your data supports — SFT, DPO, PPO, GRPO, RLOO — audit preference pairs before training, size the run with a 1% pilot, and catch reward hacking early.
$5
OpenRLHF: RLHF Training at Scale
Example session with this skill installed
I have a 14B model SFT'd on support conversations and 30k preference pairs built from thumbs-up/thumbs-down product logs. I can get an 8-GPU node for about a week. My manager wants PPO because "that's real RLHF." Tell me the method, the audit, and the run plan.
- Read your context and instructions
- Compiled the openrlhf
Method call: DPO first — and the audit matters more than the method. Thumbs-log pairs are the riskiest input in the catalog: a thumbs-up confounds correctness with speed, tone, and length, so before any training, run the four audit checks. Expect the length-bias check to bite — if "chosen" responses run much more than ~10% longer than rejected ones, you are about to train verbosity. Check comparability (do pairs differ in answer quality, or in which agent shift produced them?), decontaminate against your eval set with an n-gram pass, and audit per slice, because support data mixes intents and one poisoned slice hides inside a clean average.
On method: your pairs already encode the preference signal an online PPO loop would sample from, and you have no separate reward signal beyond them — that is the worked example's rejection rationale almost exactly. PPO would cost roughly 3x the compute (four models in flight) for speculative gain. Write that rejection into the run record now, with the escalation trigger: if DPO plateaus on the alignment eval and spot-checks show a failure class the pairs don't cover, PPO gets reconsidered with fresh eyes.
Run plan: verify the SFT checkpoint is solid (never RL a shaky base). DPO at beta 0.1, one epoch over the audited pairs — drop pairs that failed the audit rather than training on all 30k. Pilot at 1% of steps to confirm memory headroom and stable KL, and extrapolate wall-clock against your week of node time. Blind human spot-checks — 20 outputs, no scores visible — at 25/50/75%. Evaluate in both directions at the end: support-quality eval for the gain, a general-capability set for the alignment tax, both reported in numbers. Ship only if the readiness scorecard hits 9/12 with no zero on data audit or eval honesty.
Connects securely to your tools. The creator never sees your data.
About this skill
Most alignment runs fail in the planning, not the training: a method chosen because it sounds like the "real" recipe, preference data nobody audited, a launch sized on hardware optimism, and a reward curve that climbs while the outputs quietly get worse. This skill is the discipline around OpenRLHF runs at 7B+ scale. It starts with an inputs/outputs contract and a hard stop rule — no preference pairs and no verifiable reward means the deliverable is an SFT plan, not an improvised RL stage. A data-audit step checks pair comparability, length bias, eval decontamination, and per-slice quality before anything trains. A method table plus explicit decision points maps what you have to the method it supports, with the rejected alternative written into a run record before launch. The eight-step protocol then covers the SFT-first rule, a reward-model accuracy gate for PPO/RLOO, a 1% pilot that extrapolates wall-clock while killing the run is still cheap, scheduled blind human spot-checks, reward-and-KL dashboard discipline, and a two-direction evaluation that reports the alignment tax in numbers. It includes a worked DPO example, a copy-ready run plan template, a pre-launch checklist, a readiness scorecard with a ship threshold, edge cases (tiny pair sets, gameable verifiers, checkpoint storage blowup), and the failure-mode catalog: reward hacking, KL collapse, mode collapse, reward-model worship, and hardware optimism.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Education & Learning.
- Guidance targets the OpenRLHF stack (Ray + vLLM generation, ZeRO-style training); other frameworks differ in flags and defaults, though the data and evaluation discipline transfers.
- Thresholds given (length-bias tolerance, pilot size, scorecard cutoffs) are field-tested starting points, not guarantees — calibrate them to your task.
- It does not write your training configuration for you; it governs the decisions around the run.
- Reward-model accuracy thresholds must be set for your task before the gate is meaningful.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected