More screenshots
Works with the AI Agents you already use
Ray Data Pipeline Design: Scale Without Rewriting
Design and tune distributed data pipelines with Ray Data — for datasets that outgrow single-machine pandas, preprocessing that feeds trainers, and pipelines that run on a laptop and a cluster.
$5
Ray Data Pipeline Design: Scale Without Rewriting
Example session with this skill installed
Preprocessing for a training job takes 9 hours on one machine and the dataset grows weekly. Design a Ray Data pipeline that runs on a laptop now and a cluster later, and size it.
- Read your context and instructions
- Compiled the ray data pipeline
Fit check first: the dataset is 400 GB and growing, preprocessing is CPU-bound, and the trainer needs a steady feed — Ray Data fits, and single-machine pandas does not. The pipeline is staged as a lazy graph: read in streaming blocks, map the cleaning functions, filter the bad rows, and write sharded output — no stage materializes the whole dataset. The sizing example does the arithmetic: at 8 workers the map stage is the bottleneck, so the plan starts with 16 workers on the cluster and 2 on the laptop, with the same code and a different worker count. Failure modes are named up front: a straggler block from skewed keys, backpressure when the trainer stalls, and object-store pressure from oversized blocks — each with its signature and its fix. Projected runtime is 40 minutes on the cluster against the current 9 hours, with the laptop run at 2 workers taking about 5 hours for a smoke test. The pipeline was written once, with the worker count and the block size as parameters — the laptop and the cluster run the same code, and the sizing note records the numbers so the next dataset gets sized instead of guessed. When the dataset grew to 600 GB the following month, the sizing note projected the new runtime without a fresh design session — the pipeline scaled by changing two numbers, and the projection landed within ten percent of the measured run.
Connects securely to your tools. The creator never sees your data.
About this skill
A pipeline that works on a laptop and dies on a cluster was never designed, it was hoped for. This skill designs and tunes distributed data pipelines with Ray Data: the lazy-graph mental model, streaming blocks, batch inference at scale, preprocessing that feeds Ray Train or Torch and TensorFlow trainers, and the tuning checklist that turns a slow pipeline into a fast one. It starts with the fit question — Ray Data is the wrong tool for small data, and the skill says when — then covers the two recipes that answer most jobs, the failure modes and their signatures, and a worked sizing example that shows the arithmetic before the cluster runs. The promise is portability: the same pipeline runs on a laptop today and a cluster tomorrow, with the differences named instead of discovered.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
It designs pipelines; it does not run your cluster or pay for it — sizing is arithmetic, and the cluster bill is yours.
Ray Data is the wrong tool for small data; the skill says when single-machine tools are faster and cheaper.
Performance numbers are projections from the sizing example until measured on your data — the skill shows how to measure, not how to skip measuring.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
Permissions
No special permissions declared or detected