PyTorch Low-VRAM Block Streaming (Byte-Identical)
The cost is speed; paired runs produce byte-identical artifacts.
$40
Works with the AI Agents you already use
PyTorch Low-VRAM Block Streaming (Byte-Identical)
Example session with this skill installed
Our 5B video diffusion model needs 17 GB of VRAM but the job may only use 6 GB on the shared box, and it has to run under a 20 GB Windows job memory cap. Make it fit without changing the output.
- Read your context and instructions
- Compiled the pytorch low-vram block
Summary
Sharded the weights once, bound them to read-only mappings, streamed the 40 transformer blocks one at a time with pinned staging. Peak GPU 5.0 GiB, peak job commit 10.4 GiB, output byte-identical to the resident run.
What was done
python -m gpu_block_streaming shard --source model.safetensors --output shards/v1 --format video-5b-bf16-v1 --sha256 model.safetensors=<approved> --cast-floating-to bfloat16: 1,214 one-tensor shards, manifest with every hash.python -m gpu_block_streaming verify shards/v1 --format video-5b-bf16-v1: ok, no PyTorch needed.- Model built on the meta device;
map_one_tensor_shardsassigned every tensor from its verified mapping (no host copy, no commit charge). keep_on_hoston the text embedding table;place_outside_blocksfor norms and heads;BlockStreamer(model.blocks, "cuda", staging=True).
Measured
| Mode | GPU allocation | Peak GPU | Peak commit | Time |
|---|---|---|---|---|
| resident | 24 GiB | ~17 GiB | 17.8 GiB | 94 s |
| streamed + staging | 6 GiB | 5.0 GiB | 10.4 GiB | 98 s |
SHA-256 of both output files: identical.
Tests
python -m unittest test_streaming -v: 7 tests OK on the CPU device.
Connects securely to your tools. The creator never sees your data.
About this skill
A model whose weights are bigger than the GPU memory you may use (a shared card, a leased slice, a memory-capped job) can still run at full fidelity if its blocks visit the GPU one at a time. This package does that for any model that runs its blocks in order: transformer encoders, diffusion transformers. The cost is speed; paired runs produce byte-identical artifacts.
The Windows trap it avoids: under WDDM, GPU allocations and pinned host memory are charged to process commit, the number a Job Object cap counts, so "copy weights to host RAM and stream" can need more memory than keeping the model resident (a 5B model hit a 20 GiB cap with a 2.5 GiB GPU peak). Safetensors' safe_open maps copy-on-write, which is charged too. The fix: bind weights to read-only file mappings of one-tensor shards; the page cache holds them for free.
Shard writer (shard): splits safetensors sources into one-tensor shards with a hashed manifest, reads by byte range (never mmap), verifies every source against an approved SHA-256, optional bf16 cast, atomic staging folder. Verifier (verify): checks a shard set without PyTorch. Mapper: builds on meta, assigns each tensor from its verified mapping, refuses wrong format, size, hash, shape, dtype, extra or missing keys. BlockStreamer: forward hooks swap each block to device copies while it runs, prefetch the next on a side CUDA stream, optional two pinned staging buffers filled by a background thread. keep_on_host for huge embedding tables, place_outside_blocks for the rest, private_commit_bytes to log what the cap sees.
Measured: 5B diffusion transformer, 17 frames: resident 24 GiB allocation / 94 s; streamed 6 GiB / 210 s, 98 s with staging; byte-identical. Offline tests on the CPU device prove equality with and without staging and the refusals. Python 3.12+, PyTorch of your own, no other packages, no network.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Reduces speed by 2-3x due to PCIe transfer overhead. Inference only; cannot train or update weights. Commit meter is Windows-only. Requires Python 3.12+.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 19 days ago
- Software
Python 3.12+PyTorch
Use your own pinned PyTorch build.
Permissions
Shell to run the shard, verify and test commands; read files for safetensors sources and shard directories; write files for the shard output directory and manifest. No network access, no environment variables.