PyTorch Low-VRAM Block Streaming (Byte-Identical)

    1

    The cost is speed; paired runs produce byte-identical artifacts.

    $40

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    PyTorch Low-VRAM Block Streaming (Byte-Identical)

    Example session with this skill installed

    Our 5B video diffusion model needs 17 GB of VRAM but the job may only use 6 GB on the shared box, and it has to run under a 20 GB Windows job memory cap. Make it fit without changing the output.

    • Read your context and instructions
    • Compiled the pytorch low-vram block

    Summary

    Sharded the weights once, bound them to read-only mappings, streamed the 40 transformer blocks one at a time with pinned staging. Peak GPU 5.0 GiB, peak job commit 10.4 GiB, output byte-identical to the resident run.

    What was done

    • python -m gpu_block_streaming shard --source model.safetensors --output shards/v1 --format video-5b-bf16-v1 --sha256 model.safetensors=<approved> --cast-floating-to bfloat16: 1,214 one-tensor shards, manifest with every hash.
    • python -m gpu_block_streaming verify shards/v1 --format video-5b-bf16-v1: ok, no PyTorch needed.
    • Model built on the meta device; map_one_tensor_shards assigned every tensor from its verified mapping (no host copy, no commit charge).
    • keep_on_host on the text embedding table; place_outside_blocks for norms and heads; BlockStreamer(model.blocks, "cuda", staging=True).

    Measured

    ModeGPU allocationPeak GPUPeak commitTime
    resident24 GiB~17 GiB17.8 GiB94 s
    streamed + staging6 GiB5.0 GiB10.4 GiB98 s

    SHA-256 of both output files: identical.

    Tests

    python -m unittest test_streaming -v: 7 tests OK on the CPU device.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A model whose weights are bigger than the GPU memory you may use (a shared card, a leased slice, a memory-capped job) can still run at full fidelity if its blocks visit the GPU one at a time. This package does that for any model that runs its blocks in order: transformer encoders, diffusion transformers. The cost is speed; paired runs produce byte-identical artifacts.

    The Windows trap it avoids: under WDDM, GPU allocations and pinned host memory are charged to process commit, the number a Job Object cap counts, so "copy weights to host RAM and stream" can need more memory than keeping the model resident (a 5B model hit a 20 GiB cap with a 2.5 GiB GPU peak). Safetensors' safe_open maps copy-on-write, which is charged too. The fix: bind weights to read-only file mappings of one-tensor shards; the page cache holds them for free.

    Shard writer (shard): splits safetensors sources into one-tensor shards with a hashed manifest, reads by byte range (never mmap), verifies every source against an approved SHA-256, optional bf16 cast, atomic staging folder. Verifier (verify): checks a shard set without PyTorch. Mapper: builds on meta, assigns each tensor from its verified mapping, refuses wrong format, size, hash, shape, dtype, extra or missing keys. BlockStreamer: forward hooks swap each block to device copies while it runs, prefetch the next on a side CUDA stream, optional two pinned staging buffers filled by a background thread. keep_on_host for huge embedding tables, place_outside_blocks for the rest, private_commit_bytes to log what the cap sees.

    Measured: 5B diffusion transformer, 17 frames: resident 24 GiB allocation / 94 s; streamed 6 GiB / 210 s, 98 s with staging; byte-identical. Offline tests on the CPU device prove equality with and without staging and the refusals. Python 3.12+, PyTorch of your own, no other packages, no network.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Reduces speed by 2-3x due to PCIe transfer overhead. Inference only; cannot train or update weights. Commit meter is Windows-only. Requires Python 3.12+.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 19 days ago

    Software
    Python 3.12+PyTorch

    Use your own pinned PyTorch build.

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    Shell to run the shard, verify and test commands; read files for safetensors sources and shard directories; write files for the shard output directory and manifest. No network access, no environment variables.

    Listed19 days ago

    What's inside

    Frequently Asked Questions

    This skill is designed for any AI agent or application built on Python 3.12+ and PyTorch. It works with most transformer-based models (like Llama or Mistral) and diffusion models that use sequential block execution.

    This skill specifically solves the Windows WDDM 'commit limit' issue, where pinned RAM is double-counted as active GPU usage. By using memory-mapped sharding and commit metering, it allows you to run large models on Windows without hitting OS-level memory crashes.

    You will receive the BlockStreamer execution core, a weights-to-shards conversion utility, and a SHA-256 manifest verification tool. The package also includes documentation on how to integrate the wrapper into your existing inference scripts.

    While it enables you to run massive models on small GPUs, expect a performance trade-off. Because weights are streamed over the PCIe bus during the forward pass, inference speed is typically 2 to 3 times slower than if the model were fully resident in VRAM.

    Setup involves a one-time sharding process for your model weights using the provided utility. Once the weights are converted into one-tensor-per-shard directories, you simply wrap your PyTorch model with the BlockStreamer class to begin inference.

    No, this skill is strictly for inference. The memory-mapped shards are bound as read-only to maximize stability and security, meaning it cannot be used for fine-tuning or training where weights need to be updated.

    New in LLM Ops