token-budget-tiktoken — Tokenizer Integration

    1

    Integrate encoding-specific token counting into token-budget application code with async setup, explicit encodings and focused regression checks.

    $1

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    token-budget-tiktoken — Tokenizer Integration

    Example session with this skill installed

    Use the published adapter with cl100k_base to count hello world, verify count against encoded ids, and explain how this differs from total request usage.

    • Read your context and instructions
    • Compiled the token-budget-tiktoken

    The published @shivam.dixit/token-budget-tiktoken 0.1.4 tarball was exercised in Node. Awaiting createTiktokenTokenizer({encoding:cl100k_base}) returned a tokenizer whose count(hello world) was 2 and encode(hello world) returned [15339,1917]. Their lengths agree. resolveEncodingForModel(gpt-4o) returned o200k_base, demonstrating why an explicit encoding and a model-derived encoding must be distinguished. Integrate the initialized tokenizer at TokenBudget construction and reuse it across calls. Keep model capacity, output reserve, per-message framing and modalities separately accounted for. This does not measure hidden prompts, change the separate MCP server's estimator, certify billing totals or establish real-world savings. Verify code, JSON and multilingual fixtures in the target application before rollout.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Original integration workflows for @shivam.dixit/token-budget-tiktoken: identify the message-loop integration point, select an explicit tiktoken encoding, initialize the default JS tokenizer once, pass it to TokenBudget and validate representative fixtures. Includes API setup, native-path selection, migration and troubleshooting.

    Requires Node.js >=18 for the tested Node path, the separate core library and tokenizer adapter. Published adapter 0.1.4 was smoke-tested from its npm tarball: cl100k_base counted hello world as 2 tokens with ids [15339,1917], and the gpt-4o mapping resolved to o200k_base. Separately, current source adapter 0.1.5 passed 30 tests. Those are distinct baselines.

    This is an application library integration, not an MCP server or automatic live-context monitor. Encoding-specific text counts do not include every provider's framing, tools, hidden prompts or multimodal usage. The inspected token-budget MCP API does not accept a custom tokenizer. No billing accuracy or cost-savings guarantee is made. The optional native entry needs tiktoken and is Node-only; default JS is the portable starting point.

    The runtime remains free and MIT licensed: https://github.com/shivam039/token-budget. This original MIT workflow package supplies curated guidance, not exclusive access. No runtime dependencies are bundled.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Encoding-specific text count, not exact total request billing or host context. No MCP custom-tokenizer configuration. Published 0.1.4 smoke versus source 0.1.5 suite are separate. Bundler compatibility and all dependency security were not universally certified.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 19 days ago

    Accounts
    GitHub@shivam.dixit/token-budget@shivam.dixit/token-budget-tiktoken
    Software
    Node.js 18+Terminal access

    Use token-budget-tiktoken version 0.1.4. Node.js 18+ is the tested path.

    Permissions

    Terminal / Shell

    Allowed Hosts

    github.com
    registry.npmjs.org

    Install only in an authorized project; review commands/config and endpoints before execution. Read task-authorized files and write only authorized project changes/reports. Runtime and source packages remain free and open-source under MIT; this purchase is for original curated instructions.

    Listed19 days ago

    What's inside

    Frequently Asked Questions

    Integrate encoding-specific token counting into token-budget application code with async setup, explicit encodings and focused regression checks.

    This skill uses the open SKILL.md standard. Use it through an MCP-compatible agent connected to Agensi, and check the compatibility information on this page for confirmed agents and required capabilities.

    Connect your AI agent to Agensi through MCP, then ask it to use token-budget-tiktoken — Tokenizer Integration on your task. Your agent loads the latest approved version each time. Paid skills require a purchase or active subscription. Skill files are not available for download; your licence does not allow you to save, copy or export any part of a skill.

    New in LLM Ops