More screenshots
Works with the AI Agents you already use
SentencePiece: Tokenizers Without Language Assumptions
Train and use language-independent tokenizers — BPE vs Unigram, vocab sizing, character coverage, and debugging the mismatches that corrupt multilingual text.
$5
SentencePiece: Tokenizers Without Language Assumptions
Example session with this skill installed
Our multilingual model mangles Japanese text and its sequences run far longer than expected. Diagnose the tokenizer and fix the training setup.
- Read your context and instructions
- Compiled the sentencepiece
Encoded a Japanese sample and counted unknown tokens: roughly one in eight characters came back unknown, which pointed straight at character coverage set too low for CJK data. The training corpus was checked next — Japanese made up under 4 percent of the lines, so the vocabulary had almost no Japanese pieces to learn from. The fix was two changes, not one: character coverage raised to 1.0 so no CJK character is ever dropped, and the corpus rebalanced with representative Japanese samples including numbers and punctuation. Vocab was also re-sized: 32k had been stretched across 40-plus languages, so sequences ran long in every non-English language. Moving to 64k shortened Japanese sequences by about a third while keeping the embedding matrix affordable. Retrained with Unigram so sampling-based augmentation could vary segmentations during training, and the tokenizer file was versioned with the model checkpoint — byte-identical at inference, including normalization rules and added symbols. Re-encoded the same sample: zero unknown tokens, and tokens-per-word for Japanese fell in line with the other languages. The training recipe, the coverage setting, and the corpus mix are now written down as one reproducible command, so the next tokenizer is built from the same decisions instead of rediscovered. The one open item is named plainly: Korean samples in the corpus are still thin, and the same check will be run for Korean before the next training run.
Connects securely to your tools. The creator never sees your data.
About this skill
Most tokenizers start by assuming English: split on spaces, then split words. SentencePiece starts from raw Unicode and learns the splits from data, which is why it is the default choice for multilingual models and CJK languages, where words are not space-delimited in the first place. This skill covers the core ideas — whitespace as a reversible symbol, training from raw text, and the BPE-versus-Unigram choice — then a full training recipe: vocab sizing, character coverage, corpus hygiene, and control symbols. It also covers using the tokenizer, sampling-based augmentation, and debugging the four problems that actually occur: garbage for one language, train-versus-inference mismatch, unexpectedly long sequences, and weird splits at word boundaries — plus when to use a different tokenizer instead.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
It covers SentencePiece specifically; English-only standard architectures may be better served by other tokenizer libraries.
Vocab sizing guidance is a starting point — measure tokens-per-word on your own language samples before committing.
Corpus quality limits everything: a tokenizer trained on a skewed corpus inherits the skew, however good the settings are.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago
Permissions
No special permissions declared or detected