fastokens v2: Accelerating inference and training time

fastokens v2 is a big step up. It adds first-class support for tiktoken models (including Moonshot's Kimi K3), swaps the pretokenization regex for a hand-written Unicode scanner, and rebuilds the BPE core for both serving latency and bulk throughput. Every change is validated bit-for-bit against the HuggingFace tokenizer.

Load tiktoken models directly

Until now, fastokens spoke only HuggingFace's tokenizer.json. v2 loads OpenAI-style tiktoken.model files too:

tok = Tokenizer.from_tiktoken("tiktoken.model", pattern=KIMI_PATTERN, special_tokens=specials)

So Kimi K3 and any model that ships only a tiktoken vocab now work out of the box: no format conversion needed.

Regex-free pretokenizer

Pretokenization, splitting text into "words" before BPE, is usually handed to a regex engine, and it's a surprising share of the cost. v2 replaces it with a hand-written scanner for the o200k and Kimi pattern families. ASCII is classified with direct byte checks; non-ASCII with tables built from the same regex-syntax data the reference matcher uses, so there's no regex fallback and no Unicode-version drift. The boundaries are identical to tiktoken's on English, code, CJK, and diverse input alike.

Rebuilt BPE core

The encode loop was rebuilt around one insight: on real text it's overwhelmingly a cache hit, and the cache is bigger than L2, so every lookup is a near-random memory access. That shows up directly in the numbers.

On Kimi K3 (Apple M4 Pro), the result:

• 10–46× faster than HuggingFace tokenizers across a range of production models
• 10× faster than tiktoken on inference- and training-relevant workloads. tiktoken encodes one string on a single thread, while fastokens parallelizes within the document.

Built for serving

Two features target LLM serving directly:

• A prefix cache for shared system prompts. When requests share a long prefix, such as a common system prompt or a reused context, fastokens tokenizes it once and reuses the ids, paying only for each request's unique tail. A 1M-token prompt that's 90% shared tokenizes in ~1 ms/request.
• encode_batch_flat, returning a whole batch as one compact uint32 buffer plus offsets (numpy-ready) instead of millions of Python integer objects, the shape bulk pipelines actually want.

Conclusion & future work

fastokens v2 closes the gap that mattered most: native tiktoken support, so Kimi K3 and other tiktoken-only models work without a conversion step, plus a faster pretokenizer and BPE core that hold up on both single-request serving and bulk throughput. Every result here is validated bit-for-bit against the reference tokenizer, so the speedup doesn't cost you correctness.

Support for additional tiktoken pattern families and further parallelism in the prefix cache are next on the list.

fastokens is open source, and we welcome contributions and model requests: https://github.com/crusoecloud/fastokens. 

Acknowledgments

Thanks to Marcel Rød, creator of gigatoken, whose tokenizer repo gave us inspiration for this work.