Tokensift

Deterministic, local, tokenizer-level static analysis of prompt strings, Message[] arrays, and tool schemas.

Status : early, actively developed. Core engine, 20 rules, real dollar cost per finding, a CLI, and tokensift/matchers for vitest/jest all work today. OpenAI models are exact; Claude support is estimate-based, see Cost and pricing below. See DESIGN.md for tradeoffs made along the way.

LLM APIs charge per token, and token counts don't line up with characters or words as cleanly as you'd expect. A UUID, a base64-encoded file, an indented JSON blob: all of these cost more tokens than their length suggests, because the tokenizer can't find any reusable pattern in them. tokensift reads a prompt (or a whole message array, or a tool schema) and points out exactly where that's happening: this UUID cost 18 tokens and a short id would've cost 3, this block of instructions got pasted twice, this JSON would tokenize the same minified.

It does this by actually tokenizing the text with the encoder the provider uses, not by estimating from character count. For OpenAI models that means the real BPE vocabulary, so counts are exact.