ron2111

ron2111/omnitoken

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

5 stars 0 forks 5 watchers Go MIT License

anthropic bpe gemini go go-library golang gpt huggingface llama llm mistral nlp openai pure-go sentencepiece tiktoken token-counting tokenization tokenizer wordpiece

View on GitHub Website

6 Open Issues Need Help Last updated: Jul 8, 2026

Open Issues Need Help

View All on GitHub

Benchmark: Add Long Prompt and Cache Simulation Fixtures about 6 hours ago

help wanted type: benchmark area: cache area: benchmarks status: ready priority: medium

ron2111/omnitoken

5

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

Go

#anthropic#bpe#gemini#go#go-library#golang#gpt#huggingface#llama#llm#mistral#nlp#openai#pure-go#sentencepiece#tiktoken#token-counting#tokenization#tokenizer#wordpiece

CLI: Add omni models and omni encodings about 6 hours ago

good first issue type: feature area: cli status: ready priority: medium

ron2111/omnitoken

5

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

Go

#anthropic#bpe#gemini#go#go-library#golang#gpt#huggingface#llama#llm#mistral#nlp#openai#pure-go#sentencepiece#tiktoken#token-counting#tokenization#tokenizer#wordpiece

Feature: Detect Likely Cache Breakers about 6 hours ago

help wanted type: feature area: cache status: needs-design priority: medium

ron2111/omnitoken

5

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

Go

#anthropic#bpe#gemini#go#go-library#golang#gpt#huggingface#llama#llm#mistral#nlp#openai#pure-go#sentencepiece#tiktoken#token-counting#tokenization#tokenizer#wordpiece

Docs: Prompt Caching Guide about 6 hours ago

good first issue type: docs area: cache area: docs status: ready priority: medium

ron2111/omnitoken

5

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

Go

#anthropic#bpe#gemini#go#go-library#golang#gpt#huggingface#llama#llm#mistral#nlp#openai#pure-go#sentencepiece#tiktoken#token-counting#tokenization#tokenizer#wordpiece

Feature: Context Budget Planner about 6 hours ago

good first issue type: feature area: budget status: needs-design priority: high

ron2111/omnitoken

5

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

Go

#anthropic#bpe#gemini#go#go-library#golang#gpt#huggingface#llama#llm#mistral#nlp#openai#pure-go#sentencepiece#tiktoken#token-counting#tokenization#tokenizer#wordpiece

Feature: Add Local Cost Estimation API about 6 hours ago

help wanted type: feature area: cost status: needs-design priority: high

ron2111/omnitoken

5

Pure-Go LLM tokenizer and tiktoken-compatible token counter for OpenAI BPE, WordPiece, SentencePiece, Gemini, Llama, Mistral, and Hugging Face adapters.

Go

#anthropic#bpe#gemini#go#go-library#golang#gpt#huggingface#llama#llm#mistral#nlp#openai#pure-go#sentencepiece#tiktoken#token-counting#tokenization#tokenizer#wordpiece