NinjaOfNeurons

NinjaOfNeurons/TokenLens

TokenLens is a Python‑based web app that helps you visualize how large language models tokenize text right in your browser. Instead of guessing how a prompt is broken up into tokens (which drives cost and behavior in services like OpenAI), TokenLens shows it live with colors and counts.

1 stars 1 forks 1 watchers Python MIT License
View on GitHub
1 Open Issue Need Help Last updated: Feb 28, 2026

Open Issues Need Help

View All on GitHub
AI/ML • AI Applications
Add a CLI version of TokenLens about 2 months ago

AI Summary: This is a feature request to create a command-line interface (CLI) version of the existing TokenLens tool. The CLI would allow users to tokenize text directly in their terminal, supporting various encoders and potentially displaying token counts and estimated API costs. This would make TokenLens more accessible for scripting, automation, and users who prefer terminal-based workflows.

Complexity: 2/5
enhancement help wanted good first issue
NinjaOfNeurons/TokenLens
1

TokenLens is a Python‑based web app that helps you visualize how large language models tokenize text right in your browser. Instead of guessing how a prompt is broken up into tokens (which drives cost and behavior in services like OpenAI), TokenLens shows it live with colors and counts.

Python