NinjaOfNeurons

NinjaOfNeurons/TokenLens

TokenLens is a Python‑based web app that helps you visualize how large language models tokenize text right in your browser. Instead of guessing how a prompt is broken up into tokens (which drives cost and behavior in services like OpenAI), TokenLens shows it live with colors and counts.

1 stars 1 forks 1 watchers Python MIT License
View on GitHub
1 Open Issue Need Help Last updated: Feb 28, 2026

Open Issues Need Help

View All on GitHub
Add a CLI version of TokenLens about 2 hours ago

AI Summary: The issue requests a Command-Line Interface (CLI) version of TokenLens to enable users to tokenize text directly in their terminal. This CLI should accept text input, allow encoder selection, and output tokens, IDs, and counts, with potential for color-coding and API cost estimation. The goal is to make TokenLens scriptable and more accessible to terminal users.

Complexity: 2/5
enhancement help wanted good first issue
NinjaOfNeurons/TokenLens
1

TokenLens is a Python‑based web app that helps you visualize how large language models tokenize text right in your browser. Instead of guessing how a prompt is broken up into tokens (which drives cost and behavior in services like OpenAI), TokenLens shows it live with colors and counts.

Python