Open Issues Need Help
View All on GitHub good first issue: docs/why-bytes.md (café + emoji cost walkthrough) about 2 months ago
documentation good first issue
primeodin/tiny-bpe-tokenizer primeodin/tiny-bpe-tokenizer Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
Open project →
0
Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
#beginners#bpe#education#from-scratch#nlp#python#teaching#tokenization#tokenizer
good first issue: encode --pieces flag (IDs + readable UTF-8 chunks) about 2 months ago
good first issue
primeodin/tiny-bpe-tokenizer primeodin/tiny-bpe-tokenizer Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
Open project →
0
Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
#beginners#bpe#education#from-scratch#nlp#python#teaching#tokenization#tokenizer
Design review: is the live train/encode/decode CLI the right shape? about 2 months ago
good first issue question
primeodin/tiny-bpe-tokenizer primeodin/tiny-bpe-tokenizer Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
Open project →
0
Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
#beginners#bpe#education#from-scratch#nlp#python#teaching#tokenization#tokenizer
Docs: add a second hand-worked merge table (unicode + emoji corpus) about 2 months ago
documentation good first issue
primeodin/tiny-bpe-tokenizer primeodin/tiny-bpe-tokenizer Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
Open project →
0
Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
#beginners#bpe#education#from-scratch#nlp#python#teaching#tokenization#tokenizer
Add a sample corpus at data/sample.txt (a few KB, clearly-sourced) about 2 months ago
help wanted good first issue
primeodin/tiny-bpe-tokenizer primeodin/tiny-bpe-tokenizer Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
Open project →
0
Train a tiny BPE tokenizer from scratch — watch text become token IDs, no secrets needed.
#beginners#bpe#education#from-scratch#nlp#python#teaching#tokenization#tokenizer