ehtishammubarik/websieve

Docs: add a worked example with a real crawler end to end about 2 hours ago

good first issue docs

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Extraction loses content on pages with many short blocks about 2 hours ago

help wanted correctness

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Streaming input from S3 and GCS without staging locally about 2 hours ago

help wanted

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Benchmark suite so throughput claims are reproducible about 2 hours ago

help wanted performance

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Persistent dedup index: deduplicate a new crawl against an old one about 2 hours ago

help wanted roadmap-0.2

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Add --sample N to assess for calibrating against large files about 2 hours ago

good first issue

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Progress reporting: build is silent until it finishes about 2 hours ago

good first issue roadmap-0.2

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Resumable runs: a crash mid-corpus means starting over about 2 hours ago

help wanted roadmap-0.2

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Language detection: quality rules misjudge CJK and Thai text about 2 hours ago

help wanted roadmap-0.2 correctness

ehtishammubarik/websieve

0

Turn a web crawl into an ML-ready dataset. Boilerplate extraction, Gopher/C4 quality filtering, MinHash near-dedup, sharded output. Zero dependencies.

Python

#corpus#data-cleaning#data-pipeline#dataset#deduplication#llm#locality-sensitive-hashing#machine-learning#minhash#mlops#nlp#python#rag#scrapy#text-processing#training-data#web-scraping#zero-dependency

Open Issues Need Help