NVIDIA-NeMo

NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

32 good first / help-wanted issues · Python · last activity Jul 9, 2026

1.7K stars 296 forks 1.7K watchers Python Apache License 2.0

data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication

32 Open Issues Need Help Last updated: Jul 9, 2026

Open Issues Need Help

View All on GitHub

Heuristic text filters raise ZeroDivisionError/ValueError on empty or whitespace-only documents 18 days ago

good first issue community-request waiting-on-maintainers

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

RayDataStageActorAdapter never calls teardown() on stages about 1 month ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Support batchsize/process_batch in the minHash stage about 1 month ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Support GlotLID for language/script identification. 3 months ago

good first issue Feature Text

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

[Ray] Pass expected language into `FastTextLangId` filter 4 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Text readers hang instead of failing fast when using `RayDataExecutor` 4 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Add support for data-designer 0.5.0 5 months ago

enhancement good first issue community-request waiting-on-maintainers

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Add a benchmark for Cloud I/O for text 5 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Alternateive to nvcv_image 6 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

See if all workflows expose file_extensions for reader i/o 6 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Bug: Overly permissive url regex 6 months ago

bug good first issue community-request waiting-on-maintainers

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Automatically detect when `IS_FANOUT_STAGE` should be set to True 6 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Ray Data Adapter (backend) should use compute instead of concurrency due to deprecation 7 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Default jsonl_writer to write data with kwarg force_ascii=False 7 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Audio stages: most fan-out stages do not set RayStageSpecKeys.IS_FANOUT_STAGE 7 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Better error messages when reading large parquet files 7 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Update stage name for nemotron-cc SDG stage 7 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

RayDataExecutor: start GPU actor pools at full concurrency instead of scaling up one-by-one 7 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Video vLLM stages have race condition during setup due to torch.compile cache 8 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Add benchmarking script for FastText filters 9 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Documentation 'Installation" & "Configuration" links broken 9 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Unable to use models which need `trust_remote_code=True` in `EmbeddingCreatorStage` 9 months ago

bug good first issue community-request

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Make sure all reserved keywords are attributes and not properties [name / resources / batch_size] 9 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Audio stage names aren't being propogated as noticed in benchmark script 9 months ago

bug good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Reorganize directory structure for `DocumentFilter` and `DocumentModifier` 9 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Keeping imports consistent across modalities, for images 9 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Consider removing MegatronTokenizerWriter from writer/__init__.py 9 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Improve error handling for `blocksize` / `files_per_partition` which can result in error: `Invalid file path or buffer object type: <class 'tuple'>` 9 months ago

bug good first issue community-request

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Standardize ID naming for deduplication 9 months ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Warn if n_clusters < 1000 as a default in SemDedup 9 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Enhance YAML support in Curator 10 months ago

enhancement good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication

Rename `pred_column` and `prob_column` to fields about 1 year ago

good first issue

NVIDIA-NeMo/Curator

1.7K

Scalable data pre processing and curation toolkit for LLMs

Python

#data#data-curation#data-prep#data-preparation#data-processing#data-processing-pipelines#data-quality#datacuration#datarecipes#deduplication#fast-data-processing#fine-tuning#large-language-models#large-scale-data-processing#llm#llm-data-quality#llmapps#python#semantic-deduplication