lighteval good first issues (Python)

[BUG] custom model docs don't run: missing imports 10 days ago

bug documentation good first issue

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] LiteLLM concurrency parameters hard-coded 16 days ago

good first issue help wanted feature

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] Log progress bar on main process 18 days ago

help wanted feature science-team

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] Add tests for `VLLMModel` base methods about 2 months ago

good first issue feature

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

Append revision to filepath in `--output_dir`? about 2 months ago

good first issue feature science-team

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] showing count in Markdown summary table 3 months ago

good first issue feature

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] Adding PHARE 3 months ago

good first issue help wanted new-task

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[BUG] Optimize tokenization 4 months ago

bug good first issue help wanted science-team

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] Build in a way to specify specific IDs/Lines in Dataset to use as few-shot examples in the same split 4 months ago

good first issue help wanted feature

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] Manage script and language in the Language enum 5 months ago

good first issue feature

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] Long Horizon Execution 9 months ago

good first issue help wanted new-task

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] MultiChallenge 9 months ago

help wanted new-task

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[FT] Add tests for nanotron 11 months ago

good first issue help wanted feature ignore-for-release

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] Big-Bench Extra Hard (BBEH) 11 months ago

good first issue help wanted new-task science-team

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] Add kyrgyzLLM benchmark 11 months ago

good first issue new-task

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] SciCode: reasearch coding benchmark about 1 year ago

help wanted new-task

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

[EVAL] HELMET: long context evals about 1 year ago

help wanted new-task

huggingface/lighteval

2.5K

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

Python

#evaluation#evaluation-framework#evaluation-metrics#huggingface

huggingface/lighteval

Open Issues Need Help