Open Issues Need Help
View All on GitHub Add better test coverage for models 12 days ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Implement XQuAD 17 days ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Quac Dataset 23 days ago
feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Add Regression Testing 28 days ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Add New Lambada Translations about 1 month ago
good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Add TheoremQA about 2 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
CoQA's implementation only predicts the last answer of each text about 2 months ago
bug good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Add ZeroScrolls Benchmark about 2 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Implement TyDiQA 2 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
help wanted feature request
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
[New Task] COLLIE 4 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
New Task Request: LegalBench 4 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
New Task Request: InflectionAI's Physics GRE 4 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Hendrycks Math extraction rule seems too strict 6 months ago
good first issue validation
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Better Arg Typechecking / Validation 8 months ago
help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
reproduce llama 3 evals 8 months ago
good first issue validation
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
good first issue validation
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Add docstring for HFLM's many keyword args 10 months ago
documentation help wanted feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Plans to add new long-context benchmark like LongBench v2, Babilong, InfiniteBench datasets? 11 months ago
help wanted good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Ruler QA tasks do not work for `max_seq_lengths` < 4096 12 months ago
bug good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Regarding metric chrf's implementation about 1 year ago
feature request good first issue
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer
Evaluate Thinking models on MMLU tasks about 1 year ago
help wanted validation
EleutherAI/lm-evaluation-harness EleutherAI/lm-evaluation-harness A framework for few-shot evaluation of language models.
Open project →
13.3K
A framework for few-shot evaluation of language models.
Python
#evaluation-framework#language-model#transformer