evaltrust good first issues (Python)

Adapter: HuggingFace lighteval 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: SWE-bench results (convenience) 3 months ago

help wanted adapter priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: Weights & Biases / Weave eval export 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Report a headline P(model A > model B) 3 months ago

enhancement help wanted statistics priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Run-aware comparison: run-aware resampling estimator 3 months ago

enhancement help wanted statistics priority: high

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: Langfuse scores export 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: OpenCompass 3 months ago

help wanted adapter priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: HELM per-instance results 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: Arize Phoenix evals 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: Braintrust experiment export 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: MLflow evaluate 3 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Support run-level (aggregate) score input + unpaired two-sample comparison 3 months ago

enhancement help wanted statistics priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Run-aware comparison: expose repeated runs to the comparison path 3 months ago

enhancement help wanted priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

All-pairs comparison for files with >2 models 3 months ago

enhancement help wanted priority: high

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add a Makefile with common dev commands 3 months ago

documentation help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

List available model names when a --model-a/-b isn't found 3 months ago

enhancement help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add __str__ to Status and VerdictLevel enums 3 months ago

enhancement help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add unit tests for magnitude_label thresholds 3 months ago

help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add a CITATION.cff so the project is citable 3 months ago

documentation help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add CSV, JSONL, and multi-metric example files 3 months ago

documentation help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add more column-name aliases for the generic adapter 3 months ago

enhancement help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add more pass/fail score-word aliases 3 months ago

enhancement help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Parse percent-style scores like "85%" in coerce_score 3 months ago

enhancement help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Expose evaltrust.__version__ in Python 3 months ago

enhancement help wanted good first issue priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: EleutherAI lm-evaluation-harness 4 months ago

help wanted good first issue adapter priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Run judge reliability + calibration in single-model mode 4 months ago

enhancement help wanted priority: low

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Confidence interval on the effect size 4 months ago

enhancement help wanted statistics priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: Ragas (RAG evaluation) 4 months ago

help wanted adapter priority: medium

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Support pairwise-preference judgments (win / lose / tie) 4 months ago

enhancement help wanted

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: Inspect (UK AISI) .eval logs 4 months ago

help wanted adapter

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Adapter: generic JSONL of per-example records 4 months ago

help wanted adapter

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Continuous-score judge calibration (correlation, not exact match) 4 months ago

enhancement help wanted statistics

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Single-model repeatability check 4 months ago

enhancement help wanted

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Holm-Bonferroni correction for multi-metric suites 4 months ago

enhancement help wanted statistics

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

BCa (bias-corrected accelerated) bootstrap intervals 4 months ago

enhancement help wanted statistics

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Weighted / gated metrics in a suite 4 months ago

enhancement help wanted

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add a Markdown report output (`--md`) 4 months ago

enhancement good first issue

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add an `evaltrust --version` flag 4 months ago

enhancement good first issue

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add an OpenEvals format adapter 4 months ago

help wanted good first issue

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add an optional HTML report (--html) 4 months ago

help wanted

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add a LangSmith format adapter 4 months ago

help wanted good first issue

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

Add a score-word alias (good first issue) 4 months ago

good first issue

k-dickinson/evaltrust

1

An auditor for LLM evaluations — tells you whether you can trust your eval results, not just what the score is.

Python

k-dickinson/evaltrust

Open Issues Need Help