UKGovernmentBEIS/inspect_evals

[Feature Request] Run Large Docker-Based Evaluations Without Running Out of Disk Space 8 days ago

enhancement help wanted

562

Collection of evals for Inspect AI

Python

Follow-up to #935 and #942: migrate remaining registry-based evals to the SWE-bench sandbox config pattern 21 days ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Support Reproducible Evaluation Reports about 2 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Reduce number of skipped tests 3 months ago

enhancement good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Audit evals for opportunities to use Task setup parameter 4 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[WIP] Implement Anonymous Usage Telemetry for Inspect Evals 4 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] Attempt to Persuade Eval 4 months ago

help wanted implementation average

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Add per-category results table to parse_eval_logs_for_evaluation_report.py 5 months ago

good first issue python easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Adding pre-built docker images 5 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Create a Cost Estimation Tool for Inspect Logs 5 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] healthbench ValueError empty dataset 5 months ago

bug good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] LiveCodeBench Pro 5 months ago

help wanted implementation average

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] GAIA does not respect --message-limit Inspect argument 6 months ago

bug good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Bug: `medqa` `FileNotFoundError` 6 months ago

bug good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Implement Generalized harbor_task() Function 6 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] Warning: Could not extract parameters for task_name: No module named 'inspect_evals' 6 months ago

bug good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] BFCL v3 6 months ago

help wanted implementation

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Consolidate some eval tests 6 months ago

enhancement good first issue help wanted tests

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Allow subdirectory level lint exclusion in tools/run_autolint.py 6 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Mandatory task versions on the evals 6 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Set up repo CHANGELOG.md 7 months ago

documentation enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Improve the type hints on Personality Bench 7 months ago

good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] StereoSet eval hardcoded to only use 5 samples 7 months ago

bug good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Eval Quality - Add BigCodeBench tests and check against evaluation checklist 7 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Address tests that fail fue to Huggingface connection fails 7 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Update listing.py to use templating engine 8 months ago

documentation enhancement good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Chore: add types to abstention_bench recipe modules 8 months ago

good first issue help wanted easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] MASK belief-consistency checking for numerical beliefs is overly-strict 8 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] MLE-bench fails to build Docker container 8 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

abstention_bench tests fail the slow test build step 8 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Targeted slow tests run on all evaluations after README change 8 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Get rid of calls to as_posix() 8 months ago

enhancement good first issue help wanted easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Choose appropriate sample ID for datasets using auto_id=True 8 months ago

good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] BioLP-bench 8 months ago

enhancement good first issue implementation easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] `shuffle=true` hardcoded for hellaswag loader 8 months ago

bug good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Early termination for targeted-slow-tests job 8 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Add Parameters section to eval readmes 8 months ago

documentation enhancement good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Make listing.py create readmes for new evals 8 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

heavy-tests github workflow should only run on UKGovernmentBEIS repo 8 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Pubmed - Record to Sample Formating 10 months ago

good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Open Issues Need Help