UKGovernmentBEIS/inspect_evals

hf_dataset wrapper double-retries now that inspect_ai retries transient HF errors natively 13 days ago

enhancement good first issue easy

562

Collection of evals for Inspect AI

Python

Use inspect.utils.download 25 days ago

good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Feature Request] Run Large Docker-Based Evaluations Without Running Out of Disk Space about 2 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Follow-up to #935 and #942: migrate remaining registry-based evals to the SWE-bench sandbox config pattern 2 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Support Reproducible Evaluation Reports 3 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Reduce number of skipped tests 4 months ago

enhancement good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Audit evals for opportunities to use Task setup parameter 5 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[WIP] Implement Anonymous Usage Telemetry for Inspect Evals 6 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] Attempt to Persuade Eval 6 months ago

help wanted implementation average

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Add per-category results table to parse_eval_logs_for_evaluation_report.py 6 months ago

good first issue python easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Adding pre-built docker images 6 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Create a Cost Estimation Tool for Inspect Logs 7 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] healthbench ValueError empty dataset 7 months ago

bug good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] LiveCodeBench Pro 7 months ago

help wanted implementation average

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] GAIA does not respect --message-limit Inspect argument 7 months ago

bug good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Bug: `medqa` `FileNotFoundError` 7 months ago

bug good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Implement Generalized harbor_task() Function 7 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] Warning: Could not extract parameters for task_name: No module named 'inspect_evals' 7 months ago

bug good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] BFCL v3 7 months ago

help wanted implementation

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Consolidate some eval tests 7 months ago

enhancement good first issue help wanted tests

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Allow subdirectory level lint exclusion in tools/run_autolint.py 7 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Mandatory task versions on the evals 8 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Set up repo CHANGELOG.md 8 months ago

documentation enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Improve the type hints on Personality Bench 9 months ago

good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] StereoSet eval hardcoded to only use 5 samples 9 months ago

bug good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Eval Quality - Add BigCodeBench tests and check against evaluation checklist 9 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Address tests that fail fue to Huggingface connection fails 9 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Update listing.py to use templating engine 9 months ago

documentation enhancement good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Chore: add types to abstention_bench recipe modules 9 months ago

good first issue help wanted easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] MASK belief-consistency checking for numerical beliefs is overly-strict 9 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] MLE-bench fails to build Docker container 9 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

abstention_bench tests fail the slow test build step 9 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Targeted slow tests run on all evaluations after README change 10 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Get rid of calls to as_posix() 10 months ago

enhancement good first issue help wanted easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Choose appropriate sample ID for datasets using auto_id=True 10 months ago

good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Benchmark Implementation] BioLP-bench 10 months ago

enhancement good first issue implementation easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

[Bug] `shuffle=true` hardcoded for hellaswag loader 10 months ago

bug good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Early termination for targeted-slow-tests job 10 months ago

enhancement help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Add Parameters section to eval readmes 10 months ago

documentation enhancement good first issue easy

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Make listing.py create readmes for new evals 10 months ago

enhancement good first issue help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

heavy-tests github workflow should only run on UKGovernmentBEIS repo 10 months ago

bug help wanted

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Pubmed - Record to Sample Formating 11 months ago

good first issue

UKGovernmentBEIS/inspect_evals

562

Collection of evals for Inspect AI

Python

Open Issues Need Help