ai-reliability-index good first issues (Python)

Independently reproduce the v0.1.0-alpha quickstart 3 months ago

good first issue help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Audit evidence-brief-001 for ambiguity and evaluator leakage 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Design a second low-risk task from a different professional domain 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Create the project social-preview graphic and visual system 3 months ago

good first issue help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Improve the HTML report for accessibility and comparison 3 months ago

good first issue help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Define the official result manifest schema 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Implement consistency metrics and Wilson confidence intervals 3 months ago

good first issue help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Conduct an independent statistical review of methodology v0.1 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Implement transparent API and tool cost tracking 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Create the first low-risk professional benchmark task 3 months ago

good first issue help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Develop the v0.1 severe-failure taxonomy 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Design the provider-neutral agent adapter interface 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Build the reference benchmark runner CLI skeleton 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Define the Reliability Index v0.1 scoring specification 3 months ago

help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

Create a JSON Schema validator for benchmark task files 3 months ago

good first issue help wanted

AI-Benchmark-Collective/ai-reliability-index

0

An open, reproducible benchmark measuring AI-agent accuracy, consistency, failure severity, latency, and cost.

Python

AI-Benchmark-Collective/ai-reliability-index

Open Issues Need Help