nano-step

nano-step/eval-harness

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

32 good first / help-wanted issues · Shell · last activity Aug 23, 2026

12 stars 5 forks 12 watchers Shell MIT License

agent-testing ai-agents anthropic bash ci-cd claude developer-tools eval llm-evaluation llm-testing llmops opencode prompt-engineering regression-testing

View on GitHub Website

32 Open Issues Need Help Last updated: Aug 23, 2026

Open Issues Need Help

View All on GitHub

Add an `expect_exact_lines` variant to `kind: shell` about 2 months ago

enhancement good first issue help wanted

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Expand RUN_ID collision space (19010 → 31-bit or PID) 5 months ago

bug good first issue help wanted v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add `--since` filter to `eval-harness trend` (e.g. --since=7d, --since=2026-05-01) 5 months ago

enhancement good first issue help wanted

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add a `langgraph-node` runner for regression-testing LangGraph agents 5 months ago

enhancement help wanted runner

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Pretty-print `eval-harness status` output as a colored table 5 months ago

enhancement good first issue help wanted

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Document the case YAML with a JSON Schema (for editor autocomplete) 5 months ago

documentation good first issue help wanted

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add `eval-harness doctor` — preflight diagnostics command 5 months ago

enhancement good first issue help wanted

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Remove $workdir PATH-prepend in spawn.sh (fixtures can shadow system binaries) 5 months ago

bug good first issue v0.4.3 security

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Anthropic API rate-limit handling + exponential backoff in llm_judge.sh 5 months ago

enhancement help wanted v0.7.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Verify portability on macOS (BSD coreutils) — set up a CI matrix 5 months ago

help wanted v0.4.3 portability

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Write a GitHub Actions example workflow (.github/workflows/eval.yml) 5 months ago

documentation good first issue help wanted v0.5.0 ci-integration

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add JUnit XML + SARIF reporters (CI-friendly output) 5 months ago

enhancement help wanted v0.5.0 ci-integration

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add pre-push branch filter (skip WIP/feature/draft branches) 5 months ago

enhancement good first issue help wanted v0.6.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Detect 'no expectation' misconfig in shell check (silent FAIL today) 5 months ago

bug good first issue help wanted v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add python3 + pyyaml to preflight_check (currently silently breaks if missing) 5 months ago

bug good first issue help wanted v0.4.3 portability

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

🗺 Roadmap: v0.4.3 → v0.5.0 → v0.6.0 → v0.7.0 → v1.0.0 5 months ago

documentation help wanted

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Pin and document end-to-end: 'run a real eval-harness against omo-session-distiller' 5 months ago

documentation good first issue v0.5.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Document semver + deprecation policy 5 months ago

documentation good first issue v0.5.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add stochastic pass@k mode (T>0, multiple samples per case) 5 months ago

enhancement help wanted v0.7.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Cache LLM judge responses by (rubric_hash, artifact_hash) 5 months ago

enhancement help wanted v0.7.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add eval-harness ab --base=X --candidate=Y for A/B skill testing 5 months ago

enhancement help wanted v0.6.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Implement automatic warn-only → blocking promotion after N green days 5 months ago

enhancement help wanted v0.6.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Cross-skill behavioral interaction diagnosis (which other skill caused this break?) 5 months ago

enhancement help wanted v0.6.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Implement auto-fix applier (v0.4.0 only proposes; v0.5+ applies) 5 months ago

enhancement help wanted v0.5.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Self-eat: add evals/cases for eval-harness itself 5 months ago

enhancement help wanted v0.5.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Implement shared-state daily budget ledger for EVAL_BUDGET_USD 5 months ago

enhancement help wanted v0.5.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add --strict mode (flip warn-only off, exit 12 on first regression) 5 months ago

enhancement help wanted v0.5.0

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Remove dead code: propose_fixes_for_run with tautology select bug 5 months ago

bug good first issue v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Cap samples: field in llm_judge case YAML (prevent runaway cost) 5 months ago

enhancement good first issue v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

flock the history.ndjson append (concurrent runs corrupt JSONL) 5 months ago

bug good first issue v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

LLM judge verdict parser: scan only first line of response 5 months ago

bug good first issue v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing

Add SIGINT/SIGTERM trap to release lockfile fd 9 + mkdir-lock dir 5 months ago

bug good first issue v0.4.3

nano-step/eval-harness

12

Behavior-regression testing for LLM agents. 4-class attribution, 6-field FAIL schema, $-cost gating, flaky detection. Bash + jq. Works with opencode today, runner-pluggable.

Shell

#agent-testing#ai-agents#anthropic#bash#ci-cd#claude#developer-tools#eval#llm-evaluation#llm-testing#llmops#opencode#prompt-engineering#regression-testing