Icaro0310

Icaro0310/devin-evals

Deterministic eval harness: replay recorded Devin sessions against rubric graders

2 good first / help-wanted issues · Python · last activity Oct 6, 2026

1 stars 0 forks 1 watchers Python MIT License
agents ai-agents ai-tooling benchmarks cli coding-agent developer-tools devin devin-ai devin-cli devtool evaluation good-first-issue hacktoberfest llm llm-eval llm-evals local-first python testing
View on GitHub Website
2 Open Issues Need Help Last updated: Oct 6, 2026

Open Issues Need Help

View All on GitHub
New defect: D10 — claims docs updated, no docs file touched 17 days ago
enhancement good first issue
Icaro0310/devin-evals
1

Deterministic eval harness: replay recorded Devin sessions against rubric graders

Python
#agents#ai-agents#ai-tooling#benchmarks#cli#coding-agent#developer-tools#devin#devin-ai#devin-cli#devtool#evaluation#good-first-issue#hacktoberfest#llm#llm-eval#llm-evals#local-first#python#testing
Add a `regex` rubric grader 19 days ago
enhancement good first issue
Icaro0310/devin-evals
1

Deterministic eval harness: replay recorded Devin sessions against rubric graders

Python
#agents#ai-agents#ai-tooling#benchmarks#cli#coding-agent#developer-tools#devin#devin-ai#devin-cli#devtool#evaluation#good-first-issue#hacktoberfest#llm#llm-eval#llm-evals#local-first#python#testing